Version 0.1 · draft for comment · 2026-09-15

A vocabulary for recording whether two statistics may be compared, and what the comparison is worth. Written for cities producing Voluntary Local Reviews against the UN System Data Commons, and for the tools they build.

The problem this addresses. Joining a city indicator to a UN indicator is easy. Knowing whether the join is legitimate is the hard part, and it is currently done privately, inconsistently, and usually in a footnote nobody reads. More than 200 cities now produce VLRs. Every one of them is making these judgments; almost none of them are recording them in a form another city, or a tool, can use.

This spec is deliberately small. It is a vocabulary and five rules, not a methodology.

1. Grades

Every mapping between a local series and a UN series MUST carry exactly one grade.

A grade is a human judgment. It is not derived from the data, and tools MUST NOT compute or override it.

2. Tiers

A grade says whether a comparison is legitimate. A tier says what it is worth. Every mapping MUST carry one.

Tier 3 is the common case and MUST be labelled as such wherever a figure is presented. A city is not a country, and a reader who is not told will assume otherwise.

3. Required provenance

Every value presented under this spec MUST carry:

• source — the publishing organisation and dataset or variable identifier
• vintage — the observation period, and the retrieval date where the source is revisable
• unit — including the denominator where the figure is a rate
• method class — at minimum whether the figure is an administrative count, a survey estimate, or a modelled estimate

The method class is not optional. A modelled estimate beside an administrative count is the single most common way a defensible-looking chart misleads.

4. The five rules

R1 — The human grade vetoes every mechanical check. Resolution, unit agreement and period overlap are necessary conditions. None of them, in any combination, establishes comparability. Where an automated check and a recorded grade disagree, the grade wins.

R2 — Agreeing units are necessary and never sufficient. Two raw counts share a unit. So do a count of deaths under five and a count of deaths under one.

R3 — Missing data stays missing. A gap MUST NOT be interpolated, and a derived rate MUST NOT be published for a period whose denominator is estimated rather than measured. An invented value is visually indistinguishable from a measured one, and will be quoted as though it were.

R4 — Record why a mapping fails, not merely that it did. The reason is the durable artifact. A list of failures without reasons will be re-derived by the next person, and re-derived differently.

R5 — Ambiguity resolves to candidates, never to a guess. Where a query matches more than one mapping, a conforming tool MUST return the candidates and MUST NOT silently select among them.

5. Record format

A conforming crosswalk is a JSON document. Schema: spec/comparability-v0.1.schema.json.

{
  "id": "child-mortality",
  "grade": "CONTEXT",
  "tier": 3,
  "reason": "AGE BANDS DO NOT MATCH. The UN counts deaths under five; the local series counts deaths under one. Both are raw counts, so the units agree — which is the trap.",
  "comparator": {
    "source": "undata/who/CHILD_DEATHS.AGE--Y0T4",
    "place": "country/USA",
    "method_class": "modelled_estimate"
  },
  "local": {
    "source": "fcau-jc6k",
    "method_class": "administrative_count"
  }
}

reason is REQUIRED for every grade other than DIRECT, and SHOULD name the specific difference rather than restating the grade.

6. Conformance

An implementation conforms if it:

1. carries exactly one grade and one tier per mapping;
2. supplies a reason for every non-DIRECT grade;
3. presents no figure without source, vintage, unit and method class;
4. refuses to place CONTEXT, BLOCKED, NO-SOURCE or NO-SIGNAL mappings on a shared axis;
5. lets a recorded grade override any automated check (R1);
6. interpolates nothing (R3);
7. returns candidates on ambiguity (R5).

7. Reference implementation

This spec is a description of working software, not a proposal.

• Crosswalk — probe/crosswalk.json, 12 mapped pairs
• Validator — python3 probe/validate_crosswalk.py, which checks the crosswalk against the schema and the conformance rules
• Agent interface — mcp/server.py, an MCP server that refuses on non-chartable grades and returns the reason
• Worked output — the demo

8. Status and what would improve it

Version 0.1, drafted from one city's crosswalk against one publisher. Its obvious weaknesses:

• The grades were derived from 12 mappings in a single city. A second city will find cases they do not cover.
• method_class is coarse. Three values is probably too few; survey methodology varies more than that.
• Tiers are specific to the degree-of-urbanisation taxonomy the UN uses. A publisher that disaggregates differently would need a fourth tier or a different axis.
• Nothing here addresses sub-city comparison, which is what most city analysts actually want.

Corrections and counter-examples are more useful than adoption at this stage. Open an issue.