Activity feed

Wed Sep 16 · Devin Balkind · permalink

What NYC can say that the United States cannot

The asymmetry this project keeps circling: the UN graph is national-level, so a city can only be placed against countries — and for 130 of the 442 usable SDG indicators the United States reports nothing at all. On those, NYC vs the US is not a weaker comparison than NYC vs the world. It is the only one available, and the US is not in it.

Built the worksheet: us-silent-latest · python3 probe/us_silent.py

The number is 39, not 67

The briefing has said 67 since the corpus was first screened — 130 US-silent indicators, of which the scope classifier judged 67 city-scoped. Building the worksheet showed that number is inflated, and why.

28 of the 67 have a country as their subject, not a place. "Extent to which countries have laws and regulations that guarantee full and equal access to sexual and reproductive health care" is not a quantity New York City can hold a value for; the question is whether a national legislature passed something. Eighteen of the 28 are that one family — the SDG 5.6.2 legal provisions — and the matcher paired every one of them with NYC's Local Law 37/2011 Temporary Housing Assistance, on the token "Law".

The scope classifier could not have caught this. It reads an indicator's topic — health, water, education — and judges it municipal, which for 5.6.2 it is. What it cannot see is that the sentence measures a legislature. So that filter is now a lexical rule rather than a classifier: "extent to which…", "countries that…", "proportion of countries…" are excluded, and "extent of human made wetlands" is not. A rule you can read and argue with beats a score you cannot.

39 survive. 38 have a peer group of 15+ reporting countries, and 26 also have an NYC candidate above the noise floor.

The real answer is waste and water

Twelve of the 39 form one coherent group, and it is the group the briefing predicted:

These are DSNY and DEP series. NYC genuinely publishes them, and 133 countries report municipal waste collected while the United States reports none of it. That is a comparison NYC can make and its own country cannot.

Three of the twelve carry one year. That supports a level comparison and not a trend — the same finding that removed road safety as our headline demo, and the worksheet marks it rather than letting a reviewer discover it after choosing the indicator.

What corroborates it

Two results from the last two days point at the same set from opposite directions.

The corpus smell test found that sweeping all 689 indicators rather than the 442 usable ones adds almost nothing — but the 40 that do hold data are almost entirely e-waste and municipal waste import/export.

The inverse crosswalk found e-waste and hazardous waste per capita among the 123 indicators no municipal dataset comes near — nobody publishes them at city level either.

So this band of the framework is thinly reported by everyone: the US skips it, cities mostly skip it, and it is exactly where a city that does publish has something unusual to say.

The screen held

screen.py decides does the US report this from entityCoverage on a six-country panel — a proxy we have flagged as a proxy. Checked all 39 against the actual country observations: not one has a single United States datapoint. The proxy and the data agree completely.

That is the second independent confirmation of that panel this week; yesterday's --all sweep showed its NO-DATA grade predicts global emptiness 84% of the time.

Where it is still weak

• 12 of the 39 have no candidate above 0.45, the floor below which a proposal is noise. They are listed with the candidate in italics rather than silently dropped.
• Some above the floor are still wrong. Malaria incidence per 1,000 population at risk → Projected Population 2010–2040 at 0.48 is a false match, and NYC has no malaria.
• It is a worksheet, not a crosswalk. Under spec v0.1 a grade is a human judgment, so every row is graded: false. Nothing here is a mapping until a person says it is.

Also today

The platform went fully public this morning. python3 probe/launch_diff.py: 57 series, 12 variables and 689 corpus indicators, zero drift against yesterday's pre-launch baseline, and mcp/smoke.py 8/8. No public hostname resolves yet — undatacommons.unicc.biz and datacommons.un.org both still refuse — and the deployment we have always used is still answering. The demo is safe.

Wed Sep 16 · Devin Balkind · permalink

Correction: records access is eight cities, not fifteen

Wiring the inverse crosswalk's two headline categories into the demo meant checking their numbers a second way, and one of them did not survive.

The 16 Sep write-up said fifteen cities publish records-access request logs. Counted properly it is eight.

The fifteen was cluster membership — how many cities contributed a dataset to the k-means cluster whose distinctive terms were foia, request, log. That is not the same as how many cities publish a request log. The cluster had swept in Edmonton's Media Releases and NYC's City Hall Library Catalog, which sit near request logs in embedding space and are not request logs.

probe/category_gaps.py now counts the same categories without any clustering: search every named SDG indicator for the category's vocabulary, and count the municipal datasets whose titles carry it, in six languages. Both halves are greppable.

Elections went up, which is the useful part of the check: clustering was splitting one category across two runs and under-counting it, while over-counting the other. A method that is wrong in both directions is not a method you can read a number off.

What this does not change

The argument for both gaps stands, and the elections one is stronger than before:

• Zero of the 519 named indicators mention an election, a vote or a turnout.
• One mentions access to information — SDG 16.10.2, which asks whether a country has adopted guarantees. The framework measures whether a law exists; the cities measure whether it works.
• For 35 of the 469 election datasets, the closest concept in the entire framework is municipal waste collected.

The rule this leaves behind

A cluster is not a count. k-means returns k groups whether or not k categories exist, and its membership is the nearest thing to a centroid, not the things that belong to a category. We had already made it declare its own coherence and publish 14 of 40 clusters as diffuse. This goes further: where a category can be counted lexically, the lexical count is the one that gets published, and the cluster is what found it rather than what measures it.

Both categories are now on the demo, with the smaller, checkable numbers — and mcp/smoke.py asserts the page against probe/category_gaps.py, so they cannot drift apart.

Tue Sep 15 · Devin · permalink

Threshold calibration, and a registry we said didn't exist

Two corrections and one fix. The corrections are the useful part.

The CKAN registry exists. We said it didn't.

We reported that no global registry of CKAN portals survives — that every canonical one had rotted. That was wrong.

The CKAN Ecosystem Catalog is a 2025 NSF POSE II project from the CKAN core team, WPRDC and datHere. It lists 199 instances, 97 of them local or regional government, machine-readable at ckan/ckan-instances.

We missed it because we searched for the registries we already knew about — ckan.org/instances, dataportals.org, opendatainception — found all three dead, and concluded the category was dead. The correct inference was that we were looking at the previous generation. (The site sits behind Cloudflare and refuses both curl and a headless browser; the GitHub repo is the way in.)

The abandoned OKFN list really has rotted — 39 of 631 answer — which is what made the wrong conclusion feel supported.

Inventory now: 372 portals surveyed, 70 municipal, 543,000 datasets.

And a related correction: a portal that does not answer an anonymous package_search is not dead. data.gov, govdata.de and data.overheid.nl all refuse the probe and are plainly alive. The inventory now reports what responds to one specific API call, which is a much weaker claim than liveness.

The threshold fix, and the wrong turn before it

An absolute similarity cutoff cannot travel between catalogs. Measured on hand-judged matches, good pairs span 0.42–0.70 in NYC, 0.50–0.55 in Chicago, 0.37–0.59 in Madrid. One number is simultaneously too strict and too loose.

The first fix was wrong, and instructively so. We calibrated a z-score — how many standard deviations the top match sits above the catalog's own distribution — measured F1 0.68 against a labelled set, wired it in, and it admitted 60 of 80 indicators. Two independent errors:

1. The labelled set was biased. Every case in it came from candidates that had already passed the old 0.50 filter, so we measured on a pre-selected population and learned nothing about what the filter should have rejected.
2. z measures the wrong thing. Against a catalog whose similarities cluster near zero, the top hit sits many standard deviations above the mean whether it is right or garbage. z describes how peaked a distribution is, not how good its winner is.

What actually varies between catalogs — and is the right thing to normalise — is the distribution of top-1 scores across all probed indicators. The cutoff is now the best 20% of that, with an absolute floor so a uniformly hopeless catalog cannot contribute its least-bad rows anyway.

Bounded worksheets that adapt to each catalog, instead of 0 or 69.

What has not changed

No substantive SDG indicator has matched outside NYC. Better calibration produces a better worksheet; it has not produced a good mapping. NYC's pairs were found by a person who knew the data, with the matcher confirming them, and nothing yet shows the matcher can lead.

That is worth saying plainly a week out from Builders' Day: the infrastructure generalises, the judgment does not, and the judgment is the part that matters.

Tue Sep 15 · Devin Balkind · permalink

The launch diff, and the bug it found in itself

The platform goes fully public tomorrow, 17 Sep. Every figure this project publishes was measured against the pre-launch deployment, and the open question in the briefing was blunt: will staging DCIDs survive the launch? A withdrawn DCID does not raise an error. It returns an empty result, which is the same shape as a country that does not report — so the demo would show a blank chart to a room at Google NY and nothing would look broken until someone asked.

So: re-ran every probe against the live deployment and diffed it against 14 Sep, and built probe/launch_diff.py so that tomorrow's version of this is one command rather than an afternoon.

The graph has not moved

Not one DCID added, not one removed, not one value revised, not one unit changed. The three EMPTY series are the ones we already knew about — the UN holds no municipal-waste observations for the US, Japan or Mexico, which is the finding the municipal-waste card is built on.

The only differences anywhere in the crosswalk were ours, not the platform's: two figures in NYC's housing dataset moved by one (297,753 → 297,754 — NYC Open Data is live), the road-deaths pair was regraded BLOCKED → RANK-ONLY by hand yesterday, and the municipal-waste pair is new since the 14th.

The search surface has moved

The coverage probe grades whatever search_indicators returns for fifteen topic queries. Same code, same queries, same host, two days apart:

• 44 candidate variables → 56. Nine of fifteen topics gained rows. None lost any.
• Every row present on both dates is identical in all fifteen fields.
• So the grade mix moved — 30 GREEN / 9 AMBER / 5 RED → 36 / 11 / 9 — entirely because search returned more to grade, not because anything got better or worse.

The one worth noting: searching "road traffic deaths" on 16 Sep returns undata/sdg/SH_STA_TRAF — the exact DCID our road-deaths pair uses — and on the 14th it did not. Recall improved. That is good news and a warning in the same breath: the graph is frozen, the retrieval over it is not. Anything that resolves a DCID by searching at runtime can change its answer without the data changing at all. Our crosswalk hardcodes DCIDs that a human resolved once, which is exactly why the demo is insulated from this. It was a chore to do it that way. It is not a chore any more.

I checked the obvious alternative explanation before writing this down: search_indicators returns byte-identical results across three consecutive calls today, and neither coverage_probe.py nor candidates.json has been touched since the initial commit.

The bug it found in itself

The first run of launch_diff.py reported, confidently, that all twelve crosswalk DCIDs were missing from the graph — while get_observations was returning data for every one of them.

get_variable_metadata requires entity_dcids. Omit it and the server answers 200 with a completely empty body — no error, no status field, no variables key at all. A caller that trusts the response shape reads that as "every variable has been withdrawn". Two days before the launch it was written to detect, the launch detector was ready to cry wolf about the entire crosswalk.

That is the seventh entry in this project's list, and it was caught the same way as four of the other six: by printing a table of all twelve rows and reading it, rather than trusting a summary count. A checker that reports catastrophe is at least loud. The version of this bug that would have actually hurt is the mirror image — a diff that stays quiet because it cannot see.

So launch_diff.py --self-test now injects each drift class into a copy of the baseline and asserts the diff reports it: a revised value, a series going empty, a series coming back, a changed unit, a shortened span, a renamed variable, a withdrawn variable, an indicator dropping out of the corpus, a tool disappearing from the server. Plus the converse — an unmutated copy must report nothing. 10/10. A diff that returns "no change" and a diff that cannot see change are indistinguishable from the outside, and this project has shipped that mistake often enough to stop paying for it twice.

Tomorrow

The baseline is recorded as of today, pre-launch. After the platform goes public:

python3 probe/launch_diff.py        # exits non-zero if anything moved
python3 mcp/smoke.py                # the demo's published figures

If the public deployment answers on a different host, UNDC_ENDPOINT and UNDC_REST override it without a code edit, and the diff reports the host change as drift in its own right. As of this afternoon no public hostname resolves yet — undatacommons.unicc.biz and datacommons.un.org both refuse to connect, and the deployment we have been building against is still answering normally.

Tue Sep 15 · Devin · permalink

Correction: the other cities do have comparable data — our matcher just can't find it

The briefing and the last two posts carried this line:

No substantive SDG indicator has matched outside NYC.

True of our matcher. Read as a statement about the cities, it is wrong, and it was phrased so it would be read that way. Correcting it.

The data is there

Searching Chicago's and Boston's catalogs directly, by hand, for the four indicators NYC is crosswalked on:

Chicago's Traffic Crashes — Crashes is structurally the same thing as the NYC dataset behind our road-deaths card. Boston publishes a dataset literally called Vision Zero Fatality Records — the same programme, the same concept, named almost identically to what we used for NYC.

What our matcher did with it

Querying Boston for "Death rate due to road traffic injuries":

0.486  Traffic-Related Data
0.467  City of Boston Contract Award
0.444  My Neighborhood Dataset
0.387  Trash Collection Days
...
#23    Vision Zero Fatality Records        (0.286)

The correct dataset ranked 23rd of 235, below a contract-award file. No threshold rescues that; it is a retrieval failure, not a calibration one.

The matcher is not uniformly blind — querying homicide returns Homicide Clearance Rate at rank 1, because the word "homicide" appears in the title. It works when the vocabulary happens to line up and fails when a city names the same concept differently, which cities routinely do. "Vision Zero" is a programme name, not a description of its contents.

The honest position

• Chicago and Boston hold data comparable to what NYC is crosswalked on. Air quality in Chicago, road fatalities in both, crime in both.
• Some gaps are real. Neither publishes municipal waste tonnage in the DSNY sense, and Boston's only air dataset is building emissions rather than ambient concentrations.
• Our matcher cannot find what is there. That was already the stated conclusion — "nothing shows the matcher can lead" — but stating it alongside "no indicator matched outside NYC" implied the cities were empty. They are not.

This is the third time on this project that an automated check produced a confident answer a person overturns by reading for five minutes: the chartable flag that ignored human grades, the screen that reported 689 having recorded 9, and now a matcher whose silence was mistaken for absence.

A null result from a tool you have measured at ~50% precision is not evidence of absence. We wrote that lesson into the spec as R1 and then made the mistake anyway, in our own prose, about our own tool.

Tue Sep 15 · Devin · permalink

Fixed: the matcher now finds Vision Zero Fatality Records — and NYC got better too

Boston's Vision Zero Fatality Records ranked 23rd of 235 for "death rate due to road traffic injuries", below a contract-award file. It now ranks 1st. NYC's ground-truth median improved from 23 to 14 at the same time.

The diagnosis was not the query

First instinct was query expansion — add "crashes, collisions, fatalities, vision zero" to the indicator name. It made things worse (rank 23 → 30). So did rephrasing: "traffic fatalities" scored worse (32) than the clinical UN wording (23).

The problem was the document, not the query. Vision Zero Fatality Records carries excellent metadata:

title: Vision Zero Fatality Records
tags:  accidents bikes cars crashes fatalities pedestrians safety streets traffic vision zero
desc:  1,500 characters of programme mission statement — "our commitment to focus
       the city's resources on proven strategies… we are inspired by…"

A static embedding averages over every token. Sixty characters of exactly-right title and tags were being drowned by 1,500 characters of boilerplate. The dataset that beat it, My Neighborhood Dataset, won partly by having a shorter description.

The fix: score each field separately, take the best

Two vectors per dataset instead of one concatenated blob:

• head — title, category, tags, column names: the structured, high-signal fields
• body — description truncated to 600 characters, plus column descriptions

Score is max(head·q, body·q), so a dataset surfaces on whichever field actually carries its signal. Programme-named datasets are found by their tags; thinly-titled ones by their prose.

Measured both ways before adopting it:

Head-only fixes Boston and wrecks NYC — PM2.5 falls from rank 30 to 397, because NYC's air-quality dataset is found through its description. Taking the max gets both.

Boston now returns Vision Zero Fatality Records (0.544) and Vision Zero Crash Records (0.541) as the top two for road deaths.

A silent cache bug found on the way

Vector caches were keyed by city and model but not by the text representation. Changing how documents are built silently reused vectors from the old scheme — wrong rankings, no symptom, no error. Some of the earlier Boston numbers were affected by exactly this.

The cache key now includes a REPR_VERSION that must be bumped when the representation changes. Fourth entry in this project's running list of automated checks that were confidently wrong.

Still not solved

This improves retrieval; it does not make the matcher trustworthy. The cities now surface their obvious datasets, and a person still has to decide whether Vision Zero Fatality Records is comparable to SDG 3.6.1 — which, per the spec, is the one thing a machine may not do.

Tue Sep 15 · Devin Balkind · permalink

A smell test for the UN graph, and five things it found

Malaysia reports recycling 147.7% of its municipal waste. That figure is in authoritative UN data, it is on our demo, and we found it by accident while building a chart. This is the obvious follow-up, run properly: 442 indicators, 768,279 observations, every reporting country and every year, against checks dumb enough that they need no subject-matter knowledge — a percentage above 100, a negative count, a rate exceeding its own denominator, a value repeated across countries that should not agree.

Full results · python3 probe/smell.py

The tool flags; it does not judge. Everything below that I call an error, I checked by hand against the indicator's own distribution first.

Five that are wrong

1. Kyrgyzstan feels 6,990% safe walking home. SDG 16.1.4, proportion of population that feel safe walking alone around their local area after dark, unit Percent:

The other 252 observations, across 55 countries, run from 22.8 to 95.0, median 72.0. Divide the last three by 100 and you get 67.1, 68.4, 69.9 — which continues Kyrgyzstan's own trend from 66.8 exactly. Live for three consecutive years.

2. South Africa recycles 1.86 billion tonnes of municipal waste. Global municipal solid waste generation is roughly 2 billion tonnes a year, worldwide. South Africa's own earlier figures are 260,600 t (2005) and 520,800 t (2006); from 2018 they are 1.0–3.4 billion. Read as kilograms they are entirely sensible.

3. Brunei generates 12,580 tonnes of hazardous waste per person. EN_HAZ_PCAP, unit kilograms, where the global median is 22 kg and the maximum outside Brunei is 212. Brunei reports 8.7–36 million kg per capita, every year from 2016 to 2023. Against a population of ~450,000 that is 5.7 billion tonnes. It looks like a national total sitting in a per-capita field.

4. Guadeloupe's e-waste collection multiplies by a thousand in one year. 8.46, 8.54, 10.03, 10.36, 10.58, 11.97, 13.10, 13.71 — then 13,950 in 2022. The cleanest of the five: the series resumes perfectly if you divide the last point by 1,000.

5. Malawi is paid to send remittances. SI_RMT_COST, the average cost of sending $200, as a percentage. Malawi runs 16.95, 15.82, 14.47, 16.26, 14.79, 13.13 — then −0.1 and −0.93 — then 31.48. A cost of minus one percent, for two years, between normal values.

One that is not an error, and matters more

The United States recorded 345,600 disaster deaths in 2020 and 470,600 in 2021. Every other country in VC_DSR_MORT has a median of 42. Those are COVID-19 deaths: the US classified the pandemic as a disaster and reported it here. Most countries did not.

That is not a data error. It is the exact failure this whole project exists to prevent — two numbers that share a variable, a unit and an axis, and are not the same measurement. A chart of "disaster deaths, US vs peers" would be perfectly well-formed and would tell you something false. It is the strongest argument yet for the comparability grade, and we found it with a check that knows nothing about disasters.

Four that look wrong and are not

Each of these I chased and dropped, which is most of what the afternoon consisted of:

• Kuwait's water stress, 3,850%. Correct. Withdrawal beyond renewable resources via desalination and fossil groundwater. 371 observations exceed 100% across 17 countries.
• Marshall Islands, 132,810 disaster-affected persons per 100,000. Correct. A person counts once per disaster, so a small state hit repeatedly exceeds its own population. My check's premise — "more events than there are people to have them" — was simply wrong.
• Euro-area countries sharing a conversion factor of 1.08271; Benin, Burkina Faso and Cameroon sharing 710.208. Correct. The check found the euro and the CFA franc.
• 23,172 negative percentages. Correct, nearly all of them. "Annual growth rate of real GDP per capita", "Current account balance as a proportion of GDP", "Change in minimum river flow (%)" — the graph's Percent unit covers both bounded proportions and signed rates, and no unit string distinguishes them.

What the checks learned about themselves

The first full sweep returned 41,350 findings — 5.4% of all observations, which is not a result, it is a broken instrument. Three false-positive classes, each caught by reading rows rather than totals:

The first fix over-corrected: it gated the jump check on a unit whitelist, and thereby discarded Mauritius' food waste going 207 tonnes → 177,570 tonnes in a year because WEIGHT_TN was not on the list. Gating on the shape of the quantity — does this indicator ever go negative anywhere? — keeps it.

What came out of that is a principle worth more than any individual finding: an indicator is its own control group. A rule broken by most of an indicator's observations is its definition. A rule broken by three country-years out of three thousand is Kyrgyzstan. So checks are now suppressed per-indicator when they fire often enough to be structural — 165 suppressed groups, 38,943 would-be findings — and the suppressions are published, because the list of indicators that publish signed values under a Percent unit is itself a finding about the graph's vocabulary.

That left an outlier check that works against each indicator's own distribution rather than against what a unit is supposed to mean: 20× the indicator's 99th percentile. It survives suppression, which matters — "proportion of hazardous waste treated" exceeds 100% for 37 countries, so its range check is structural and gets dropped, and Guatemala's 44,825% would have gone with it. It is also the only check that reaches the 123 indicators and 248,457 observations whose units (CR_USD, WEIGHT_TN, INDEX, SCORE) have no meaningful range at all. The report states that coverage gap on its own face.

Final count: 2,457 findings, 92 of them HIGH. Small enough to read.

What this is for

Two things. It is a QA gate on the data our own crosswalk depends on — and it is the kind of thing only an outsider runs, because it requires no authority and no access, just the published graph and a willingness to look at 768,279 numbers. The five errors above are reportable to the platform team as they stand.

Tue Sep 15 · Devin · permalink

Denominators: Eurostat is the obvious answer and it is wrong

Most SDG indicators are rates per 100,000, so every non-US city needs a population figure before it can produce a chart. Two are now wired, and the route we did not take is the more useful finding.

Eurostat's Urban Audit publishes greater cities

urb_cpop1 covers 969 European cities annually and is exactly the single source this needs. It reports greater cities, not municipalities:

For Madrid the table contains no municipality entry at all — the greater city is the only option. A city's open data covers its municipality, so this is the denominator mismatch the whole project exists to catch, and it would have been completely invisible in the output.

So denominators come from the cities themselves

Better provenance as well: numerator and denominator then share a publisher.

Madrid — Padrón municipal (200076-0-padron), summing the four Spanish/foreign × male/female columns of a 34 MB CSV: 3,520,396 as of 2026-09-01. A snapshot, so Madrid supports levels and not trends until the historic padrón is wired.

Milan — Popolazione calcolata (ds1494), year-end population 1880–2025, ISTAT to 2002 then the city's own anagrafe: 1,399,079 in 2025. A series, so Milan supports trends — the first non-US city that does.

Two cities, two shapes, two resolvers. City statistical publications do not share a format, and pretending otherwise is how you end up with Eurostat.

The bug this work exposed

Building a summary table showed Chicago and Boston both reporting 8,478,072 — New York's population.

The census resolver fell through to NYC's place code whenever a city declared none of its own. Chicago's registry entry named its place code in a prose source string but carried no structured field, so it silently inherited NYC's 8.5 million. Every Chicago rate would have been three times too low, and nothing in the output would have looked wrong.

There is now no fallback. A city using the census module without explicit state and place FIPS raises, with the reason: refusing to guess, a wrong denominator is invisible in the output.

That is the fifth automated path on this project that produced a confident wrong answer, and the second where the failure mode was a silent default rather than an error.

Where it stands

Plus 23 more US cities resolvable by ACS place code. Buenos Aires declares no source and is refused rather than estimated. 24 non-US cities still need one, and each needs a person to find its statistical publication — the part that does not automate.

Tue Sep 15 · Devin Balkind · permalink

The whole corpus: what the other 247 indicators hold

The smell test ran against the 442 indicators screened usable. That left 247 graded NO-DATA or THIN-COVERAGE — and the grade was assigned against a six-country panel, so it only ever meant "no data for those six", not "no data anywhere". Ran it with --all to find out.

689 indicators · 773,335 observations · 2,503 findings.

The screening panel was right, and now that is measured

The extra 247 indicators contributed 5,056 observations — 0.65% of the corpus — and 207 of the 247 returned no country-level data at all. A grade assigned on six countries predicted global emptiness correctly 84% of the time.

That closes a question I had raised against our own method. The six-country panel was chosen for speed, with the honest caveat that it could not distinguish "the panel does not report this" from "nobody does". It turns out to distinguish them almost perfectly, and 442 remains the right denominator for everything downstream.

The 40 indicators that do hold data are worth knowing about, because of what they are: e-waste collected, recycled and its proportions; municipal waste imported and exported. Those sit exactly in the band we identified months ago as the interesting one — indicators a city can report internationally precisely because its own country skips them.

Guadeloupe, again, four times over

The previous sweep found Guadeloupe's e-waste collection multiplying by a thousand in 2022. With the full corpus it turns out the same error propagates through every derived indicator:

The largest total any other country has ever recorded for e-waste recycled is 899,300 tonnes. So Guadeloupe — population about 380,000 — is published as the world's biggest e-waste recycler, by a factor of six. One unit slip in a 2022 submission, carried into four indicators.

This is also the first time the checks corroborated each other: the same anomaly surfacing in four related series is much stronger evidence than the same anomaly surfacing once.

Small states reporting zero for two decades

The new coverage brought a pattern rather than a single error. Municipal waste exported and imported, both flatlined at exactly zero for very long runs: Mauritius 23 years, Dominica 22, Singapore 21 and 22, Cuba 20, Palestinian Territories 19, Jamaica 18, Saint Lucia 16, Liechtenstein 24.

Some of that is surely true — a small island genuinely exports no municipal waste. But a twenty-year run of exact zeros is also what "no submission" looks like when it is stored as a number instead of a gap, and the two are indistinguishable from outside. Worth one question to the platform team: is zero here a measurement or a default? Our own rule is that a missing year stays missing and is never interpolated; the same distinction matters just as much in the other direction.

Alongside it, "Proportion of electronic waste that is collected" sits at exactly 100% for Niger for eight consecutive years and Iran for six.

Everything else held

The finding counts barely moved — 2,457 to 2,503, with one new HIGH — which is the correct result for adding 0.65% more data. The five verified errors from the earlier sweep stand unchanged.

Tue Sep 15 · Devin Balkind · permalink

The inverse crosswalk: what cities measure that the SDGs do not

Every analysis in this repo has run city → UN: take an SDG indicator, find the municipal dataset that matches it. That direction can only ever discover what the framework already asks about. So: run it backwards. Take 11,206 datasets from 37 city portals, find each one's nearest neighbour among all 689 SDG indicators, and look at what is left over.

The question is not which cities are behind. It is what would SDG 11 look like if it had been written from municipal data upward?

Full results · python3 probe/inverse.py

The answer, as far as it goes

Themes that recur across many independent city catalogs and sit in the bottom quartile of every one of them:

The sharpest one is records access. Fifteen cities publish request logs — how many were filed, how many answered, how long it took. The nearest SDG indicator the matcher can find is 16.10.2, "countries that adopt and implement constitutional, statutory and/or policy guarantees for public access to information." The framework asks whether a law exists. The cities publish whether the law works. A country can score full marks on 16.10.2 and answer nothing, and no indicator in the framework would notice.

The same shape appears in call-centre performance: the SDGs have no concept of how quickly a government responds to its residents, and ten cities publish exactly that, in seconds.

The other direction

123 of the 689 indicators were the nearest neighbour of no municipal dataset at all. Many are honestly national — ODA flows, external debt, climate finance, which no city could report. But the list also contains e-waste collected, generated and recycled per capita, and hazardous waste generated — which is a double gap. Those are the same indicators we found the United States does not report, and it turns out cities do not publish them either. Nobody is measuring them at any level of government.

What this is worth, stated honestly

One positive control in seven failed. NYC's Housing Maintenance Code Violations is a hand-verified match for SDG 11.1.1 inadequate housing, and it landed at the 24th percentile — inside the tail. So the tail contains real matches, and any individual dataset in it may simply have been missed. That is why the unit of evidence above is cities, not datasets: one dataset scoring low is retrieval failure, and twenty cities independently publishing the same category is not.

Fourteen of forty clusters are noise, and the report says which. k-means returns k clusters whether or not k themes exist; each one now carries the mean cosine of its members to its own centroid, and below 0.62 it is published as diffuse rather than read as a theme. The first run had no such measure and cheerfully labelled a cluster "school" that contained building violations and lobbyist registrations.

English-language portals only. Comparing cosine similarities across two embedding models is meaningless, so the multilingual catalogs need their own run.

The bug underneath it

The first run's clusters were incoherent, and the reason was not the clustering.

embed.Index decided whether cached vectors could be reused by checking len(cached_ids) == len(datasets). Re-fetch a portal months later and it returns the same datasets in a different order — measured: 24 of 45 city catalogs did exactly that. Same count, same IDs, different sequence. The length check accepted it, and every dataset was handed another dataset's vector.

There is no symptom. Similarity scores stay in range, rankings look ordinary, nothing errors. It surfaced only because a cluster labelled "school" was full of facade-compliance filings and I looked at the rows.

It now matches on IDs, and when the set is identical but re-ordered it permutes the cached vectors instead of re-embedding. Worth noting what this didn't affect: the published municipal table was computed on the run that created its cache, in the order it created it, so those numbers were right. The bug only bites on a later re-run — which is to say, it was waiting for the next person to re-render the table.

That is the eighth entry in this project's list of automated checks that produced a confident wrong answer, and the fourth caught by reading rows rather than totals.

Tue Sep 15 · Devin Balkind · permalink

The inverse crosswalk in five more languages

Ran the inverse crosswalk over the non-English catalogs: 5,956 datasets from 11 cities in Italian, Portuguese, Spanish, German and Croatian, pooled into one run under the multilingual model. Results · python3 probe/inverse.py --language non-en

Pooled rather than run per language, because cosine similarities from two different models are not comparable and two-to-four cities per language could not support an analysis whose unit of evidence is how many cities. Pooled, they share one space and eleven cities.

Elections, in both language groups, independently

The tightest cluster in anything this project has produced: 241 datasets across 6 cities, coherence 0.90 — Milan's Elezioni Politiche: Risultati di Sezione, Madrid's Elecciones Autonómicas, polling-station-level results going back to 1996.

And the English run found the same thing separately: election results by voting station, 10 cities, coherence 0.71 — Calgary, Edmonton, and others.

Two disjoint sets of cities, two different embedding models, one category. The SDG framework has no indicator for electoral administration at all; the nearest thing the matcher can find for the English cluster is "number of local governments". Turnout, results, polling-station coverage — cities publish all of it and the framework has no word for any of it.

COVID-19 case reporting corroborates the same way: 6 English cities and 7 non-English cities, independently. The nearest indicators the matcher offers are "new HIV infections per 1,000 uninfected population" and "number of total conflict-related deaths". There is no city-level pandemic-surveillance indicator to match.

What only the non-English catalogs show

The property-valuation series is the interesting one. Housing affordability is squarely an SDG 11 concern (11.1.1 is inadequate housing), but the framework counts people in inadequate housing and the Italian cities publish what property costs, by neighbourhood, twice a year, for twenty years. Those are not the same measurement and only one of them has an indicator.

The controls came out better than the English run

11 of 11 passed, against 8 of 9 in English. These are hand-built: no city outside NYC has a graded worksheet, so the controls are datasets whose titles plainly correspond to an SDG concept, chosen across all five languages and recorded in probe/inverse_controls.json.

The two weakest are both Portuguese — Fortaleza's Número de Óbitos por Acidentes de Trânsito at the 44th percentile and Recife's Acidentes de Trânsito com Vítimas at the 32nd. Those are about as unambiguous as a match gets (deaths from traffic accidents ↔ SDG 3.6.1) and they sit in the bottom half. Portuguese retrieval is the weakest link, so the Brazilian portion of the tail is the least trustworthy part of this run, and anything resting on Fortaleza, Recife, Belo Horizonte or São Paulo alone should be read with that in mind.

A quarter of the framework was unmatched text

Both runs improved when I stopped matching against indicators that have no name.

170 of the 689 base indicators carry no name in any source we hold — they return neither metadata nor observations. The first version fell back to the DCID mnemonic, so a quarter of "the framework" was represented by query strings like DI ILL OUT. Nothing can match that, which inflates the tail — datasets look further from the framework than they are — and it put mnemonics in the nearest-indicator column, where VC VAW SXVLN was being reported as a cluster's closest SDG concept.

Excluding them moved every weak control up: Fortaleza from the 28th percentile to the 44th, Recife from the 27th to the 32nd. The framework is now represented by its 519 named indicators, and the count is stated on the page rather than folded into a total.

Two smaller fixes in the same pass. Cluster examples were being spread one-per-city to show reach, which made them unrepresentative of clusters dominated by one city — a cluster labelled for Italian property valuations was illustrated with a Madrid shop register. Each cluster now shows its most central members as well as the city spread. And the phrase view returns nothing here, which the page now says is expected by construction: Spanish, Italian, Portuguese, German and Croatian titles share essentially no bigrams.

Where it stands

The non-English run has better controls and worse coverage — 183 indicators are nearest to no dataset at all, against 69 in English, which is what you would expect from half the datasets and a weaker cross-lingual model. It is not evidence that those cities publish less.

Thu Sep 17 · Devin Balkind · permalink

Procurement: the framework has the slot and it is empty

Added public procurement as a third category to probe/category_gaps.py, expecting the elections result again — no indicator, lots of city data. The first run said exactly that: zero of 519 named indicators, 150 datasets across 28 cities.

It was wrong, and finding out why produced a better result.

The method was searching names, and a quarter of the framework has none

SG_SCP_PROCN sits in the SDG goal tree. It is the 12.7.1 series — sustainable public procurement — and it carries no name, so a search over indicator names cannot see it. Neither can it see SG_SCP_PROCN_HS or SG_SCP_PROCN_LS.

We have known since Wednesday that 170 of the 689 base indicators return no name. What we had not done is account for them when claiming a concept is absent. The category search now looks at DCID mnemonics too, and reports them as candidates for a human to read rather than counting them — mnemonics are noisy, and SE_ACS_ELECT is schools with access to electricity, not elections.

What the slots actually contain

All three procurement DCIDs exist. All three hold zero observations for zero countries.

The one adjacent indicator that does carry data is SG_SCP_CNTRY — countries with sustainable consumption and production national action plans — and its value is 1 for all 75 countries that report it. Nobody reports a 0. It is a list of countries that have a plan, recorded as a number.

So the framework does not lack a procurement indicator. It has three, and they are empty; and the thing next to them that works counts whether a country has written a policy down.

Against that, 28 cities publish 150 procurement datasets — tenders, awards, vendors, purchase orders. New York City alone registers 55,806 contracts worth $147 billion, every one with an agency, a vendor, a value and a status.

The pattern, now three for three

Twice the framework asks has a policy been adopted and the cities answer here is what happened. Once it does not ask.

Procurement is the sturdiest of the three

By this project's own evidence rule — count cities, not datasets — procurement is the best of them:

• 28 cities, against 23 for elections and 8 for records access.
• No dominant publisher. New York is 24% of the procurement datasets. Milan is 66% of the election datasets and Chicago 64% of the request logs. Procurement is the one category that is not mostly one city's filing habit.

One honest wrinkle: procurement datasets are not unusually far from the framework in embedding space — mean affinity 0.393 against 0.407 for municipal data generally, and only 32% fall in the bottom quartile of their own catalog, against 64% for elections. The reason is that a procurement dataset is about something — road contracts, health contracts, water contracts — and those subjects do have indicators. The data gets absorbed by its topic rather than recognised as procurement. The lexical search is what makes the gap visible; the embedding would have missed it.

Also true of the other two

Every unnamed DCID the mnemonic search surfaced across all three categories — 19 of them, including the democratic-institutions and judiciary series SG_DMK_JDC* and SG_DMK_PARLCC_* — holds zero observations. Unnamed and empty are the same set. That is consistent with Wednesday's --all sweep, where 207 of the 247 indicators outside the usable set returned no country data at all, and it is worth stating plainly: the framework's governance slots are largely unfilled, not merely unnamed.

Procurement is now a third block on the demo, and mcp/smoke.py (32 checks) asserts the empty-slot claim against the artifact, so it cannot quietly become false if the platform fills them in.

Thu Sep 17 · Devin Balkind · permalink

Enumerating the corpus depends on where you start

Every count this project publishes rests on one number: 689 base indicators, enumerated by walking ->relevantVariable from the seventeen SDG goal trees. Ran the same walk from the graph's actual root to see what else is there.

978 base indicators are unreachable from the SDG goal framework — WHO 354, UNICEF 245, ILO 93, UNIDO 69, UNAIDS 41, OHCHR 36, UNFPA 32, UNDP-HDRO 28, ITU 25, UNESCO 18, UNODC 12, ECLAC 6, UNDRR 5, IOM-DTM 1, UNHCR 1. The goal framework is 41% of the governed corpus.

The part worth reporting

Six base indicators are reachable from the goal trees and not from Root:

undata/sdg/SG_DSR_SILN   undata/sdg/VC_DSR_AGLH
undata/sdg/SG_DSR_SILS   undata/sdg/VC_DSR_CHLN
undata/sdg/SM_POP_REFG_OR  undata/sdg/VC_DSR_HOLH

All seventeen goal trees are direct children of Root — we checked, they are 17 of its 42 children. So a traversal from Root should be a strict superset of a traversal from the goal trees, and it is not.

Three things rule out the boring explanations:

• Neither walk reported a single fetch error. The walker prints and retries on failure; the logs are clean.
• The goal-tree walk is exactly reproducible. Re-run two days later it returned 689 again, the identical set. This is not run-to-run noise.
• The disagreement runs both ways. Twelve undata/sdg/ indicators are reachable from Root and not from the goal trees — the youth-in-parliament series SG_DMK_PARLYTH* and the global citizenship education series SE_SGE_* among them.

So ->relevantVariable is not transitive across these hierarchies, and no single entry point enumerates the graph completely. A client that picks one — as we did, for principled reasons — gets a silently incomplete corpus and no way to know it. That is worth the platform team knowing, and it is the third structural item we have for them, after the 170 unnamed indicators and the Percent unit covering both bounded proportions and signed rates.

What it does not change

The 689 is still the right denominator for this project, and nothing published needs correcting. A Voluntary Local Review reports against the SDG goal framework; enumerating from those seventeen trees is the correct scope, not a shortcut. The demo's masthead says "689 base indicators in the goal framework", which is exactly what it is.

What changes is that we now know the framework's share: 689 of 1,661, and the other 978 are the agency series — WHO's health indicators, ILO's labour series, UNICEF's child statistics — which a city might well want and which no SDG-scoped enumeration will ever surface.

Next

The smell test is running against the 978 now. The first twenty-five rows already show ILO series with 1,067 and 1,471 observations across 135 countries, so this is not a thin surface. All five data errors found so far came from undata/sdg/ alone, which was 41% of the graph; this is the other 59%.

probe/corpus.py --roots all now writes to its own file and keeps its own resume state, so a whole-graph walk can no longer overwrite the SDG corpus that every downstream count resolves against. It would have, silently, before today.

Sun Sep 13 · Devin · permalink

Systematic enumeration: 689 SDG indicators, 248 with US data, and a matcher that is only a third right

The first eleven crosswalk pairs came from topics I chose out of my head, which meant the crosswalk could only ever confirm my assumptions about NYC's data. This replaces that with a three-stage pipeline driven by the UN corpus.

python3 probe/corpus.py      # enumerate  -> 689 base SDG indicators
python3 probe/screen.py      # US coverage -> 248 GREEN
python3 probe/match_nyc.py   # NYC candidates -> ranked shortlist

The denominator

689 base SDG indicators. The eleven hand-picked pairs were 1.6% of it.

55% of SDG indicators have no US data at all. The real candidate pool is 248, of which my hand-picking sampled 4%.

Two traps in the platform worth knowing

The goal trees do not expose variables. Walking sdgf/goal-* yields undata/svpg/... nodes — StatVarPeerGroups — which carry no observations. The tempting move is rewriting svpg/sdg/X to sdg/X, which looks right and is precisely the DCID guessing the platform forbids. The correct path is following each group's ->member arc. Verified the svpg nodes return nothing before building on them.

get_variable_metadata silently truncates above ~10 variables per call — status: None and an empty map, not an error. The first screening run used batches of 40 and reported "screened 689/689" having actually recorded nine. Now capped at 10, with any short response treated as failure and re-split, and anything the graph never returned reported explicitly.

That is the third time on this project that every automated check passed and the number was still wrong. It keeps being the same lesson.

The matcher is the weak link, and it should be said plainly

First run produced mostly false positives. Short indicator names have few keywords, so one coincidental word scored 1.0:

• Number of local governments → EEO-4 Reports
• Secure tenure rights to land → City Council September Attendance Report
• Domestic material consumption → Mayor's Office to End Domestic Violence

Tightened two ways: a match now needs at least two distinct indicator terms rather than high proportional coverage alone, and inherently national indicators (balance of payments, ODA, tariffs, treaties, fisheries) are excluded before searching, since a city does not publish them and matching could only manufacture noise.

Result: 248 GREEN → 73 not city-scoped → 43 with candidates, 39 not yet in the crosswalk. Roughly a third of those look plausible on inspection. Term overlap is a genuinely poor proxy for semantic equivalence and no amount of tuning will fix that; the output is labelled a shortlist for human review and should be read as nothing more.

What it surfaced that I would never have picked

• CO2 emissions from fuel combustion (EN_ATM_CO2, 24 obs) against NYC's Climate Budgeting emission factors.
• Energy intensity of primary energy (EG_EGY_PRIM, 34 obs) against Local Law 84 building energy benchmarking — an area where NYC's data is unusually strong.
• Government spending on essential services (SG_XPD_ESSRV, 24 obs) against Agency Spending by Budget Function.

Those are the pipeline earning its keep. None were on my list.

Next

The honest improvement is not more scorer tuning. It is using a real embedding model against NYC dataset titles instead of my keyword overlap — the platform already does this properly for its own search, and I am reimplementing it badly.

Sun Sep 13 · Devin · permalink

The UN graph is national-level — NYC has no UN data

Probed the live staging deployment before writing any integration code. The headline result changes the shape of the project.

NYC resolves as an entity in the graph (geoId/3651000, type City) but appears in no variable's placesWithData — not for road deaths, not for PM2.5, not even for total population. Scoping a search to NYC alone returns zero variables and zero topics. places turns out to be a hard availability filter, not a hint.

So there is no "look up NYC in UN Data Commons," and anything that assumed we would query both sides and join is wrong. The crosswalk is the product, not a feature of it. That is awkward for the build but good for the pitch — the gap we proposed to fill is verifiably there.

Two supporting findings. The MCP surface only ever returned governed undata/ variables, while REST is federated with the wider Data Commons graph and will answer for other publishers without warning — so discovery goes through MCP, always. And provenance is first-class: every observation carries a provenanceUrl, an observationPeriod, and a unit DCID that encodes the denominator, which means part of our comparability check can be automatic rather than hand-curated.

Full detail and reproducible commands: platform probe findings.

Sun Sep 13 · Devin · permalink

Coverage probe: road safety is out, homicide and waste are in

Built the coverage probe harness and ran it across 15 candidate topics. It sweeps indicators, pulls each full series, and grades them on whether they can actually carry a chart. First run: 30 GREEN, 9 AMBER, 5 RED.

Road safety is dead as our headline demo. All three road-traffic variables came back RED with a single observation each — SDG 3.6.1 has exactly one US data point, 2021. That was the worked example in our application. Better to find out now than at 4pm on demo day, which is the whole reason the harness exists.

The replacements are stronger anyway. Homicide rate (16.1.1) has 21–31 annual points back to 1990 and NYC publishes closely matching data. Municipal waste recycled (11.6.1) has 19 points and pairs naturally with the DSNY diversion rate. Renewable energy share and unemployment are both dense too.

The genuinely interesting find: several variables carry an URBANIZATION--DOU_CITY slice, e.g. undata/sdg/EN_ATM_PM25.URBANIZATION--DOU_CITY. It is a national figure cut by degree of urbanization — "the city parts of the US", not any particular city — but it may be the most honest comparator available for a NYC number, and it partially softens the no-city-data finding. Worth an hour before we commit to a framing.

Two limitations, documented rather than hidden: the semantic search drifts (querying "urban public open space" returned PM2.5 variables, which is exactly why the crosswalk needs human review), and the modelled-estimate flag is name-based and under-detects — AIR_DEATH_R is modelled but does not say so in its name.

Latest coverage report · regenerate with python3 probe/coverage_probe.py.

Sun Sep 13 · Devin · permalink

Collaboration hub is live

Repo and briefing hub are up at sarapis/undatacommons-nyc, everything public.

The briefing page is written to be read by an AI assistant as well as by a person — paste the URL into Claude and ask it to get up to speed. Verified that flow actually works before calling it done, rather than assuming.

Two pages, two jobs. The briefing holds current state and gets overwritten, so it is always true now. This feed holds history and never gets overwritten. When a finding invalidates something in the briefing, post here and correct there.

Open decision that blocks the benchmark view design: do we compare NYC to peer cities (needs a non-UN source — OECD metro, Eurostat Urban Audit, UN-Habitat) or to nations (free and honest today: "NYC's rate sits between Portugal and Slovenia")? Logged in the decision log.

Sun Sep 13 · Devin · permalink

Embedding search: correct dataset moves from median rank 1535 to 23

Replaced the keyword matcher. The headline is not that embeddings are better — it is how much, and that we measured it instead of assuming.

The measurement

Ground truth is the seven NYC datasets already verified by hand in crosswalk.json. The question: out of 2,400 datasets, where does each method rank the correct one?

A ~65× improvement in median rank. Keyword overlap was not merely imperfect — at a median rank of 1535 out of 2400 it was worse than useless, since it ranked the right answer below the midpoint of a random shuffle. Every "candidate" it produced was effectively arbitrary.

Two changes, and the second was the surprise

Retrieval over the whole catalog. The old matcher could only re-rank whatever Socrata's keyword search returned, so retrieval was the real ceiling, not scoring. All 2,400 datasets are now cached and embedded locally.

Columns, tags and category are worth as much as the description. Adding them moved Proportion of municipal waste recycled → DSNY Monthly Tonnage from rank 882 to 23, because the dataset's title never says "recycled" — the concept lives in its fields and its tags.

What it still cannot do, and why that is structural

Two of the seven stay unfindable at any rank. NYC's homicide series is offence code 101 inside "NYPD Complaint Data Historic" — a fact that appears nowhere in that dataset's metadata. No text method can recover a mapping that depends on knowing what is inside a dataset.

So roughly 30% of real mappings need human domain knowledge and always will. That is the argument for the shortlist being a triage aid with a person in the loop, which is the same conclusion as the grade veto and for the same reason.

Result

248 GREEN → 73 excluded as inherently national → 53 with a candidate above the similarity floor, 50 not yet in the crosswalk. Eyeballing the top of the list, most are now plausible rather than most being noise.

New finds the keyword matcher never surfaced:

• Fixed broadband subscriptions → Broadband Adoption and Infrastructure by Community District
• CO2 emissions from fuel combustion → Office of Climate and Sustainability GHG inventory
• Government consumption expenditure growth → Mayor's Management Report Spending and Budget

Dependency

pip3 install model2vec — static embeddings, CPU-only, no torch, tens of megabytes. Without it the pipeline falls back to keyword matching and prints exactly how bad that is. The repo keeps its clone-and-run property; it is just measurably worse in that mode.

The similarity floor of 0.50 is a calibration choice with no theory behind it — the median top candidate across all indicators is 0.46, so it keeps roughly the better half.

Sun Sep 13 · Devin · permalink

DOU_CITY recovers peer comparison, and the NYC half is messier than the UN half

Two probes done. One unblocks the open decision; the other says the NYC side needs the same rigour we gave the UN side.

The peer-comparator question has a third answer

URBANIZATION--DOU_CITY is a real population-weighted aggregate, not a token. For the US it covers 121M of 347M people, and the city figure differs meaningfully from the national one (PM2.5 2019: 7.57 city vs 7.18 national).

The important part: it exists for other countries too, with full 10-year series. PM2.5 city aggregate, 2019 — US 7.57, Canada 6.87, UK 10.06, Japan 11.32, Germany 11.92, France 12.03, Colombia 15.97, Mexico 19.04.

So we can compare NYC against the city aggregate of other countries using nothing but UN Data Commons — one source, one method, one unit, consistent DEGURBA definitions. That is arguably better than the NYC-vs-London comparison we originally promised: no cherry-picked comparator, no second source to reconcile. Recommending this as the primary framing.

Caveat to state on any such chart: NYC is itself inside the US city aggregate (~7% of it).

The NYC half needed the same treatment

Built a pair probe that verifies both sides of every crosswalk entry. It immediately found things a human eye would have missed:

• NYPD complaint data carries junk pre-2006 incident dates — 17 murders in 1990, when the real figure was over 2,000. Filtering to 2006+ gives the true series (569 in 2006 → 277 in 2025).
• PM2.5 has summer/winter rows alongside the annual mean. Query the obvious way and you silently average three different measures together.
• DSNY's month field is text ("2026 / 08"), so date_extract_y fails outright — and rows before 1993 predate curbside recycling entirely.
• The published Recycling Diversion and Capture Rates dataset has not updated since 2020; the live path is deriving the rate from monthly tonnage.

Two of three pairs now resolve end to end. Homicide is blocked: NYC publishes a count, the UN publishes a rate per 100,000, and the Census ACS denominator now needs a free API key.

What the numbers say

PM2.5 — NYC is below the US city average and the gap is widening: 8.93 vs 9.23 in 2014, 6.60 vs 7.57 in 2019.

Waste recycling — NYC at 17.2% against 23.6% for the US in 2018. But NYC's figure is DSNY residential collection only, while the UN municipal-waste definition includes commercial. The gap is probably overstated, and the honest answer is that these are not measuring the same universe.

That contrast is the demo. One indicator where NYC looks good, one where it looks bad, and the bad one carries a caveat that changes the interpretation. It is the argument for the whole project in two charts.

Crosswalk status · python3 probe/pair_probe.py

Sun Sep 13 · Devin · permalink

Correction: the city-level comparator covers five indicators, not the general case

Earlier today I reported that URBANIZATION--DOU_CITY recovers peer-city comparison and recommended it as the primary framing. That was an over-generalisation from the one indicator I tested it on. Scoping it properly changes the picture.

Scanned 23 topic areas, 5,320 variables. Indicators carrying a DOU_CITY slice: five.

• undata/sdg/EN_ATM_PM25 — PM2.5
• undata/unicef/DM_BU_PC_DOU — built-up area per capita
• undata/unicef/DM_POP — population
• undata/sdg/AG_PRD_FIESS / AG_PRD_FIESMS — food insecurity

They cluster in indicators derived from gridded geospatial data (the GHSL settlement layer), which is the only family where a national figure can be cut by settlement type. Homicide, waste, unemployment, poverty and renewable energy have no such dimension and will not get one — they come from administrative reporting with no spatial component.

About eight more indicators carry only DOU_U (urban vs rural). DEGURBA "urban" bundles cities with towns and suburbs, so it is a coarser class, and the indicators it covers skew toward electricity access, handwashing and open defecation — little NYC relevance. Slums and school completion are the exceptions.

The comparator is tiered, not general

Tier 1 is real and PM2.5 is an excellent demo of it. The general case is still Tier 3.

That is arguably a better story than "we found city-level UN data." It means the tool's core job is telling a user which tier they are in and what that permits them to claim. A naive dashboard renders all three tiers as identical bar charts, and that is precisely the failure we are building against.

Sun Sep 13 · Devin · permalink

Denominators wired: all three pairs chartable, and NYC's homicide rate crossed below the US in 2013

Census key in hand, so the denominator problem is solved and every Tier 3 rate indicator is unblocked, not just homicide. All three crosswalk pairs now resolve end to end.

The homicide comparison

NYC counts murders; the UN publishes a rate per 100,000. With ACS annual population as the denominator, the two finally share an axis — and the result is a real finding:

NYC crossed from above the national homicide rate to below it around 2013, and stayed below through the 2021 spike. That is the kind of statement a city analyst can actually use, and it is invisible from either dataset alone.

The 2020 hole is real and we are leaving it

There is no ACS 1-year release for 2020 — the Bureau withheld the standard product after COVID disrupted collection. So 2020 has no denominator, and therefore no rate.

2020 is the year NYC homicides jumped to 456. The gap lands precisely on the most interesting year in the series.

We leave it empty. An interpolated denominator produces a rate that is visually indistinguishable from a measured one, and this is exactly the case where someone would quote it. The probe records dropped years explicitly rather than silently omitting them. 2025 is also absent for the ordinary reason that the vintage is not published yet.

Secret handling

The Census key lives in a gitignored .env and is read from the environment. It is in no tracked file and no commit — verified against the full history, not just the working tree. This repo is public and its parent workspace has a live incident from committed keys, so the check is deliberate rather than assumed.

Collaborators: get your own free key at https://api.census.gov/data/key_signup.html and export CENSUS_API_KEY. The probe explains this if the key is missing.

Sun Sep 13 · Devin · permalink

Demo built: three honest comparisons, eight blocked ones

Builders' Day is Tue 22 Sep, the morning is single-track demos, and until today we had no demo — a research pipeline and a docs site, neither of which is watchable for ten minutes in a room of forty builders. Now there is one.

Eleven Pairs, Three Charts

Three charts that hold, each carrying its source, vintage, unit and grade on the face of the card. Then the eight that don't, with the reason each one fails. The blocked ledger is not an appendix — it is the claim.

The screen the whole thing is built around

Municipal waste recycling. NYC at 17.2% against 23.6% for the US in 2018: a six-point gap and an easy headline. Except DSNY tonnage covers residential collection only and the UN definition includes commercial waste, so the two numbers do not cover the same universe.

Same units. Same period. Same apparent shape. Every automated check passes. Only a person reading both definitions catches it — which is the argument for the entire project, in one chart, in about ninety seconds.

Details worth keeping

• The homicide chart breaks at 2020 rather than bridging it, annotated no ACS 2020. The missing denominator is on the face of the chart instead of hidden by interpolation.
• Every card shows its DCID and its NYC dataset id, so any number can be re-derived from the page itself.
• The closing panel is the inversion: NYC has every road fatality geocoded daily, the UN has one 2021 modelled estimate. That is the argument for the evening showcase — city data flowing toward the UN system.

Published as a Claude Artifact and mirrored on GitHub Pages. The Pages copy is public and needs no sign-in, which is what a showcase screen should depend on.

Sun Sep 13 · Devin · permalink

Crosswalk at 11 pairs: only 3 can honestly share an axis

Expanded the crosswalk from 3 pairs to 11. Three are chartable. That ratio is the finding.

The harness had the exact bug we are building against

First run reported BLOCKED and CONTEXT pairs as "chartable: yes". The mechanical checks — does it resolve, do units agree, is there overlap — all passed, so the probe waved them through over the top of a human judgment that said these must not share an axis.

Road traffic deaths was the clearest case: marked chartable on a single overlapping year.

Fixed two ways. The human grade now vetoes the mechanical result, and a trend needs at least five overlapping years rather than a shared endpoint. Worth dwelling on, because it is the product's whole thesis reproduced in our own tooling: every automated check passed, and the answer was still wrong. Units agreeing is necessary and nowhere near sufficient.

Why the eight fail, which is the interesting part

• Age bands — UN counts child deaths under five, NYC's series is under one. Both are raw counts, so units "agree". A chart would look perfect and compare different populations.
• Denominators — maternal mortality is per 100,000 live births, not per population, and NYC's births live in a different dataset. Also NYC publishes pregnancy-associated deaths, a deliberately broader definition.
• Concept — the UN poverty series is the international extreme-poverty line (~$2.15/day). For any US city that is near zero and carries no signal. NYC's own measure is an order of magnitude higher. Two numbers sharing a name and measuring nothing alike.
• Measure type — UN reports the share of urban population in inadequate housing; NYC's nearest analogue is a count of hazardous housing violations, which tracks inspection activity as much as conditions.
• No NYC source — built-up area per capita has Tier 1 UN data with city-level slices across countries, and no NYC equivalent found yet.
• Ceiling effect — safely managed drinking water sits at ~100% for the US and would for NYC. Two flat lines at 100 tell an analyst nothing. Excluded deliberately, and written down as such.

The inversion worth showing at the event

Road traffic deaths is blocked by the UN side, not by us. NYC has Vision Zero collision data updated daily, every fatality geocoded. The UN has one modelled estimate from 2021. For the city's flagship street-safety programme, the international comparison simply does not exist.

That is the argument for city data flowing toward the UN system rather than only the reverse, and it is a better conversation to have with UNSD in the room than another dashboard.

Sun Sep 13 · Devin · permalink

NYC against the world, not just the US — and the homicide result inverts

Every chart in the demo compared NYC to the United States. That was a limit of my framing, not of the data: get_child_observations over Earth / Country returns every reporting country in one call, and the response carries entityMetadata names, so no separate lookup is needed.

Each card now shows where NYC sits among all reporting countries in 2019.

The homicide result inverts the story

Against the United States, NYC crossed below the national rate in 2013 and stayed below — a success story, and the one we put on the chart yesterday.

Against the world, NYC sits in the bottom half, 85th of 137, between Pakistan and Montenegro, with 84 countries reporting a lower rate.

Same number. Different comparator. Opposite conclusion. Neither is wrong. A tool that shows only the first is not neutral — it is flattering, and it is flattering by omission. This is the definitional-caveat lesson again, arriving through a different door: the choice of who you compare to is as load-bearing as what you measure.

Tier still governs. PM2.5 is Tier 1, so NYC is measured against other countries' city aggregates — genuinely like-for-like. Homicide and waste are Tier 3: a city against whole nations, which is real context and not a peer comparison, since cities generally run above their national averages.

One year, deliberately

date: "latest" returns each country's own latest vintage — Afghanistan 2023 sitting beside Aruba 2014. That is precisely the mixed-vintage comparison this project exists to catch, so everything is pinned to 2019, the last year with wide coverage across all three.

An impossible number in the authoritative data

Malaysia reports 147.7% of its municipal waste recycled in 2019. You cannot recycle more waste than exists. It is in the UN SDG database, and one bad figure was flattening the entire distribution on the chart.

We clipped it off the axis, drew it in red at the edge, kept it in the data and named it in the caption. Deleting it would have produced a cleaner chart and a dishonest one — a silent drop is how a dataset launders its own errors. Two countries also return empty names from entityMetadata; those fall back to their ISO codes rather than rendering blank.

For a room asked how do you hold the line on truth in a world flooded with synthetic data, a demonstrably impossible figure sitting in authoritative UN statistics is a more useful exhibit than anything we could have contrived.

Mon Sep 14 · Devin · permalink

Road deaths unblocked: NYC is 18th of 196, and the blocker was our framing

We graded road deaths BLOCKED because the UN holds one observation for the United States. That is true of a trend and irrelevant to a ranking: WHO publishes a single global round, and in 2021 195 countries reported. One year is more than enough to place a city.

The blocker was our framing, not the data.

The result

NYC: 297 road deaths in 2021, 3.51 per 100,000 — 18th of 196.

NYC's streets are as safe as Western Europe's and roughly four times safer than the country it sits in. That finding was completely invisible while the comparator was the US — which is the best argument yet for the world view existing at all.

The card puts one line beside one dot

NYC's eleven years of rates and the UN's single 2021 observation share one axis. The asymmetry is the argument: a dense blue line running low and flat, and a lone orange dot four times higher.

The grade is new — RANK ONLY. Not blocked, not chartable as a trend. NYC can be placed among 195 countries and cannot be tracked against them, because the world has one snapshot and NYC has eleven years. The inversion panel now says that instead of claiming the comparison fails.

A bug worth owning

The first version drew NYC's line straight through 2020 while annotating "no ACS 2020" beside it. The series omitted 2020 rather than carrying an explicit null, so the line bridged the gap. The homicide chart already broke correctly.

On a page whose entire argument is that missing data should stay visible, that was the wrong bug to ship. Fixed: an explicit null, a real break.

What else the world view opens, honestly

• Drinking water stops being flat — 162 countries, real variation. Still low value for a NYC analyst, so it stays excluded, with the reason corrected.
• The other six blocked pairs stay blocked. Mismatched age bands, a denominator in another dataset, the international poverty line, share-versus-count, no NYC source. Those are definitional; more countries cannot fix them.
• The 377 NO-US-DATA indicators look like a doubling and are not. They are overwhelmingly ODA, debt service, climate finance and "gross receipts by developing countries" — the US does not report them because it is a donor, and a city cannot report them either. Our city-scope filter passes 328 of the 377, so it is far too weak to catch that class. That is a real gap in the pipeline, not a finding about the data.

The screening criterion should change too: screen.py asks does the US report this? when the comparator is now the world. The right question is do enough countries report it?

Mon Sep 14 · Devin · permalink

Fixed the scope filter and re-keyed screening: 67 indicators the US doesn't report, and NYC can

Two gaps closed, both measured rather than assumed.

The scope filter was letting nearly everything through

The keyword blocklist was supposed to exclude indicators no city can report. Against 50 hand-labelled cases it scored precision 0.69 — passing 11 of 26 nation-only indicators. The failure was structural: the class it had to catch is development finance, spelled a hundred ways ("gross receipts by developing countries of official non-concessional sustainable development grants"), and no word list covers that.

Replaced with an embedding classifier that compares each indicator against prototype descriptions of what a city measures versus what only a sovereign state has.

Same perfect recall, false positives more than halved. The threshold comes from a sweep, not taste: a false negative drops a real city indicator forever and invisibly, while a false positive only adds a row to a shortlist a human is already reading. Asymmetric costs, asymmetric threshold.

The 50 labels live in probe/scope_eval.json and are explicitly a judgment call — argue with one by editing the file and re-running python3 probe/scope.py --eval.

Screening now asks the right question

screen.py asked does the United States report this? That was right when the US was the comparator. It is now every reporting country, so the question is do enough countries report this? — measured against a six-country panel spanning income levels and regions.

442 usable, of which 130 are indicators the US does not report — invisible to the old screen. After the scope filter, 67 are genuinely city-scoped. That is the real unlock, and it is smaller than the raw 130 while being far more real than the 328 the keyword filter would have waved through.

RANK-ONLY is a new grade for single-observation indicators. The old grading collapsed those into RED, which is how we wrote off road deaths before discovering 195 countries report it.

What the 67 actually are

Mostly waste and water — and that is a finding about the United States, not about NYC:

• Municipal waste collected · Total waste generation
• Hazardous waste generated, treated, exported, imported
• Total wastewater generated · groundwater quality · human-made wetlands
• Land degradation

The US does not report municipal waste collected to the UN. Dozens of countries do, and NYC has DSNY tonnage updated monthly. A city can be compared internationally on exactly the indicators its own country skips. That inverts the usual assumption about who has the data, and it is the second time this project has found the gap on the UN/national side rather than the city side.

Two honest caveats

A panel of twelve countries was my first attempt and it was wrong — the response cap applies to variables × entities together, so every batch overflowed and split down to single variables. Correct but needlessly slow. Six countries runs clean with zero short responses, which is the signal that nothing is being silently truncated.

And the classifier is not clean at 0.83: "Total inbound official flows for infrastructure" and "International financial flows to developing countries" both survive into the 67. That is the expected error rate showing up exactly where predicted, and it is why the output stays a shortlist for human review.

Mon Sep 14 · Devin · permalink

A fifth card, and the caveat runs the other way

Added municipal waste collected — the indicator the United States does not report to the UN at all. EN_MWT_COLLV has zero US observations in every year; ninety other countries report it.

So this is the case where a city can be placed among the world because its own country is absent from the table. There is no US line on the chart, and that is not a gap in our pipeline.

NYC: 397.9 kg per person (2019), 41st of 91 — between Hungary and Belarus. Its own series runs from 453.6 kg in 2005 to 387.6 in 2024, about 15% less waste per New Yorker over two decades.

Why this card earns its place

The caveat does not merely qualify the reading — it reverses it, and in the opposite direction to the recycling card two sections above.

DSNY collects from residences and institutions only; commercial waste goes to private carters and never appears. The UN definition includes commercial waste. So NYC's figure is an undercount, its true per-capita is higher, and its true rank is worse than 41st.

The same residential-only gap makes NYC look worse on the recycling chart and better here — because it sits in the numerator there and is simply missing here. One definitional difference, two opposite distortions, on one page. That is a stronger demonstration than either chart alone, and it is not a point we could have made before the world view existed.

An assumption stated rather than buried

DSNY publishes "Tons" and never says short or metric. US municipal practice is short tons, so that is what we assumed (×0.90718) — and the card says so, along with the consequence: if the figures are metric, every NYC number here is about 10% low.

An undocumented unit is not a reason to skip the comparison. It is a reason to put the assumption on the face of the chart where a reader can disagree with it.

What we did not add

Three other domains looked promising and were checked before anything was built:

• Suicide mortality — the NYC dataset carries two different ICD code strings across years, has no "All Sexes" aggregate, and switches the sex encoding (F / Female) mid-series.
• Broadband — a single snapshot with no year field, measuring household adoption share against the UN's subscriptions per 100 inhabitants.
• Electronic waste — thin coverage, and a second waste chart making the same point.

Three for three, the NYC side was the binding constraint. That has been the pattern since the first day of this project, and it is the opposite of what we expected going in.

Mon Sep 14 · Devin · permalink

The MCP server that refuses

The application promised three layers: a crosswalk, benchmark views, and an agent. Two were built. This is the third.

mcp/server.py composes the UN System Data Commons with NYC Open Data and serves the graded crosswalk over MCP. Stdlib only, no framework, no virtualenv:

claude mcp add nyc-un-benchmarks -- python3 /path/to/undatacommons-nyc/mcp/server.py

The point is that it refuses

Every other agent tool in that room will answer. This one returns a comparison only where a human graded the pair DIRECT or PROXY. For CONTEXT, BLOCKED and RANK-ONLY it declines and names the definitional difference:

benchmark("child mortality")
  → refused: true, grade: CONTEXT
    "AGE BANDS DO NOT MATCH. The UN counts deaths under five; NYC's series counts
     deaths under one. Both are raw counts so the units agree — which is the trap."
    what_is_possible: "Show the two series side by side, never on one axis."

The refusal carries guidance: "This is a refusal, not an error. Report the reason to the user; do not route around it by fetching the sides separately." — because an agent's instinct on being blocked is to find another way, and here the block is the answer.

That makes a trust & QA gate a tool contract rather than a policy document. It is the afternoon's topic arriving as working software.

Three behaviours worth stealing

Ambiguous names return candidates, not a guess. "municipal waste" matches two mapped indicators. The server says so and asks you to choose. Silently picking one is exactly the quiet guess this project exists not to make.

A missing comparator is reported as a finding. benchmark("municipal-waste") refuses because the UN holds zero US observations — and points at world_position, since 90 other countries do report it. Not a gap in the tool; the declared comparator reports nothing.

Derivations are data the server executes, not prose it ignores. The municipal-waste pair declares its short-ton assumption and per-capita conversion in crosswalk.json, and the server runs it. The demo and the server now produce identical figures — 397.9 kg per capita, 41st of 91 — which is the check that they have not drifted.

Four bugs found by building it

The server is an interface over work we had already verified, and it still surfaced four errors:

• The crosswalk had 11 pairs, not 12. Municipal waste existed only in the demo; the briefing already claimed 12. Now in crosswalk.json, which is the source of truth.
• benchmark("road deaths") matched nothing — "road deaths" is a substring of neither road-deaths nor "Road traffic deaths". Now matched on word overlap.
• The municipal-waste SoQL returned nothing at all: sum(a + b + c) is NULL whenever any column is NULL, and several DSNY streams did not exist in early years. Fixed with coalesce.
• Per-capita derivation existed only in the demo script, so the server would have served raw tons against country tonnes. Implemented properly rather than left as prose.

python3 mcp/smoke.py now checks every tool and asserts the figures against the demo's published numbers. 8/8.

Mon Sep 14 · Devin · permalink

Multi-city: the plumbing generalises, the matching does not

Generalised the pipeline to any city, over Socrata or CKAN, and ran it against five. The infrastructure works. The recommender does not travel, and that is the useful result.

Cross-language matching fails outright

Not degradation. Madrid tops out at 0.265 and Milan at 0.294 against Boston's 0.575, with medians around 0.15 — noise. Milan has a larger catalog than NYC and matches nothing.

Lowering the threshold would admit garbage rather than signal: Madrid's best single match pairs carbon dioxide emissions per unit of GDP with municipal parking permit lists. The embedding model is English-only and no amount of tuning fixes that.

Since most VLR cities are not anglophone, this is the blocker for the whole multi-city idea.

We were wrong about field names

Last note predicted CKAN cities would be handicapped because CKAN does not publish column names and Socrata does — columns being what moved NYC's waste pair from rank 882 to 23.

Chicago publishes field names on 915 of 915 datasets and found fewer candidates than Boston, which publishes none. Hypothesis dead. Recording it because a prediction that survives only until it is tested is worth more written down than quietly dropped.

The threshold does not transfer between catalogs

0.50 was calibrated on NYC's 2,400-dataset catalog. In a smaller catalog the nearest neighbour is whatever is least unrelated, so the same number now admits nonsense — Boston's top match pairs Food waste with Trash Schedules by Address at 0.57.

Reading all 13 English-city candidates by hand: roughly 2–3 per city are plausible, and no substantive SDG indicator matched in any city. No air quality, no homicide, no waste tonnage, no road deaths. What matched was generic budget, land and performance vocabulary sharing words with indicator names.

An absolute cosine score has no fixed meaning across catalogs. It needs to be a percentile, or a margin over that catalog's own distribution.

One constraint that fell out of the spec

The bootstrapper cannot emit a crosswalk. Spec v0.1 makes a grade a human judgment, so a machine producing one would violate the spec this project just published. Output is candidates.json with is_crosswalk: false — the worksheet a local analyst grades.

That constraint was not designed in; it fell out of the spec, which is a sign the spec is doing real work rather than describing what we already did.

Where that leaves it

NYC's good pairs were found by a person who knew the data, with the matcher confirming them. Nothing here shows the matcher can lead. Fixes, in order: a multilingual model, per-catalog threshold calibration, and demoting the matcher to a hint behind human search.

Until the first two, this is sound infrastructure with a known-poor recommender attached — and saying so now is cheaper than a city discovering it.

Mon Sep 14 · Devin · permalink

Multilingual embeddings: Madrid 0 → 19, Milan 0 → 5, and a trade-off we kept

The blocker for every non-anglophone city was the embedding model, not the pipeline. Adding potion-multilingual-128M unblocks them — and measuring it showed the fix is not free.

Why it is not a straight swap

On the seven hand-verified NYC pairs, against the full 2,400-dataset catalog:

Swapping wholesale would have bought non-English coverage with English accuracy — a 3.5× worse median rank on the catalog we know best. Interestingly it is not uniform: the multilingual model improves the two hardest cases (homicide 786→401, slums 1033→716) while degrading the easy ones.

So the model is chosen per catalog language. English cities keep the English model; everything else gets the multilingual one. The vector cache is keyed by model, because vectors from two models are not interchangeable and reusing them across would be a silent, invisible corruption.

The matches, read honestly

Milan is good — 4 of 5 plausible:

• Annual inflation (consumer prices) → Tasso di inflazione mensile (0.60)
• Installed renewable electricity capacity → Produzione netta di energia elettrica (0.67)
• Energy intensity of primary energy → Energia elettrica erogata da A2A (0.60)

Madrid is noisy — six near-identical "water area of lakes and rivers" indicators all matched Fuentes de agua para mascotas, drinking fountains for pets. The word agua dominated. But buried in the noise: installed renewable capacity → Inventario de instalaciones fotovoltaicas, which is exactly right.

The threshold problem is language-independent: 0.50 was calibrated on NYC and means something different in every other catalog. That remains the open issue, and it is now the main thing standing between this and a usable recommender.

Buenos Aires is unreachable, not empty

The portal closes the connection on anonymous package_search — RemoteDisconnected, on two attempts a day apart. Recorded in the registry as status: unreachable rather than as zero candidates, because those are different findings and collapsing them would overstate what we tested.