Adversarial Data Review

Adversarial data review

Attack the dataset the way a hostile referee would: treat every metadata sentence and every data value as a claim that must survive verification against (a) the producer's own documentation and (b) independent sources — because the producer itself can be wrong.

Two failure classes to hunt:

1. Our mistakes — misread units, metadata content in the wrong field, stale producer text, scope overclaims, processing bugs that ship wrong numbers.
2. The source's mistakes — unit slips, wrong-year values, transcription errors, stale pre-revision values. These are hard to detect precisely because our pipeline faithfully reproduces them; only an independent source can expose them.

Scope — what this skill is NOT

Factual accuracy only. Don't duplicate:

• Style-guide compliance → /check-metadata-style
• Spelling typos → /check-metadata-typos
• Jinja whitespace artifacts → /check-metadata-spacing
• Field-coverage / freshness / link-liveness audits → /update-dataset § 6c (that step checks the links resolve; this skill reads what's behind them)

Any rewrites you propose use American spelling.

Inputs

• A step path garden/<namespace>/<version>/<short_name> (the data:// URI form also works), a bare <short_name> — resolve via the DAG like /update-dataset does (rg "/<short_name>:?$" dag/ -g "*.yml" | grep -v "^dag/archive", latest active version, ask the user if ambiguous).
• Optional: --top N (deep-review cap, default 10) or --full (deep-review every indicator regardless of chart usage).
• Also accepts a single chart (slug or id) as the target: scope the whole review to that chart's indicator(s) — verify every displayed country-category/value for the latest year, compare against the previous dataset version, and review the chart's own FAUST text as claims. Drafts count (find them in the staging DB; they have no production row).
• Precondition: the garden dataset is built locally (.venv/bin/etlr data://garden/<ns>/<version>/<short_name>; PREFER_DOWNLOAD=1 is fine for already-published upstream deps). Exception: the /create-snapshot context runs Phase 0 only against the .dvc and the fetched docs — no meadow/garden step exists yet, so skip this precondition (and Steps 1, 3–5) there.

Scope by calling context:

In most workflow skills this review is an optional, offered step (it can consume many tokens — see the estimates in Step 1), except where the table marks it mandatory; when invoked from one of them, scope accordingly:

Step 1 — Prioritize indicators by chart views (heaviness control)

The deep per-indicator work (metadata claim review + online cross-checks) costs real time (~25–45 web calls on a typical dataset), so scope it: deep-review the top N indicators (default 10) ranked by summed 365-day views of the charts that use them, plus every indicator the anomaly scan flags. For a brand-new dataset with no charts, review all indicators.

Never cap silently. The report's Part 2 must list every skipped indicator with its rank — a truncated review that reads as complete is itself a factual error.

Views rank prioritization; the DB defines coverage. Analytics only sees published charts — when the task is "all charted indicators" (or the chart under review is a draft), inventory usage from the grapher/staging DB instead (chart_dimensions JOIN variables on catalogPath, no publishedAt filter), then rank the published subset by views.

Map the ranking onto the NEW build before selecting. Pre-merge, the ranked catalog_paths are the old version's — extract a version-independent identity (<table>#<column>, i.e. everything after the version segment) and join it onto the new garden/grapher build's actual table/column list. Old-charted identities missing from the new build are renames — resolve them via the indicator-upgrade mapping before ranking, or list them explicitly. New-build indicators absent from the ranking are uncharted (including newly added ones) — they are candidates for the anomaly-driven track and must appear in the reviewed-or-SKIPPED inventory, never silently dropped by the chart join.

from etl.analytics.config import SEMANTIC_LAYER_SCHEMA as S  # tables must be schema-qualified for BigQuery
from etl.analytics.data import read_analytics  # Metabase; auto-falls back to analytics Datasette without creds

NS, SHORT = "<namespace>", "<short_name>"
ind = read_analytics(f"SELECT indicator_id, catalog_path FROM {S}.indicators WHERE catalog_path IS NOT NULL")
# Match namespace + short_name across ANY version — before the PR merges, live charts still point at the OLD version.
ind = ind[ind["catalog_path"].str.contains(f"/{NS}/") & ind["catalog_path"].str.contains(f"/{SHORT}/")]
cxi = read_analytics(f"SELECT chart_slug, indicator_id FROM {S}.charts_x_indicators")
charts = read_analytics(f"SELECT chart_slug, views_365d FROM {S}.charts").drop_duplicates("chart_slug")
usage = (
    # LEFT-join the views: a newly published or unvisited chart has no analytics row yet, and an
    # inner join would silently drop its indicators from the charted inventory (they rank at 0 instead).
    ind.merge(cxi, on="indicator_id").merge(charts, on="chart_slug", how="left")
    .groupby("catalog_path")
    .agg(n_charts=("chart_slug", "nunique"), views_365d=("views_365d", "sum"))
    .sort_values("views_365d", ascending=False)
)

If the analytics layer is entirely unreachable, fall back to the public grapher Datasette and rank by number of charts (it has no view counts):

from etl.http import session  # OWID infra → tagged User-Agent

sql = (
    # NOTE: variables.catalogPath DOES carry the channel prefix ("grapher/<ns>/<version>/<short>/<table>#<col>");
    # it's datasets.catalogPath that is channel-less — don't mix the two conventions up.
    "SELECT v.catalogPath, COUNT(DISTINCT cd.chartId) AS n_charts "
    "FROM variables v JOIN chart_dimensions cd ON cd.variableId = v.id "
    "WHERE v.catalogPath LIKE 'grapher/<namespace>/%/<short_name>/%' GROUP BY 1 ORDER BY 2 DESC"
)
rows = session.get("https://datasette-public.owid.io/owid.json", params={"sql": sql}).json()["rows"]

Step 2 — Phase 0: source verification (mandatory before any critique)

Establish what you're looking at before criticizing anything. Investigative, not adversarial.

1. Collect the source's links. From the snapshot .dvc (url_main, url_download, license.url) and every URL embedded in metadata texts and step files:

   rg --no-filename -No "https?://[^\"' )>]+" snapshots/<ns>/<version>/ etl/steps/data/{meadow,garden,grapher}/<ns>/<version>/
   

2. Compare the source's file-modification dates/hashes against our snapshot's date_accessed/md5 (e.g. the OSF API lists date_modified and hashes per file). Producers replace files in place without bumping version labels — an unchanged version string proves nothing, and an in-place revision is the single highest-yield thing this phase can find. While at it, diff the source's file inventory against what we snapshot: a new companion file (pre-built index, summary table, construction script) is a new-indicator candidate that no within-file diff can surface — route it to the update workflow's "Surface new indicators" step rather than just cataloguing it.
3. Fetch and READ the producer's own documentation — methodology pages, indicator definitions, codebooks/data dictionaries, release notes. Not secondary commentary, not a blog post about the source: the source itself. Follow the links from url_main to the actual methodology document when the landing page is thin. Access escalation per repo convention: curl → WebFetch → Wayback Machine before treating a 4xx as real.
4. Check for an existing <short_name>.corrections.yml next to the garden step. Its entries are known, already-handled source errors — acknowledge them in Part 0 of the report; never re-flag them as new findings.
5. Establish the pipeline. For each metric under review: what does the source publish (exact indicator name, unit, definition, granularity, upstream data — official statistics, modeled, survey)? What does OWID add on top (read description_processing, the garden step code, and the corrections file)? Every later critique must state whose layer it targets. While reading, harvest the codebook's worked examples as test vectors: any country/value/date the documentation itself cites must match the data — a codebook example contradicting the shipped file is the strongest class of source error (provable entirely from the producer's own materials).
6. Field-placement audit. The .dvc meta.origin.description must carry producer content only; garden description_processing must carry OWID content only; description_from_producer must be verbatim producer text — diff it against the fetched docs (typography-only drift is fine, paraphrase is not). Beyond placement, each field must be factually consistent with what the docs actually say (units, scope, coverage, method).

HARD RULE — proportionality. The severity of any provenance or factual critique must be proportional to the depth of verification you achieved. If you read the documentation and confirmed a gap, make a strong claim. If the docs were unreachable after the full escalation, cap the language at "I was unable to verify … — worth checking before merge" and the severity at 🟢. Never assert an error you couldn't check.

Step 3 — Phase 1: internal anomaly scan (local, cheap — runs before any web call)

Scan the built garden dataset for internal red flags. Paste-and-adapt sketch — adjust the key columns (extra dimensions like sex/age need to join the groupby keys), the thresholds, and which units are additive:

from owid.catalog import Dataset
import numpy as np
import pandas as pd

ds = Dataset("data/garden/<ns>/<version>/<short_name>")
findings = []
for tname in ds.table_names:
    tb = ds.read(tname, safe_types=False)  # read() resets the index, so key columns are regular columns
    if tb.index.names != [None]:  # defensive: if keys sit in the index (e.g. the table came via ds[tname]), restore them
        tb = tb.reset_index()
    year_col = "year" if "year" in tb.columns else ("date" if "date" in tb.columns else None)
    has_country = "country" in tb.columns  # year-only tables exist (gravitational-wave counts), as do static ones (GWP factors)
    vals = [c for c in tb.columns if c not in ("country", year_col) and pd.api.types.is_numeric_dtype(tb[c])]
    for col in vals:
        unit = (tb[col].metadata.unit or "").lower()
        s = tb[(["country"] if has_country else []) + ([year_col] if year_col else []) + [col]].dropna(subset=[col])
        if s.empty:
            findings.append((tname, col, "EMPTY", "all-NaN column")); continue
        mx, mn = s[col].max(), s[col].min()
        # 1. Unit/magnitude sniffs
        if any(w in unit for w in ("%", "percent", "share")):
            if mx <= 1.5: findings.append((tname, col, "UNIT", f"unit is % but max={mx:.3g} — fraction stored?"))
            if mx > 150: findings.append((tname, col, "UNIT", f"% column max={mx:.3g} — can it exceed 100?"))
        if mn < 0 and any(w in unit for w in ("people", "number", "deaths", "tonnes", "count")):
            findings.append((tname, col, "SIGN", f"negative min={mn:.3g} in count-like unit"))
        if not has_country or year_col is None:
            continue  # year-only or static table: only the unit/magnitude sniffs apply; checks 2-6 need country + time
        # 2. Robust per-country outliers (median/MAD z-score; skip short series — MAD is unstable under ~8 points)
        g = s.groupby("country")[col]
        mad = g.transform(lambda x: (x - x.median()).abs().median()).replace(0, np.nan)
        z = ((s[col] - g.transform("median")) / mad).where(g.transform("count") >= 8)
        for _, r in s[z.abs() > 6].head(20).iterrows():
            findings.append((tname, col, "OUTLIER", f"{r['country']} {r[year_col]}: {r[col]:.4g} (|z|>6)"))
        # 3. Trend breaks with a unit-error signature (~×10/×100/×1000 jumps)
        ss = s.sort_values(["country", year_col])
        ratio = ss.groupby("country")[col].pct_change().add(1).abs()
        for _, r in ss[np.log10(ratio.replace(0, np.nan)).abs() >= 1].head(20).iterrows():
            findings.append((tname, col, "BREAK", f"{r['country']} {r[year_col]}: ≥×10 year-over-year jump"))
        # 4. Coverage drop in the most recent period
        cov = s.groupby(year_col)["country"].nunique()
        if len(cov) > 1 and cov.iloc[-1] < 0.7 * cov.iloc[-2]:
            findings.append((tname, col, "COVERAGE", f"{cov.index[-1]}: {cov.iloc[-1]} countries vs {cov.iloc[-2]}"))
        # 5. Suspicious constants: long runs of identical NON-ZERO values (source forward-fill?) —
        #    repeated zeroes are normal for sparse count/event indicators and must not count.
        runs = ss.groupby("country")[col].apply(lambda x: ((x == x.shift()) & (x != 0)).mean())
        for ctry in runs[runs > 0.5].index[:10]:
            findings.append((tname, col, "CONSTANT", f"{ctry}: >50% of series identical to previous period"))
        # 6. World vs sum of countries — ONLY for additive units (counts, tonnes, deaths; never rates/shares/indices)
        #    flag when |World − Σ countries| / World > 5% in a spot-checked year

The scan is a candidate generator, not a verdict. Review the raw findings yourself and discard the obviously legitimate ones (wars, pandemics, currency redenominations, real policy shocks) with a stated reason each before spending any web calls in Phase 2.

Step 4 — Phase 2: independent online cross-check (anomaly-led + anchors)

This is the half that catches the source's mistakes — the ones invisible to every local check because our pipeline reproduces them faithfully.

What to check:

• Every anomaly that survived your Phase-1 triage. Cap the WebSearch effort at ~10 values; list anything beyond the cap as unchecked in Part 2.
• Fixed anchors, regardless of anomalies: the World total (if the dataset has one), 2–3 major or topic-relevant countries, the latest year, and one mid-series historical year.

Independence rules (anti-circularity — read before searching). An independent source is a different producer measuring the same quantity (WHO vs. IHME, IEA vs. Energy Institute, IMF vs. World Bank, UN WPP vs. a national statistics office), or the primary source the producer aggregates. Never count as independent: ourworldindata.org itself; sites that republish OWID (Wikipedia charts and infoboxes frequently cite us — check the citation); mirrors of the same producer (tradingeconomics and friends scrape WB/IMF); or the producer's own secondary pages.

Procedure per value: WebSearch the quantity + entity + year → open 1–2 authoritative hits with WebFetch → record source, value, and link in the Part 2 table. Never cite a number straight from the search-results summary — summaries blend several sources and lag living pages; every figure that reaches a finding, a PR body, or a producer question must be quoted from a page you actually opened (a stale search-summary count once shipped into a producer email as "21 of 32" when the opened page said 22, later 26).

Measurement-artifact scrutiny (per source, not per value): search "<producer> completeness bias", "<producer> coverage <region>", "<indicator> revision history" and read what comes back. When you flag a comparability problem, name the specific mechanism by which the data misleads (e.g. "death registration completeness below 60% in region X inflates apparent improvement"); a bare "comparisons should be made with care" is banned.

Internal accounting identities beat external sources. When a producer publishes components and their aggregate, check that they reconcile — a contradiction inside the producer's own release is arithmetic, needs no independent source, and can't be waved away as a methodology difference. It also survives the common case where every external source is bot-blocked. This is confirmation route (b) in the Tolerance gate below — it settles a source error on its own, provided the guards hold: every term of the identity comes from the same release and vintage (never mix an old download's share with a new download's rate), the terms are defined so the identity holds by construction (shares that sum to 1, urban+rural weighted by the same population split), and the contradiction is far beyond rounding. (WDI 2026-07: Chad's rural electricity access published as 100.0% for 2024, when the same release's rural population share of 73.1% and urban access of 51.0% imply a national rate of 86.8% against a published 13.4% — solving the identity for the rural term gives ≈0%, matching the 0.4% of the prior year. IEA, the AfDB portal, the Tracking SDG7 PDF and the UN SDG API were all unreachable; the identity settled it alone.) Two habits that make this reusable: sweep the identity across all entities, not just the suspicious one, to prove the error is isolated rather than systematic; and check the producer's footnote table (WDIfootnote.csv and equivalents) — an unqualified bad value is a stronger finding than a flagged one. One more guard before anything reaches the corrections route: the contradiction proves an error exists among the identity's terms — it does not by itself say which term is wrong. Identify the bad term with evidence beyond the identity: the entity's own adjacent years (in the Chad case the solved rural rate ≈0% matched the prior year's 0.4%, while the other terms sat in line with their own histories), the all-entity sweep isolating a single term, or a footnote. If nothing singles out one term, the finding is still a confirmed producer error — report it and hand it to producer follow-up, but don't guess which value to overwrite in corrections.yml.

A "no charts use this indicator" clearance goes stale the moment charts are remapped. Blast-radius checks are version-scoped; a chart stranded on an older version won't match a query filtered to the version you're updating, and will silently come into scope once the stranded-chart sweep runs. Re-run any such check after the indicator upgrade before relying on it. (Six corrections were approved partly on "zero published charts use these"; one was wrong by the time it mattered.)

Tolerance: rounding, vintage/revision drift, and methodology gaps of a few percent are not findings. The targets are magnitude errors (×10/×100/×1000), wrong-year values, sign errors, entity mix-ups, and stale pre-revision values. Declaring a confirmed source error requires one of two routes: (a) ≥2 independent sources that agree with each other and disagree with ours beyond methodology tolerance, or (b) an accounting-identity contradiction internal to the producer's own release (the paragraph above) — components and aggregate from the same release and vintage, defined so the identity must hold by construction, disagreeing far beyond rounding. Route (b) stands alone; it does not additionally need external sources.

Attribution before routing. Before routing any confirmed bad value, read the raw snapshot (from etl.snapshot import Snapshot; Snapshot("<ns>/<version>/<file>").read() — read() picks the reader from the file's format; use the format-specific read_csv/read_excel/read_json only when auto-detection needs overriding) to determine where it entered: present in the source file → source error (corrections route); absent → our processing introduced it (trace snapshot → meadow → garden and fix the step).

Step 5 — Phase 3: adversarial metadata review (top-N + anomalous indicators only)

Treat each prioritized indicator's user-facing text as a set of claims and attack them against the Phase-0 documentation:

• Does unit (and any display.unit/conversion) match the producer's stated unit? A (mils)/(000) marker in the source's column header demands a visible conversion in garden.
• Does the title / description_short overclaim scope — "global" when the source covers reporting countries only, "countries" when it's high-income countries?
• Does description_key state contested definitions as settled, or omit a caveat the producer's own docs (or your Phase-2 literature search) prominently state — coverage gaps, comparability breaks, denominator choices?
• Are causal or certainty words ("shows", "proves", "leads to", "drives") backed by the source's methodology, or do they smuggle in an interpretation?
• For categorical indicators built from label maps, list the distinct source labels and verify each maps explicitly — values routed to a fallback bucket ("unknown", "other") are silent misclassifications, because the fallback is an existing category and no validation fires. Recommend an observed labels ⊆ map keys assert where one is missing.
• For Jinja-templated metadata, spot-check the rendered text readers actually see: Dataset("data/grapher/<ns>/<version>/<short_name>").read(t, load_data=False)[col].metadata.

Lead with the concrete rewrite, not the objection. "Add a link" is a valid fix. Match the register and length of the original — prefer a word swap over an added clause.

Routing findings

Output report

Write to ai/adversarial-review-<short_name>-<YYYY-MM-DD>.md:

# Adversarial data review — <ns>/<version>/<short_name>

## TL;DR
(max 3 sentences: overall verdict + the one thing to fix first)

## Part 0 — Source verification
- Source(s) + documents accessed (URL, access status: read / paywalled / unreachable)
- What the source publishes (indicators, units, definitions, granularity, upstream data)
- What OWID adds (from description_processing + step code + corrections.yml)
- Field-placement check (.dvc description / description_processing / description_from_producer)
- Data-quality flags stated in the source's own docs
- Flags the docs SHOULD state but don't (from the artifact-literature search)
- Existing corrections.yml entries acknowledged

## Part 1 — Findings (numbered, ordered 🔴 → 🟡 → 🟢)
N. 🔴|🟡|🟢 [data-level|text-level] <one-line defect>
   Evidence: <quote or value + link>
   Fix: <concrete action + routing per the table above>
   Why: <one tight sentence>

## Part 2 — Cross-check appendix
- Values checked: indicator | entity | year | our value | independent value(s) | source link | verdict ✓/✗/~
  (mark each row as anchor or anomaly)
- SKIPPED indicators (name, views_365d rank, why skipped) — mandatory, never silent
- Anomalies beyond the WebSearch cap, listed as unchecked

Severity rubric (aligned with /review-data-pr): 🔴 = confirmed factual error (metadata contradicted by the producer's own docs, or a value confirmed wrong — via ≥2 independent sources or a same-release accounting-identity contradiction — with snapshot-level attribution); 🟡 = likely issue needing confirmation; 🟢 = informational or unverifiable.

In author flows, apply the 🔴 fixes immediately (they're why the skill ran before commit); leave 🟡/🟢 as report items for the user to triage.

Key constraints

• Always fetch and search — never judge data plausibility from memory or training data. The whole point is a fresh look at what the source and the wider literature actually say today.
• Establish source-vs-OWID attribution (Phase 0) before critiquing either layer; every finding names whose layer it targets.
• Severity proportional to verification depth — the HARD RULE in Step 2.
• Don't manufacture objections. A clean report is a valid outcome; if uncertain whether something is wrong, say "verify this" rather than asserting it.
• Separate data-level from text-level findings — a legitimate metric can sit under overclaiming text, and carefully hedged text can sit on top of broken data.
• Methodology differences are not errors. Name the specific mechanism before calling a mismatch an error.
• Factual accuracy only — no style/typo/spacing duplication (see Scope).
• corrections.yml override values on categorical columns must come from the source's current vocabulary — assigning a retired label fails with Cannot setitem on a Categorical; choose the current-vocabulary value that yields the same published output.
• No persistent files beyond the ai/ report — plus, in author flows, the metadata/corrections edits themselves. Ad-hoc analysis scripts run from the session and are not committed.