Source Verify

Source Verify

Overview

The mandatory checkpoint every evidence record passes through before it can be cited by any finding (constitution rule 6). No source skill verifies its own output — the actor that collects evidence never grades it. This skill's only job: turn verification_status: pending into verified, rejected, or unreachable, honestly, one record at a time.

Layer rule: this is where network-dependent verification happens (constitution rule 8) — archive liveness checks, re-fetching URLs, confirming a page still says what the record claims. scripts/validate_run.py never touches the network; everything network-dependent about "is this evidence real" happens here, once, before the validator ever runs.

The two checks (constitution rule 6)

Every pending record gets both checks, in order. Both must pass for verified.

Check (a) — Mechanical: does the quote exist?

Fetch source.url. Search for the evidence.quote text, whitespace-normalized (collapse multiple spaces/newlines, ignore leading/trailing whitespace) but otherwise exact — no fuzzy matching, no "close enough."

Multi-part quotes. A quote trimmed with [...] is split on that marker into parts. Every part must appear at the source, in the same order, with no more than ~2000 characters between consecutive parts. Without the order and distance constraints, "exact quote" means something different in every implementation, and a quote could be assembled from fragments scattered across an entire page.

Fall through to the archive on either failure mode, not just one:

1. Live URL loads and contains the quote → check (a) passes, checked_via: live.
2. Live URL loads but the quote is not there → try the archive anyway. A page edited after collection is the normal case here, not an exception: the evidence was real when collected, and the snapshot is what proves it. Found in the archive → passes, checked_via: archive, and limitations records that the live page has since changed. Treating a content-drifted page as a rejection would discard true evidence and quietly punish sources that update.
3. Live URL fails to load → try the archive. Found → passes, checked_via: archive.
4. Not found in either → rejected (mechanical: not-found). Do not proceed to check (b).
5. Neither loads at all → unreachable (mechanical: unreachable), not rejected — an availability failure is not evidence the claim is false.

archive-failed: is not a URL. When archive_url starts with archive-failed:, there is no archive to try — skip straight to the outcome the live check produced. Never issue a fetch against that marker.

Check (b) — Semantic: does the quote, in context, support the claim?

Read the passage surrounding the quote — not just the quote in isolation. Ask: standing in its actual context, does this passage assert what claim says, at the scope claim states (same version, region, date range, magnitude)?

How much context. "Surrounding" needs a definition or the same record verifies differently on different days. Start with the quote's own paragraph, expand to the paragraph before and after, and expand once more to the enclosing section only if the claim is still ambiguous. Per source type: HTML → paragraph, then section; PDF → paragraph, then page; GitHub code → the enclosing function or block, then the file's surrounding context; GitHub issues and community threads → the parent comment plus the thread's opening post, since a reply's meaning frequently lives in what it is replying to.

• Full support → check (b) passes → verified.
• No support (context contradicts, or the quote was cherry-picked to imply something the passage doesn't actually say) → rejected.
• Partial support (the passage supports part of the claim but not all of it — a narrower scope, a smaller number, a qualified rather than absolute statement) → not a rejection. Run the shrink procedure below.
• The passage itself attributes the claim to another source ("according to X", "as reported by", wire-service framing, a citation to a paper) → do not verify this record as-is. Route back to engine/origin-tracing.md to resolve the actual root, then re-run both checks against the root's own record. The original record, if kept at all, is marked with root_origin pointing at the resolved root and is not independently verified as if it originated the claim.

Shrink → re-verify (constitution rule 6, "partial is not a status")

When check (b) finds partial support:

1. Rewrite claim to state exactly what the quote supports — no more. ("Revenue grew 40% globally" → "Revenue grew 40% in the enterprise segment" if that's what the passage actually says.)
2. Append to limitations: "claim narrowed from: <original claim text>" — the narrowing is never silent.
3. Increment claim_revision. This is what makes the narrowing visible downstream: any finding citing this record must carry a matching acknowledged_revisions entry, or validation fails (engine/synthesis-contract.md). Without this, a finding written against the original broad claim would silently outlive the evidence that no longer supports it — the validator cannot compare a finding's sentence to a claim semantically, so the acknowledgment is what forces a human/synthesizer to look again.
4. Re-run check (b) against the new claim. It should now pass (the claim was built from what the quote supports) — if it somehow still doesn't, treat as rejected rather than looping.
5. Set verification_status: verified on the narrowed record.

This record is never discarded for being narrower than first collected — a correctly-scoped verified record is more valuable than a broad rejected one. Log the narrowing in the run ledger too, so the synthesizer knows which findings need re-examination.

Procedure

1. Pull every pending record from the run's evidence.jsonl.
2. For each: run check (a), then check (b) if (a) passed. Apply shrink if partial. Apply the origin-tracing routing if the quote is itself a relay.
3. Write the resolved verification_status back to the record, together with the verification block (mechanical, semantic, checked_via, checked_at). Recording only the final status loses the two things an auditor most wants later: which of the two checks decided it, and whether the quote was confirmed against the live source or an archived snapshot. This station is the only one allowed to write either field after collection.
4. Never delete a rejected or unreachable record. They stay in evidence.jsonl as a permanent trail of what was checked and why it didn't hold up — rejected evidence simply becomes uncitable (engine/synthesis-contract.md: no finding may cite it).
5. Log a short verification summary per run in the ledger: counts of verified/rejected/unreachable/narrowed, and which records were rerouted to origin-tracing.

What this skill does not do

• Does not decide finding status (confirmed/probable/contested/insufficient) — that ladder is computed only by scripts/validate_run.py, deterministically, from verified evidence. This skill's output (verification_status) is one of that computation's inputs, nothing more.
• Does not synthesize, compare across records, or write findings.
• Does not re-collect evidence — if a claim needs different/better evidence, that goes back to the relevant source skill, not to this station.

Common mistakes