Source Evaluation

Source Evaluation

Use When

• Assess every candidate source before its claims enter the evidence corpus.

Do Not Use When

• Do not use as the final URL, quotation, statistic, or claim-link check; use source-verification after evaluation.

Required Inputs

Workflow

1. Load evidence-discipline and classify the source type.
2. Apply the credibility ladder and matching source rubric.
3. Separate source content from analyst inference and record contradictions.
4. Admit, qualify, or reject the source for each intended claim.
5. Route admitted claims to source-verification; stop on fabricated material and recover from untraceable claims by removing them or recording a gap.

Outputs

Evidence Produced

Capability Contract

Evaluation defaults to read-only. Network lookup may verify provenance when authorised; downloading restricted material, contacting sources, altering evidence, or publishing findings requires explicit authority.

Degraded Mode

If provenance, network, media-forensics tools, or corroboration are unavailable, mark the relevant checks unassessed and return the narrowest qualified assessment. Never upgrade an unassessed source.

Decision Rules

Quality Standards

Every admission decision is claim-specific, traceable, explicit about limitations, and independent of whether the source supports the desired conclusion.

Anti-Patterns

• Scoring a source without its intended claim. Fix: evaluate claim fit.
• Equating official with infallible. Fix: test incentives and method.
• Hiding credible contradiction. Fix: preserve it.
• Treating popularity as authority. Fix: assess provenance.
• Filling missing metadata from memory. Fix: mark a gap.

Worked Example

For a statistic quoted by a news article, locate the originating dataset or publication, evaluate that source for the precise statistic, record any definition mismatch, and send the claim-source pair to verification.

References

• Freshness and uncertainty
• Evidence discipline
• Credibility ladder
• Primary-source doubt
• Media evaluation
• Misinformation and bias
• Media forensics
• Automated triage

Freshness and uncertainty gate

Before admitting a material source, classify the intended claim as stable, context-bound, time-sensitive, partial/early-release, or unusable using references/freshness-and-uncertainty.md. Record the source status, completeness, access date, and next verification trigger in the source register. A historical source may support a historical claim but must not silently support a current command, law, product capability, benchmark, price, safety instruction, or platform fact. When evidence is incomplete, narrow the wording and mark the claim unassessed, partial, or historical-only; never fill a gap from memory.

Research improvement handoff

After a research wave, send source defects, duplication, verification failures, access gaps, and analyst toil to the Kaizen research loop in ../research-orchestration/references/kaizen-research-loop.md. The experiment must preserve source immutability and use a guardrail such as verification coverage, source independence, or unresolved-claim count.

Single entry skill for vetting any source the engine consults. Plagiarism prevention is in academic-writing; this skill governs whether a source is trustworthy in the first place and whether claims drawn from it have hallucinated detail.

The non-negotiable guardrail (load first)

Always load references/evidence-discipline.md. The engine's first rule: no claim, statistic, quote, name, case, statute, URL, date, or place appears in output unless traceable to a real source. This applies to every output of every skill.

Source-type router — load the matching reference

The 5-tier credibility ladder (universal)

Detail in references/credibility-ladder.md. Quick reference:

Rule: any tier-5 claim must be triangulated against ≥2 tier-1 to tier-3 sources before it appears as fact. Tier-4 claims need ≥1 tier-1 to tier-3 source for any load-bearing assertion.

Burke's five-term source-doubt pentad (primary documents)

Detail in references/burke-five-term-doubt.md. For every primary document, ask:

1. Author — who wrote it; what was their position, motive, knowledge?
2. Provenance — chain of custody from creation to your hands; gaps; tampering opportunities.
3. Production — when, how, under what conditions; first or revised; original or copy.
4. Mechanics — language, format, conventions of the era; do they match?
5. Aims — what was the document trying to do; who was the intended audience?

A document that survives the pentad is evidence with known limits. A document that hasn't faced it is evidence with unknown limits, which is dangerously close to hallucination input.

Tudor's twelve-point evaluation (media / journalism / analyst)

Detail in references/tudor-twelve-points.md. Twelve criteria:

1. Recency · 2. Relevancy · 3. Authority · 4. Completeness · 5. Accuracy · 6. Clarity · 7. Verifiability · 8. Statistical validity · 9. Internal consistency · 10. External consistency · 11. Context · 12. Comparative quality

Apply when a media or analyst source carries a load-bearing claim. The audit produces a one-line verdict that the engine logs in the manifest.

Misinformation and bias checks (viral / politicized / contested claims)

Detail in references/misinformation-and-bias-checks.md. Before a contested or viral claim enters the corpus:

1. Separate fact, opinion, and interpretation.
2. Check the claim against the original source, not just reposts or reactions.
3. Identify the likely biases in the source and in the analyst.
4. Treat virality as a warning sign, not corroboration.
5. Downgrade any claim whose chain leads only to commentary, screenshots, or clipped excerpts.

Silverman / Bellingcat media-forensics (images, video, documents)

Detail in references/silverman-media-forensics.md. Workflow:

1. EXIF extraction — camera, GPS, timestamp.
2. Reverse image search — Google Lens, Yandex, TinEye, Bing.
3. Geolocation by landmark / shadow-time computation.
4. Archive recovery — Wayback, archive.today, original posting site.
5. Provenance tracing — who first published; cross-platform trail.
6. Tampering detection — error-level analysis, metadata anomalies.

Output: provenance chain logged with timestamps, URLs, and confidence.

Universal output rule

Every cited claim in engine output carries:

• Source reference (URL, DOI, archive link, document ID).
• Source tier (1–5 from the credibility ladder).
• Verification trail (Burke / Tudor / Silverman audit if applicable, otherwise "credibility-ladder" only).
• Confidence (high / medium / low).
• Date accessed (UTC).

If any of those is missing, the claim does not ship.

If automated reliability scoring was used, the output must also expose the axis-level breakdown and human-review flags from references/automated-reliability-triage.md. A composite score alone is not evidence.

Universal anti-patterns

• Trusting a tier-5 claim without triangulation.
• Treating an automated reliability score as a source verdict instead of a triage signal.
• "Studies show" without footnote.
• Citing a stat older than 3 years without flagging.
• Single-source paragraph on contested terrain.
• Image used as evidence with no reverse-image-search trail.
• Wikipedia footnote chasing skipped — citing the article instead of its source.
• Author of a document never named, motive never considered.
• AI-generated source list passed through unchecked — must be 100 % verified before any output.
• Quote attribution to a person without locating the original utterance.
• URL cited without archival snapshot — link rot kills evidentiary trails.
• Mixing source-tier verdicts in one paragraph without flagging the weakest.

Universal ship gate

For book-informed work, apply book-driven source admission and currentness before admitting any time-sensitive claim.

• evidence-discipline.md audit run on every claim.
• Every source assigned a tier (1–5).
• Every primary document has Burke pentad notes.
• Every media / journalism source has at least a 12-point quick verdict.
• Every load-bearing image / video has a Silverman provenance trail.
• Every tier-5 claim triangulated against ≥2 tier-1 to tier-3 sources.
• No URL ships without an archive.org or archive.today snapshot.
• Manifest entry per source: {ref, tier, verification, confidence, accessed}.
• Hallucination check: any claim absent from the source list is removed.

Companion skills

• data-quality-assessment — for source-vetting at the dataset level (Walker four axes).
• web-scraping-foundations — when you collect the source yourself.
• osint-investigation — when sources sit inside a broader investigation.
• academic-writing — when sources feed an academic artifact (citation styles in references).
• analytic-tradecraft — when source disputes become estimative judgments.