Evidence Audit

Evidence Audit

Falsificationism and evidence quality assessment for auditing whether claims are actually supported by evidence.

Core Principle

Karl Popper's key insight: a claim is only meaningful if you can specify what would disprove it. The Evidence Audit mode extracts claims from proposals, designs falsification criteria, assesses evidence quality, and surfaces competing explanations. The goal is not to disprove — it is to determine whether the evidence actually supports the conclusion.

Process

1. Extract claims — Identify the specific claims being made
2. Design falsification criteria — For each claim, specify what would disprove it
3. Assess evidence quality — Evaluate the evidence supporting each claim
4. Identify cognitive biases — Check for systematic errors in reasoning
5. Surface competing explanations — Find alternative explanations for the same evidence

Claim Extraction

Proposals contain claims — often implicit. Extract them before evaluating.

Types of Claims

Extraction Method

For each statement in the proposal:

1. Is this a claim or a definition?
2. If a claim, what type?
3. What evidence is cited (or implied)?
4. What would make this claim false?

Example Extraction

Statement: "Based on our pilot, migrating to Kubernetes will reduce deployment time by 60%."

Claims extracted:
1. The pilot results are representative of production (Predictive)
2. Kubernetes is the cause of the deployment time reduction (Causal)
3. The 60% reduction will persist at scale (Quantitative)

Falsification Criteria

For each claim, design a test that would disprove it.

Unfalsifiable Claims (Red Flag)

Some claims cannot be falsified. These are red flags.

When you encounter unfalsifiable claims, ask: "What specific, measurable outcome would tell us this worked or didn't work?"

Evidence Quality Assessment

Not all evidence is equal. Assess each piece of evidence on these dimensions.

Evidence Quality Matrix

Evidence Grading Scale

Common Weak Evidence Patterns

Cognitive Bias Awareness

Check for these biases in the reasoning chain.

Competing Explanations (Abductive Reasoning)

For every conclusion, ask: "What else could explain this evidence?"

Method

1. State the evidence
2. State the proposed explanation
3. Generate 2-3 alternative explanations
4. Compare explanatory power

Example

Evidence: "Deployment failures dropped 50% after adopting tool X."

Proposed explanation: Tool X is better than the old tool.

Alternative explanations:
1. The team also started doing more code review in the same period
2. A particularly error-prone service was retired last month
3. The team gained experience that would have improved results with any tool

Output Template

## Evidence Audit: [Proposal/Decision]

### Claims Extracted

| # | Claim | Type | Evidence Cited |
|---|-------|------|---------------|
| 1 | [Specific claim] | Causal/Predictive/etc. | [What evidence supports it] |
| 2 | [Specific claim] | Causal/Predictive/etc. | [What evidence supports it] |
| 3 | [Specific claim] | Causal/Predictive/etc. | [What evidence supports it] |

### Falsification Criteria

| Claim | What Would Disprove It | How to Test |
|-------|----------------------|-------------|
| #1 | [Specific criterion] | [Concrete test] |
| #2 | [Specific criterion] | [Concrete test] |

### Evidence Quality

| Claim | Evidence Grade | Key Weakness |
|-------|--------------|--------------|
| #1 | A/B/C/D/F | [Primary concern] |
| #2 | A/B/C/D/F | [Primary concern] |

### Bias Check

| Bias Detected | Where | Impact |
|--------------|-------|--------|
| [Bias name] | Claim #X | [How it affects the conclusion] |

### Competing Explanations

| Evidence | Proposed Explanation | Alternative Explanations |
|----------|---------------------|------------------------|
| [Data point] | [Original claim] | 1. [Alternative] 2. [Alternative] |

### Verdict

**Overall evidence strength:** Strong / Moderate / Weak / Insufficient

**Recommendations:**
1. [Specific action to strengthen the weakest claim]
2. [Specific action to test the riskiest assumption]