Auditing Intelligence Context Fit

Auditing Intelligence-Context Fit

The Multiplicative Model

output_quality = reasoning_tier × context_quality

This is multiplicative. A mismatch in either dimension degrades output non-linearly:

The last row is the "long context + weak reasoning = harmful output" failure. It's worse than simply poor because the model generates confident-sounding wrong answers.

The Degradation Curve

As context complexity increases beyond a model's reasoning ceiling:

1. Phase 1 — Normal operation: Model handles context well; output quality matches expectation
2. Phase 2 — Reasoning plateau: Model is at capacity; quality stops improving even as context improves
3. Phase 3 — Context poisoning: Additional context overwhelms the model; it begins drawing spurious connections; quality degrades
4. Phase 4 — Expensive RAG fallback: System compensates by adding more retrieval passes, reranking, and chain-of-thought prompting — all of which add cost without fixing the root mismatch

Symptom of Phase 3–4: "We added better context and the answers got worse, so we added more prompting."

flowchart LR
    P1["Phase 1\nNormal Operation\nQuality ✓"] -->|complexity grows| P2["Phase 2\nReasoning Plateau\nQuality flat"]
    P2 -->|more context added| P3["Phase 3\nContext Poisoning\nQuality ↓\nSpurious connections"]
    P3 -->|team compensates| P4["Phase 4\nExpensive RAG Fallback\nCost ↑↑  Quality unchanged"]

    style P1 fill:#dcfce7,stroke:#22c55e
    style P2 fill:#fef9c3,stroke:#eab308
    style P3 fill:#fed7aa,stroke:#f97316
    style P4 fill:#fee2e2,stroke:#ef4444

The Fit Matrix

quadrantChart
    title Intelligence-Context Fit — Output Outcomes
    x-axis "Low Model Tier (Haiku)" --> "High Model Tier (Opus)"
    y-axis "Simple Context (L1–2)" --> "Complex Context (L4–5)"
    quadrant-1 "Optimal: excellent output"
    quadrant-2 "Over-resourced: high cost, correct output"
    quadrant-3 "Cost-efficient: correct output"
    quadrant-4 "Harmful: confident wrong answers"
    Haiku + Factual: [0.15, 0.15]
    Sonnet + Relational: [0.5, 0.5]
    Opus + Causal: [0.85, 0.85]
    Haiku + Causal: [0.15, 0.85]
    Opus + Factual: [0.85, 0.15]

The Audit Framework

Step 1: Classify Context Complexity

Rate the complexity of the context being passed to the model on this scale:

Step 2: Assess Model Reasoning Tier

Note: Model capability evolves rapidly. Retest your tier assignments when new model versions release.

Step 3: Identify the Mismatch

Scenario A: Context too complex for model tier

• Symptom: Confident but incorrect answers; spurious causal claims; inconsistent reasoning across runs
• Fix: Upgrade model tier OR simplify context (decompose into sub-queries at lower complexity levels)

Scenario B: Model over-qualified for context

• Symptom: Correct output but unnecessarily high cost and latency
• Fix: Downgrade model tier for this query class; reserve advanced models for Level 4–5 queries

Scenario C: Context quality is the bottleneck, not model

• Symptom: Upgrading the model doesn't improve output; answers are plausible but missing key facts
• Fix: Improve retrieval pipeline (check diagnosing-rag-failure-modes); model is not the constraint

Scenario D: Context volume exceeds model window effectively

• Symptom: Model ignores content from the middle of long contexts; only uses beginning/end
• Fix: Apply windowed compression (see temporal-reasoning-sleuth); reduce context length, not model

Step 4: Prescribe the Fix

Common Anti-Patterns

Anti-Pattern: Haiku + Organizational History

Using a basic-tier model on Level 4–5 context complexity. The model confidently fills gaps it cannot reason over. The output looks plausible but contains fabricated causal connections.

Fix: Reserve causal/temporal queries for Sonnet or Opus tier. Use Haiku only for factual lookup (Level 1–2).

Anti-Pattern: Opus on Simple RAG Lookups

Using an advanced-tier model for point-query factual retrieval. Correct output, but 5–10× the cost and latency of an appropriate tier.

Fix: Route Level 1–2 queries to Haiku; reserve Opus for Level 4–5 queries where reasoning depth matters.

Anti-Pattern: Compensating with More Prompting

When output quality degrades, adding more system prompt, chain-of-thought instructions, and few-shot examples to compensate for a model-context mismatch. This adds cost and complexity without fixing the root cause.

Fix: Identify which step of the audit framework is the actual mismatch and fix it there.

Output: Fit-Audit Report

INTELLIGENCE-CONTEXT FIT AUDIT
================================
System / Query Class: [what you're auditing]
Context Complexity Level: [1–5] — [description of why]
Current Model Tier: [Haiku / Sonnet / Opus]
Observed Symptom: [what's going wrong]

Mismatch Type: [Scenario A / B / C / D from Step 3]

Root Cause:
[One paragraph explaining the mismatch]

Recommended Fix:
[Specific action — upgrade model / decompose queries / fix retrieval / compress context]

Estimated Impact:
[Quality improvement expected] / [Cost change expected]

Related Skills:
[Link to diagnosing-rag-failure-modes / designing-hybrid-context-layers as applicable]