Harness Scaling

Attribution: Derived from arXiv:2605.26112v1 — From Model Scaling to System Scaling: Scaling the Harness in Agentic AI by Shangding Gu (UC Berkeley), May 2026. Reference implementation: CheetahClaws.

Harness Scaling for Agentic AI

Agent performance emerges from the interaction among multiple components, not from model capability alone. This skill encodes the insight that investing only in stronger foundation models while neglecting the surrounding harness — memory, context assembly, skill routing, and governance — yields unreliable agents regardless of model strength.

When to Activate

Activate this skill when:

• Designing a new agentic system from scratch and choosing architectural components
• Debugging persistent agent failures that survive model upgrades
• Evaluating whether to invest in a stronger model vs. improving harness design
• Auditing a production agent for reliability, auditability, or safety gaps
• Implementing multi-agent coordination systems
• Choosing memory, context, or routing strategies for an agent deployment
• Answering: "why does my agent confidently do the wrong thing?"

Core Concepts

The Harness Equation

Agent harness performance is a function of six interacting components:

𝒫_H = Φ(ℛ, ℳ, 𝒞, 𝒮, 𝒪, 𝒢)

Key insight: Scaling 𝒮 (skills) without scaling 𝒢 (governance) produces faster but less reliable progress. Each component must scale together.

The Three Primary Bottlenecks

The paper identifies three failure modes that survive model upgrades:

1. Exposure without access (context governance failure) — large context windows create signal dilution, not improved access
2. Stale-but-confident (memory trust failure) — outdated facts remain highly ranked; the agent acts destructively on invalidated assumptions
3. Confident-but-unchecked (skill routing failure) — specialized subagents return plausible outputs without downstream validation

Why Model Scaling Alone Is Insufficient

Three objections and their rebuttals:

• "Stronger models will solve system problems" → Stale memory, over-broad permissions, missing provenance, and unsafe execution are system failures, not prediction failures. Stronger models don't eliminate the need for governance.
• "End-to-end training will replace modular systems" → Deployed agents require auditability, permission control, rollback, and provenance — these aren't optional, they're deployment requirements.
• "System evaluation is too expensive" → Cost and standardization challenges are precisely why evaluation is needed. Real agents face latency, monetary cost, tool risk, and memory drift.

Detailed Topics

Component 1: Memory Store (ℳ)

Memory has four quality axes that must be actively maintained:

System move: Make trust a runtime decision, not a stored property. Retrieved content is a hypothesis until re-checked.

Retrieval ranking formula (CheetahClaws):

rank = relevance × (1 - staleness_penalty) × confidence_factor

Implementation pattern:

@dataclass
class MemoryEntry:
    content: str
    confidence: float          # 0.0 – 1.0
    last_verified: datetime    # When was this last checked against live env?
    valid_until: Optional[datetime]  # None = no expiry
    source: str               # Provenance

def retrieve(query: str, entries: list[MemoryEntry]) -> list[MemoryEntry]:
    now = datetime.utcnow()
    ranked = []
    for e in entries:
        staleness = (now - e.last_verified).total_seconds() / 86400  # days
        staleness_penalty = min(staleness / 30, 0.9)  # cap at 90% penalty
        score = semantic_similarity(query, e.content) \
                * (1 - staleness_penalty) \
                * e.confidence
        ranked.append((score, e))
    ranked.sort(reverse=True)
    # Treat top results as hypotheses — re-verify before acting
    return [e for _, e in ranked[:5]]

Component 2: Context Constructor (𝒞)

Context has four quality axes:

System move: Treat each turn's context as output of a selection policy, not a fixed buffer.

Context assembly policy:

def assemble_context(task: str, budget: int) -> Context:
    # Layer 1: Persistent priors (loaded at session start)
    persistent = load_persistent_priors()  # e.g., CLAUDE.md equivalent

    # Layer 2: Just-in-time retrieval (not static index)
    jit_facts = retrieve_verified_memory(task)

    # Layer 3: Live environment search (always fresh)
    live_state = search_live_env(task)  # glob/grep/tool calls

    # Assemble with token budget enforcement
    context = pack_by_relevance(
        sources=[persistent, jit_facts, live_state],
        budget=budget,
        weight_fn=lambda x: semantic_score(task, x) * recency_weight(x)
    )
    return context  # Every token has a source ID and timestamp

The "lost in the middle" problem: Attention degrades for content in the middle of the context window. Place most critical facts at the start or end.

Component 3: Skill-Routing Layer (𝒮)

Skill routing has four quality axes:

System move: Couple learned routing policy with verification at every step.

Routing with post-condition verification:

async def route_and_verify(task: SubTask) -> Result:
    # Estimate task type from available context
    task_type = classify_task(task)
    confidence = task_type.confidence

    # Confidence-aware escalation
    if confidence < 0.7:
        skill = FALLBACK_SKILL  # more capable / general
    else:
        skill = SKILL_REGISTRY[task_type.label]

    result = await skill.execute(task)

    # Post-condition check — mandatory, not optional
    if not skill.verify_postcondition(task, result):
        result = await VERIFICATION_AGENT.recheck(task, result)

    return result

Component 4: Orchestration (𝒪) and Governance (𝒢)

The orchestration loop coordinates all components. Governance gates:

• Intermediate reasoning outputs (before accepting a plan step)
• External action effects (before committing a file write, API call, etc.)
• Memory write-backs (traced and auditable)

Governance checklist before any external action:

□ Has the relevant memory been re-verified against live state?
□ Does the skill's post-condition check pass?
□ Is there an audit trace for this action?
□ Is rollback available if the action has side effects?
□ Is the permission scope appropriate (principle of least privilege)?

Practical Guidance

Three Production Harness Comparisons

The paper compares three reference harnesses with similar frontier models but different harness designs:

Key insight: Similar frontier models yield radically different agents based on harness design alone.

Temporal Lever Framework

Three levers operating at different timescales:

Design all three levers together. Prompt tuning without memory governance leads to tasks succeeding in isolation but failing across sessions.

Multi-Agent Performance Data

From Anthropic research cited in the paper:

• Multi-agent (Opus 4 lead + Sonnet 4 subagents) outperformed single-agent Opus 4 by 90.2% on internal research tasks
• Token usage explained 80% of performance variance; adding tool-call count and model choice raised it to 95%
• Breadth-first tasks show strongest gains through parallel context windows

Multi-agent failure modes to design against:

• Decomposition is easier than collaboration
• Inter-agent misalignment (no shared state)
• Inadequate task verification between agents
• Missing: uncertainty communication, contradiction detection, task de-duplication, conflict resolution

Examples

Example: Diagnosing a failing agent with the harness framework

Agent symptom: Confidently deletes files based on outdated assumptions.

Diagnosis checklist:
  ℳ Memory: Was the file-existence assumption re-verified before action?
             → NO. Stale-but-confident failure.
  𝒞 Context: Was the file listing freshly retrieved (JIT) or from a stale index?
              → STALE INDEX. Exposure-without-access failure.
  𝒮 Skills: Did the delete skill have a post-condition check?
             → NO. Confident-but-unchecked failure.
  𝒢 Governance: Was rollback available for the delete action?
                 → NO.

Fix:
  ℳ: Add staleness check — re-verify file existence via tool call before destructive action
  𝒞: Replace static file index with JIT glob/ls tool call
  𝒮: Add post-condition: verify file is gone after delete AND previous state was as expected
  𝒢: Require user confirmation for irreversible actions OR implement soft-delete

Example: Harness design review checklist

## Harness Review: [System Name]

### ℳ Memory
- [ ] Confidence and recency tracked as first-class fields?
- [ ] Retrieval ranking penalizes staleness?
- [ ] Retrieved content treated as hypothesis until re-verified?
- [ ] Periodic re-verification against live environment?

### 𝒞 Context
- [ ] Context assembled by selection policy (not fixed buffer)?
- [ ] Persistent priors + JIT retrieval + live search (three layers)?
- [ ] Every token has source provenance?
- [ ] Token budget enforced with relevance ranking?

### 𝒮 Skill Routing
- [ ] Each skill has documented capability scope?
- [ ] Post-condition checks defined per skill?
- [ ] Confidence-aware escalation to more capable models?
- [ ] Composition verification between chained skills?

### 𝒢 Governance
- [ ] Audit trace for memory writes?
- [ ] Audit trace for routing decisions?
- [ ] Audit trace for tool permissions?
- [ ] Rollback available for side-effecting actions?

Guidelines

1. Design all six components together — a strong ℛ model with weak ℳ/𝒞/𝒮/𝒢 produces unreliable agents
2. Make memory trust a runtime decision — retrieved content is a hypothesis until re-verified against live environment
3. Assemble context as a selection policy — weight semantic relevance + compactness + recency
4. Couple every skill invocation with a post-condition check — fluent output ≠ correct output
5. Scale 𝒮 (skills) and 𝒢 (governance) together — adding capabilities without governance adds speed but reduces reliability
6. Use three context layers — persistent priors (loaded upfront) + JIT retrieval + live environment search
7. Propagate provenance — every token in context should have a traceable source and timestamp
8. Design for audit — inspectable traces for memory writes, routing changes, tool permissions, agent failures

Gotchas

1. Model upgrades masking system problems — a stronger model can paper over a bad harness temporarily, until edge cases expose the underlying failure. Audit the harness, don't just upgrade the model.
2. Stale-but-confident is silent — outdated memory entries that remain highly ranked via semantic similarity are the most dangerous failure: the agent acts with high confidence on wrong assumptions. Add explicit staleness penalties.
3. Context exposure ≠ context access — longer context windows do not improve retrieval; they dilute signal. Treat context as output of a ranking policy, not a dump buffer.
4. Governance debt — adding skills/tools without corresponding governance (audit traces, rollback, permission scoping) creates compounding risk. Each new capability needs a governance counterpart.
5. Pass-k collapse — agents that score well on single-shot benchmarks collapse under repeated rollouts (pass^k). Design for consistency, not just peak performance.
6. Decomposition ≠ collaboration — multi-agent systems easily decompose tasks but struggle to collaborate (shared state, uncertainty communication, contradiction detection). Decomposition is a solved problem; collaboration is not.
7. Reward hacking in longitudinal evaluation — optimizing for benchmark proxy metrics instead of actual task quality. Track regression and earlier-failure recurrence alongside rolling success rates.

Integration

This skill provides the architectural foundation for:

• memory-systems — implements ℳ in depth
• multi-agent-patterns — implements 𝒪 coordination patterns
• context-fundamentals — implements 𝒞 in depth
• agentic-eval — measures harness quality

References

Internal references:

• Six-Component Architecture — detailed breakdown of ℛ, ℳ, 𝒞, 𝒮, 𝒪, 𝒢 with sub-axes and implementation patterns
• Context Governance Patterns — context selection policies, JIT refresh, provenance tracking
• Trustworthy Memory Design — confidence/staleness fields, verification loops, conflict resolution
• Skill Routing & Verification — adaptive routing policies, post-condition specs, escalation
• Safe Agent Evolution — the four-pillar maturity framework: persist, update, measure, audit

External resources:

• Paper: arXiv:2605.26112v1 — primary source
• CheetahClaws reference implementation — Python harness with explicit trust axes
• SWE-bench — coding agent benchmark showing interface design matters as much as models
• τ-bench — pass^k evaluation for agent reliability across repeated rollouts

Skill Metadata

Created: 2026-05-26 Source Paper: arXiv:2605.26112v1 — Shangding Gu, UC Berkeley Version: 1.0.1