Agent Architecture Audit
Diagnose failures in agent systems by inspecting each layer between the raw model call and the user-visible output.
Contract
Inputs:
- Agent or LLM application repository, failing run, trace, log, or symptom
- Optional model/provider list, tool definitions, memory files, and UI output
Outputs:
- Severity-ranked findings with evidence references
- Layer-by-layer failure diagnosis
- Ordered fix plan that prefers code gates over prompt-only changes
Creates/Modifies:
- None in audit mode
- Follow-up fixes only when explicitly requested
External Side Effects:
- None by default
- External logs, observability tools, or production systems only when already authorized
Confirmation Required:
- Before changing prompts, memory, tool contracts, persistence, production config, or user data
Delegates To:
debug for ordinary software defectsevaluation or advanced-evaluation for benchmark designsecurity-audit for prompt injection, secrets, auth, or privileged tool risk
When to Use
- An agent works in a model playground or direct API call but fails in the app.
- A wrapper, orchestration layer, memory system, or tool router was added and
quality regressed.
- The model skips required tools, claims tool use that did not happen, or
misreads tool output.
- Old conversation facts, stale memories, or compressed summaries leak into new
tasks.
- Logs show a correct answer but the CLI, API, UI, streaming layer, or renderer
shows a different result.
- Hidden retry, fallback, repair, or summarization loops may be mutating output.
Do not use for general code review — only when the failure is likely in the agent stack.
Layer Model
Audit the stack from source instructions to delivered output:
| Layer |
What Can Fail |
| System prompt |
Conflicts, bloat, stale rules, unclear priorities |
| Session history |
Old turns dominate or contradict the current task |
| Long-term memory |
Cross-project leakage, stale preferences, agent-written facts |
| Distillation |
Compaction turns guesses into pseudo-facts |
| Active recall |
Redundant summaries waste context or revive old context |
| Tool selection |
Required tools are optional in code, not enforced |
| Tool execution |
Missing calls, failed calls, unvalidated arguments |
| Tool interpretation |
Output is ignored, overtrusted, or read backward |
| Answer shaping |
Format, schema, markdown, or JSON is changed after reasoning |
| Transport/rendering |
Streaming, API, CLI, or UI mutates valid content |
| Hidden repair loops |
A second model pass silently rewrites the answer |
| Persistence |
Cached artifacts or expired state are reused as live evidence |
Audit Process
1. Scope the System
Identify:
- user entrypoints and output surfaces
- model providers, model tiers, and retry/fallback behavior
- prompt assembly path and instruction sources
- memory, retrieval, compaction, and persistence paths
- tool schemas, tool routing, and execution validation
- known symptoms and the first release or commit where they appeared
2. Collect Evidence
Search for:
- prompt templates and instruction injection points
- tool schemas, routers, validators, and execution logs
- memory admission, retrieval, and compaction code
- provider calls outside the primary agent loop
- fallback, retry, repair, or output rewriting passes
- response serialization, markdown rendering, JSON parsing, and stream handling
Prefer file and line evidence. Compare raw model output, post-processed output, transported output, and rendered output when logs are available.
3. Map Failure Mechanisms
For each suspected issue, record:
- symptom: what the user sees
- layer: where the corruption enters
- mechanism: how the layer changes behavior
- root cause: code, config, prompt, memory, transport, or process
- evidence: file, line, trace id, log row, or reproduced run
- confidence: high, medium, or low
4. Prioritize Fixes
Prefer fixes in this order:
- Enforce required tool use in code.
- Remove or expose hidden repair and fallback agents.
- Reduce duplicated context across prompt, memory, history, and summaries.
- Tighten memory admission so user corrections outrank agent assertions.
- Narrow compaction triggers and preserve uncertainty markers.
- Pass through already-valid output instead of rewriting it.
- Use typed envelopes for internal protocol boundaries.
- Add trace tests that compare raw, processed, transported, and rendered output.
Do not solve tool discipline, memory safety, or transport corruption only by adding stronger prompt wording.
Severity
| Severity |
Meaning |
| Critical |
The system can confidently perform wrong or unsafe operational behavior |
| High |
Correctness or stability degrades frequently under normal use |
| Medium |
The agent usually works, but the stack is fragile, wasteful, or hard to debug |
| Low |
Maintainability, observability, or cosmetic output issues |
Output Format
Lead with findings:
## Findings
| Severity | Layer | Finding | Evidence | Fix |
| --- | --- | --- | --- | --- |
| High | Tool selection | Required retrieval is prompt-only and can be skipped | `src/agent/router.ts:42` | Gate final answer on retrieval result |
## Diagnosis
[Explain which layer corrupts the behavior and why.]
## Fix Plan
1. [Code-first fix]
2. [Validation]
3. [Follow-up hardening]
If no issues are found, say so and list the evidence checked plus remaining
blind spots.
Anti-Patterns
- Blaming the model before testing wrapper-layer behavior.
- Accepting "must use tool" prompt text when no code enforces the requirement.
- Treating compressed summaries as ground truth.
- Allowing hidden model passes to rewrite final output without traceability.
- Reporting green status without comparing internal output to delivered output.
1---2name: agent-architecture-audit3description: Audit LLM and agent applications for wrapper regressions, prompt or memory contamination, tool discipline failures, hidden repair loops, and output rendering corruption. Use before shipping agent features or when an agent works in a direct model call but fails inside the product.4---5
6# Agent Architecture Audit
7
8Diagnose failures in agent systems by inspecting each layer between the raw model call and the user-visible output.
9
10## Contract
11
12Inputs:
13
14- Agent or LLM application repository, failing run, trace, log, or symptom
15- Optional model/provider list, tool definitions, memory files, and UI output
16
17Outputs:
18
19- Severity-ranked findings with evidence references
20- Layer-by-layer failure diagnosis
21- Ordered fix plan that prefers code gates over prompt-only changes
22
23Creates/Modifies:
24
25- None in audit mode
26- Follow-up fixes only when explicitly requested
27
28External Side Effects:
29
30- None by default
31- External logs, observability tools, or production systems only when already authorized
32
33Confirmation Required:
34
35- Before changing prompts, memory, tool contracts, persistence, production config, or user data
36
37Delegates To:
38
39- `debug` for ordinary software defects
40- `evaluation` or `advanced-evaluation` for benchmark design
41- `security-audit` for prompt injection, secrets, auth, or privileged tool risk
42
43## When to Use
44
45- An agent works in a model playground or direct API call but fails in the app.
46- A wrapper, orchestration layer, memory system, or tool router was added and
47 quality regressed.
48- The model skips required tools, claims tool use that did not happen, or
49 misreads tool output.
50- Old conversation facts, stale memories, or compressed summaries leak into new
51 tasks.
52- Logs show a correct answer but the CLI, API, UI, streaming layer, or renderer
53 shows a different result.
54- Hidden retry, fallback, repair, or summarization loops may be mutating output.
55
56Do not use for general code review — only when the failure is likely in the agent stack.
57
58## Layer Model
59
60Audit the stack from source instructions to delivered output:
61
62| Layer | What Can Fail |
63| --- | --- |
64| System prompt | Conflicts, bloat, stale rules, unclear priorities |
65| Session history | Old turns dominate or contradict the current task |
66| Long-term memory | Cross-project leakage, stale preferences, agent-written facts |
67| Distillation | Compaction turns guesses into pseudo-facts |
68| Active recall | Redundant summaries waste context or revive old context |
69| Tool selection | Required tools are optional in code, not enforced |
70| Tool execution | Missing calls, failed calls, unvalidated arguments |
71| Tool interpretation | Output is ignored, overtrusted, or read backward |
72| Answer shaping | Format, schema, markdown, or JSON is changed after reasoning |
73| Transport/rendering | Streaming, API, CLI, or UI mutates valid content |
74| Hidden repair loops | A second model pass silently rewrites the answer |
75| Persistence | Cached artifacts or expired state are reused as live evidence |
76
77## Audit Process
78
79### 1. Scope the System
80
81Identify:
82
83- user entrypoints and output surfaces
84- model providers, model tiers, and retry/fallback behavior
85- prompt assembly path and instruction sources
86- memory, retrieval, compaction, and persistence paths
87- tool schemas, tool routing, and execution validation
88- known symptoms and the first release or commit where they appeared
89
90### 2. Collect Evidence
91
92Search for:
93
94- prompt templates and instruction injection points
95- tool schemas, routers, validators, and execution logs
96- memory admission, retrieval, and compaction code
97- provider calls outside the primary agent loop
98- fallback, retry, repair, or output rewriting passes
99- response serialization, markdown rendering, JSON parsing, and stream handling
100
101Prefer file and line evidence. Compare raw model output, post-processed output, transported output, and rendered output when logs are available.
102
103### 3. Map Failure Mechanisms
104
105For each suspected issue, record:
106
107- symptom: what the user sees
108- layer: where the corruption enters
109- mechanism: how the layer changes behavior
110- root cause: code, config, prompt, memory, transport, or process
111- evidence: file, line, trace id, log row, or reproduced run
112- confidence: high, medium, or low
113
114### 4. Prioritize Fixes
115
116Prefer fixes in this order:
117
1181. Enforce required tool use in code.
1192. Remove or expose hidden repair and fallback agents.
1203. Reduce duplicated context across prompt, memory, history, and summaries.
1214. Tighten memory admission so user corrections outrank agent assertions.
1225. Narrow compaction triggers and preserve uncertainty markers.
1236. Pass through already-valid output instead of rewriting it.
1247. Use typed envelopes for internal protocol boundaries.
1258. Add trace tests that compare raw, processed, transported, and rendered output.
126
127Do not solve tool discipline, memory safety, or transport corruption only by adding stronger prompt wording.
128
129## Severity
130
131| Severity | Meaning |
132| --- | --- |
133| Critical | The system can confidently perform wrong or unsafe operational behavior |
134| High | Correctness or stability degrades frequently under normal use |
135| Medium | The agent usually works, but the stack is fragile, wasteful, or hard to debug |
136| Low | Maintainability, observability, or cosmetic output issues |
137
138## Output Format
139
140Lead with findings:
141
142```markdown
143## Findings
144
145| Severity | Layer | Finding | Evidence | Fix |
146| --- | --- | --- | --- | --- |
147| High | Tool selection | Required retrieval is prompt-only and can be skipped | `src/agent/router.ts:42` | Gate final answer on retrieval result |
148
149## Diagnosis
150
151[Explain which layer corrupts the behavior and why.]
152
153## Fix Plan
154
1551. [Code-first fix]
1562. [Validation]
1573. [Follow-up hardening]
158```
159
160If no issues are found, say so and list the evidence checked plus remaining
161blind spots.
162
163## Anti-Patterns
164
165- Blaming the model before testing wrapper-layer behavior.
166- Accepting "must use tool" prompt text when no code enforces the requirement.
167- Treating compressed summaries as ground truth.
168- Allowing hidden model passes to rewrite final output without traceability.
169- Reporting green status without comparing internal output to delivered output.
shipshitdev