Letta 8-Failure-Modes Diagnostic
Overview
The Letta Leaderboard for benchmarking agentic memory names eight distinct
failure modes that tend to co-occur (Ch4):
- No-retrieval-when-available — model fails to recognize when relevant
info is already in memory; issues unnecessary searches.
- Hierarchy-collapse — trivia in prime memory; critical facts archived
or dropped.
- In-conversation-misses — agent misses key pieces of info even when
present in the immediate context.
- Volume-degradation — retrieval accuracy degrades as data volume
grows; performance drops at scale.
- Silent-overwrite — new info overwrites old facts instead of being
layered; system cannot explain how or why things changed.
- Cross-reference-failure — related info isolated in separate silos;
no pattern recognition.
- Temporal-blur — event timelines blur; agent loses temporal coherence.
- Threshold-collapse — works at hundreds of facts, quietly collapses
at thousands.
This skill takes a snapshot of an agent's memory state (or a structural
description) and reports which of these 8 are present, with evidence and
recommended fixes. It runs static analysis — no agent inference loop required.
When to Use
- Pre-launch review of a new memory implementation
- Root-cause analysis when an agent in production is "forgetting" or
"drifting"
- Periodic audit (weekly / monthly) on long-running agents
- Code review of a colleague's memory layer
Phrases: "audit my memory architecture", "why is my agent forgetting",
"is my memory production-ready", "Letta Leaderboard", "memory diagnostic".
When NOT to Use
- Single failure mode you've already identified. If you know it's
silent-overwrite, just go fix it; this skill's value is the cross-cutting
audit, not the depth on one mode.
- Runtime monitor. This is a static diagnostic. Production needs
metrics + alerts, not periodic full-audit invocation.
- Benchmark/eval. This does not produce a comparable accuracy number.
For Letta Leaderboard scoring, run the actual benchmark suite.
Process
| Step |
Input |
Action |
Output |
Verification |
| 1 |
Memory snapshot dict OR architecture description JSON |
lib.diagnose(snapshot) |
DiagnosticReport with 8 entries (one per failure mode) |
report covers all 8 modes; each entry has status ∈ {ok, warning, present}; severity score 0-3 |
| 2 |
Report |
lib.format_text(report) |
Human-readable diagnostic |
every "present" mode includes evidence + fix recommendation |
| 3 |
Report |
lib.format_json(report) |
machine-readable JSON |
round-trip serializable |
| 4 |
Report |
lib.total_score(report) |
int 0-24 (sum of severities) |
0 = production-ready; ≥10 = ship at risk; ≥18 = do not ship |
| 5 |
Scenario name |
cli.py scenario broken-vs-clean |
showcase that EXERCISES all 8 modes: an anti-pattern snapshot triggers all 8 (present) and a clean composed snapshot triggers 0 |
broken reports 8 present, clean reports 0 |
Diagnostic Heuristics (per failure mode)
| Mode |
Static signal |
Threshold |
| No-retrieval-when-available |
recall layer is empty or query log not preserved |
recall.size == 0 with mature memory |
| Hierarchy-collapse |
>50% short-lived facts in core OR durable facts > 50% in archival vs core |
(composes with hierarchical-memory.diagnostics) |
| In-conversation-misses |
extract_fn not wired up — interactions logged but no facts promoted |
recall.size > 5 AND core.size == 0 |
| Volume-degradation |
retrieval method is linear-scan with no index |
flagged via architecture description tag |
| Silent-overwrite |
edges have no valid_until mechanism / no invalidation_reason |
flagged if bi-temporal-edge primitive is absent |
| Cross-reference-failure |
average node degree < 1 OR graph is a forest with many disconnected components |
components > nodes/10 |
| Temporal-blur |
timestamps not preserved on edges / facts |
facts without created_at |
| Threshold-collapse |
core_limit OR retrieval pipeline declared O(n) on architecture form |
size-test scenario fails at 10x scale |
Rationalizations
| Agent rationalization |
Documented rebuttal |
| "My agent works fine on the demo — I'll skip the audit." |
Demo workloads are small. The Ch4 anchor: "Systems that work acceptably with hundreds of facts quietly collapse when exposed to thousands." Threshold-collapse is the failure mode that hides best in demos. |
| "I'll fix the failures as users complain." |
The first 7 failure modes are silent — the agent "feels" wrong without producing a single error log. You will not know which mode to fix unless you scan for them. |
| "I'll skip diagnostics for short-lived agents." |
Short-lived agents that suffer in-conversation-misses still produce wrong answers. The discipline is cheap (one call); the failure cost is incident-response time. |
| "I already use a hierarchical memory — that covers most of these." |
Letta-style hierarchy covers 1, 2, 3, and partially 8. The other 4 (silent-overwrite, cross-reference-failure, temporal-blur, volume-degradation) are orthogonal and need separate primitives (bi-temporal-edge, graphiti-incremental-update, indexed retrieval). The hierarchy alone is necessary, not sufficient. |
| "The diagnostic scoring is arbitrary." |
The 0-24 score is a forcing function. The Ch4 chapter quote — "The quality of memory management directly determines agent performance on long-running tasks" — converts to a single number that goes up when memory degrades. Track the number. |
Red Flags
- All 8 modes flagged
ok. Either the diagnostic is broken (likely)
or the architecture is exceptional (suspicious — verify by running a
size-stress scenario).
silent-overwrite flagged with no bi-temporal-edge. The fix is
mechanical: integrate bi-temporal-edge for the affected relationships.hierarchy-collapse and volume-degradation both flagged. The
memory architecture is unrescuable without a redesign; recommend
switching to a structured 3-tier hierarchy (composes with
hierarchical-memory).temporal-blur flagged on a regulated-domain agent. Compliance
blocker — fix before shipping.
Non-Negotiable Verification
- Run the benchmark battery.
python cli.py benchmark must report:
- all 8 failure modes detectable from a synthesized broken memory
- no false-positives on a known-clean memory built from the other
three Ch4 skills (bi-temporal-edge + hierarchical-memory +
graphiti-incremental-update)
- round-trip report serialize / deserialize
- Run the showcase scenario.
python cli.py scenario broken-vs-clean
reports 8 vs 0 failure modes respectively.
- Verify CLI help.
python cli.py --help exits 0 and prints SKILL.md.
Security Posture
- Prompt injection. Memory snapshots and architecture descriptions are
untrusted input analyzed statically - nothing in them is executed. The risk
is a crafted snapshot that hides its failure modes to earn a false clean
report; diagnose real exported state, not a hand-written summary.
- Data exfiltration. Snapshots contain actual memory content (facts,
conversation traces). No network calls, no file writes; the diagnostic
report goes to stdout and the caller owns where it flows.
- Privilege escalation. No shell invocation, no eval, no dynamic import.
A zero-failure-mode report is advisory - it is a static diagnostic, not a
production go-ahead, and does not replace runtime observability.
Source Attribution
Distilled from Agentic GraphRAG (O'Reilly, by Anthony Alcaraz and Sam Julien),
Chapter 4 — The Problem section (Letta Leaderboard 8 failure modes).
Composes with sibling Ch4 skills: bi-temporal-edge / hierarchical-memory /
graphiti-incremental-update.
1---2name: letta-failure-modes3description: Reviewer skill: diagnose an agent's memory architecture against the 8 Letta Leaderboard failure modes (Ch4). Takes a memory snapshot (or a description of the architecture) and reports which failure modes are present, with concrete evidence and recommended fixes. Use BEFORE shipping any memory implementation to production and BEFORE root-causing why a deployed agent "forgets" or "drifts." NOT a benchmark (does not produce a single accuracy number), NOT a substitute for production observability (this is a static diagnostic, not a runtime monitor).4---56# Letta 8-Failure-Modes Diagnostic78## Overview910The Letta Leaderboard for benchmarking agentic memory names eight distinct11failure modes that tend to co-occur (Ch4):12131. **No-retrieval-when-available** — model fails to recognize when relevant14 info is already in memory; issues unnecessary searches.152. **Hierarchy-collapse** — trivia in prime memory; critical facts archived16 or dropped.173. **In-conversation-misses** — agent misses key pieces of info even when18 present in the immediate context.194. **Volume-degradation** — retrieval accuracy degrades as data volume20 grows; performance drops at scale.215. **Silent-overwrite** — new info overwrites old facts instead of being22 layered; system cannot explain how or why things changed.236. **Cross-reference-failure** — related info isolated in separate silos;24 no pattern recognition.257. **Temporal-blur** — event timelines blur; agent loses temporal coherence.268. **Threshold-collapse** — works at hundreds of facts, quietly collapses27 at thousands.2829This skill takes a snapshot of an agent's memory state (or a structural30description) and reports which of these 8 are present, with evidence and31recommended fixes. It runs static analysis — no agent inference loop required.3233## When to Use3435- Pre-launch review of a new memory implementation36- Root-cause analysis when an agent in production is "forgetting" or37 "drifting"38- Periodic audit (weekly / monthly) on long-running agents39- Code review of a colleague's memory layer4041Phrases: "audit my memory architecture", "why is my agent forgetting",42"is my memory production-ready", "Letta Leaderboard", "memory diagnostic".4344## When NOT to Use4546- **Single failure mode you've already identified.** If you know it's47 silent-overwrite, just go fix it; this skill's value is the cross-cutting48 audit, not the depth on one mode.49- **Runtime monitor.** This is a static diagnostic. Production needs50 metrics + alerts, not periodic full-audit invocation.51- **Benchmark/eval.** This does not produce a comparable accuracy number.52 For Letta Leaderboard scoring, run the actual benchmark suite.5354## Process5556| Step | Input | Action | Output | Verification |57|------|-------|--------|--------|--------------|58| 1 | Memory snapshot dict OR architecture description JSON | `lib.diagnose(snapshot)` | `DiagnosticReport` with 8 entries (one per failure mode) | report covers all 8 modes; each entry has status ∈ {ok, warning, present}; severity score 0-3 |59| 2 | Report | `lib.format_text(report)` | Human-readable diagnostic | every "present" mode includes evidence + fix recommendation |60| 3 | Report | `lib.format_json(report)` | machine-readable JSON | round-trip serializable |61| 4 | Report | `lib.total_score(report)` | int 0-24 (sum of severities) | 0 = production-ready; ≥10 = ship at risk; ≥18 = do not ship |62| 5 | Scenario name | `cli.py scenario broken-vs-clean` | showcase that EXERCISES all 8 modes: an anti-pattern snapshot triggers all 8 (`present`) and a clean composed snapshot triggers 0 | broken reports 8 `present`, clean reports 0 |6364## Diagnostic Heuristics (per failure mode)6566| Mode | Static signal | Threshold |67|------|---------------|-----------|68| No-retrieval-when-available | recall layer is empty or query log not preserved | recall.size == 0 with mature memory |69| Hierarchy-collapse | >50% short-lived facts in core OR durable facts > 50% in archival vs core | (composes with hierarchical-memory.diagnostics) |70| In-conversation-misses | extract_fn not wired up — interactions logged but no facts promoted | recall.size > 5 AND core.size == 0 |71| Volume-degradation | retrieval method is linear-scan with no index | flagged via architecture description tag |72| Silent-overwrite | edges have no `valid_until` mechanism / no `invalidation_reason` | flagged if bi-temporal-edge primitive is absent |73| Cross-reference-failure | average node degree < 1 OR graph is a forest with many disconnected components | components > nodes/10 |74| Temporal-blur | timestamps not preserved on edges / facts | facts without `created_at` |75| Threshold-collapse | core_limit OR retrieval pipeline declared O(n) on architecture form | size-test scenario fails at 10x scale |7677## Rationalizations7879| Agent rationalization | Documented rebuttal |80|------------------------|--------------------|81| "My agent works fine on the demo — I'll skip the audit." | Demo workloads are small. The Ch4 anchor: "Systems that work acceptably with hundreds of facts quietly collapse when exposed to thousands." Threshold-collapse is the failure mode that hides best in demos. |82| "I'll fix the failures as users complain." | The first 7 failure modes are silent — the agent "feels" wrong without producing a single error log. You will not know which mode to fix unless you scan for them. |83| "I'll skip diagnostics for short-lived agents." | Short-lived agents that suffer in-conversation-misses still produce wrong answers. The discipline is cheap (one call); the failure cost is incident-response time. |84| "I already use a hierarchical memory — that covers most of these." | Letta-style hierarchy covers 1, 2, 3, and partially 8. The other 4 (silent-overwrite, cross-reference-failure, temporal-blur, volume-degradation) are orthogonal and need separate primitives (bi-temporal-edge, graphiti-incremental-update, indexed retrieval). The hierarchy alone is necessary, not sufficient. |85| "The diagnostic scoring is arbitrary." | The 0-24 score is a forcing function. The Ch4 chapter quote — "The quality of memory management directly determines agent performance on long-running tasks" — converts to a single number that goes up when memory degrades. Track the number. |8687## Red Flags8889- **All 8 modes flagged `ok`.** Either the diagnostic is broken (likely)90 or the architecture is exceptional (suspicious — verify by running a91 size-stress scenario).92- **`silent-overwrite` flagged with no `bi-temporal-edge`.** The fix is93 mechanical: integrate `bi-temporal-edge` for the affected relationships.94- **`hierarchy-collapse` and `volume-degradation` both flagged.** The95 memory architecture is unrescuable without a redesign; recommend96 switching to a structured 3-tier hierarchy (composes with97 `hierarchical-memory`).98- **`temporal-blur` flagged on a regulated-domain agent.** Compliance99 blocker — fix before shipping.100101## Non-Negotiable Verification1021031. **Run the benchmark battery.** `python cli.py benchmark` must report:104 - all 8 failure modes detectable from a synthesized broken memory105 - no false-positives on a known-clean memory built from the other106 three Ch4 skills (bi-temporal-edge + hierarchical-memory +107 graphiti-incremental-update)108 - round-trip report serialize / deserialize1092. **Run the showcase scenario.** `python cli.py scenario broken-vs-clean`110 reports 8 vs 0 failure modes respectively.1113. **Verify CLI help.** `python cli.py --help` exits 0 and prints SKILL.md.112113## Security Posture114115- **Prompt injection.** Memory snapshots and architecture descriptions are116 untrusted input analyzed statically - nothing in them is executed. The risk117 is a crafted snapshot that hides its failure modes to earn a false clean118 report; diagnose real exported state, not a hand-written summary.119- **Data exfiltration.** Snapshots contain actual memory content (facts,120 conversation traces). No network calls, no file writes; the diagnostic121 report goes to stdout and the caller owns where it flows.122- **Privilege escalation.** No shell invocation, no eval, no dynamic import.123 A zero-failure-mode report is advisory - it is a static diagnostic, not a124 production go-ahead, and does not replace runtime observability.125126## Source Attribution127128Distilled from *Agentic GraphRAG* (O'Reilly, by Anthony Alcaraz and Sam Julien),129Chapter 4 — The Problem section (Letta Leaderboard 8 failure modes).130Composes with sibling Ch4 skills: bi-temporal-edge / hierarchical-memory /131graphiti-incremental-update.
AnthonyAlcaraz