ContextJet-ai Debug an agent from its trace
A trace is a tree of spans (parent request → LLM/tool/retrieval children). Most agent failures are visible in it if you read it in the right order. Don't guess from the final output - walk the tree.
Triage by symptom
| Symptom | Look here first |
|---|---|
| Wrong answer | The LLM span just before the bad output: was the prompt/context correct? Then the retrieval span that fed it. |
| Empty / truncated output | finish_reason (length? content_filter?), max_tokens, and any span that raised an exception then got swallowed. |
| Hallucinated facts | Retrieval spans - were the right docs retrieved (check scores/IDs)? If not, it's a retrieval bug, not an LLM bug. |
| Too slow | Span durations - find the critical path. Usually one slow retrieval, a serial loop that should be parallel, or a huge context. |
| Too expensive | Token counts per LLM span - find the span with the largest input_tokens (usually bloated context or a retry storm). |
| Silent failure | A span with an error/exception that was caught and returned empty. A missing subtree = a step that never ran. |
The systematic walk
- Start at the root, confirm the user input is what you expect.
- Follow to the first LLM call. Read the actual rendered prompt + context (not the template). Most "model is dumb" bugs are actually "we fed it the wrong context".
- Check each tool/retrieval span in order: inputs correct? output sane? error?
- Find the divergence point - the first span where reality differs from intent. Fix there, not at the output.
- Check token + latency on every LLM span to catch cost/perf issues even when the answer is right.
Common root causes (in order of frequency)
- Wrong/empty retrieved context → the LLM was set up to fail. (Retrieval bug.)
- Prompt/template rendering bug → a variable didn't interpolate; context is blank or duplicated.
- Swallowed exception → a
try/exceptthat logs and returns empty, dropping context or output silently. Grep for these. - Retry storm → the same call repeated N times (rate limit / transient error) inflating cost + latency.
- Context bloat → the whole history re-sent each turn;
input_tokensgrows every step.
Turn the fix into a regression test
Once you find the divergence, capture that input as an eval case (see the add-llm-evals skill) so the bug can't come back. Debugging a trace and not adding a test means you'll debug it again next month.
Anti-patterns
- Re-prompting the model when the real bug is upstream retrieval/rendering.
- Reading only the final output and inferring the cause.
- Fixing the symptom without adding a regression test.