AI Agents Debugging Playbook

ai-agents Debugging Playbook

Symptom-first triage for this repository's known failure modes. Every row below was earned by a real incident; the retro path is cited so you can read the full story. The playbook answers one question: given this symptom, what is the FIRST command to run, what experiment discriminates between causes, and what trap has already cost someone real time here?

Vocabulary used once: a "guard" is a PreToolUse or pre-push hook that can block an action (exit 2 blocks, exit 0 allows). A "drift gate" is a CI check that fails when a generated tree no longer matches its canonical source. A "discriminating experiment" is one cheap action whose outcome splits the hypothesis space in two.

Triggers

• triage this failure
• why is my push blocked
• debug this CI red
• what does this gate failure mean
• hook blocked my tool call

When NOT To Use This Skill

Process

Phase 1: Match the symptom

Find your symptom in the master table. Run the first command exactly as written (from repo root). Do not guess at fixes before the discriminating experiment tells you which cause you have.

Local blocks (a hook stopped you)

CI reds (a gate failed on the PR)

Copilot CLI runtime failures

Phase 2: Run the discriminating experiment

Before writing any fix, answer three questions in order:

1. Does it reproduce on main? If yes, it is not your PR. Stop debugging your branch.
2. Is the gate doing its job, or is the gate broken? A guard block with a clear violation message is the gate working; fix the violation. A fail_open EVENT, a timeout, or a gate red that reproduces on main is gate infrastructure; file it, do not route around it.
3. If it is a drift red: which tree is canonical? Never edit until you can answer this from AGENTS.md or the generator docstring, not from the diff.

If three read-only commands have not identified the cause, stop and escalate to a real investigation (analyze skill) instead of shotgun-fixing. If you are looping on the same failed fix, stuck-detection names the pattern.

Phase 3: Fix, then prove it locally

1. Apply the fix on the canonical surface only.
2. Re-run the exact failing check locally, not a proxy for it:
   • Tests: uv run pytest tests/ -x (new skill tests belong in tests/skills/<name>/; the default bundle runner reaches legacy colocated suites, while direct legacy-path invocation is only for targeted diagnosis)
   • Full shift-left sweep: uv run python scripts/validation/pre_pr.py (exit codes per ADR-035: 0 ok, 1 logic, 2 config)
   • Drift: the specific surface command from the table above
3. Prove the check can still fail. Re-run it with a deliberate defect injected; a gate that prints OK both ways is not covering your change. Three here pass in ways that read as success: validate_install_parity.py is a co-change check, not a content comparison, and it skips RULE groups entirely (:378) and exempts diffs touching only hand-maintained agent copies (:389); that script and run_plugin_version_bump_ci.py both diff a base ref, so an uncommitted control is invisible and you must commit the control first; and detect_agent_drift.py compares only the 18 headings named in its SECTIONS_TO_COMPARE allowlist (:57-76), at an 80 percent similarity threshold (:668), whose exit code .github/workflows/drift-detection.yml:35-42 captures and discards. Anything under an unlisted heading is never compared at all: replacing all 22 agent names in the orchestrator's capability matrix in one install copy still reports "OK (100.0% similar)" and exits 0.
4. Commit with the discipline gates expect: 5 files or fewer per commit. Commit count itself is advisory only (needs-split label plus a WARNING/ALERT notice at 10/15 commits, via git rev-list --count HEAD ^origin/main; no PR can be blocked on commit count alone, ADR-099).
5. Escape hatches ([skip-drift-check], etc.) require documented justification and are cataloged in ai-agents-config-catalog. Using one IS the incident report; say so in the PR.

Traps That Cost Real Time

One line each; read the retro before repeating history. All paths relative to repo root.

Deeper history and the settled-battles list live in ai-agents-failure-archaeology. The 11-pattern failure catalog is .agents/governance/FAILURE-MODES.md (numbered sections 1-11, e.g. section 10 silent defaults, section 11 unrun generated artifacts).

Anti-Patterns

• Editing a generated tree to make a drift gate green. The gate reads difference, not direction; you may be destroying the source of truth.
• Reaching for an escape hatch ([skip-drift-check], etc.) as a first move. Escapes are for infrastructure failure, not friction.
• Debugging your branch before checking main. Reproduce-on-main is the cheapest discriminator in this playbook.
• Treating exit 2 and exit 143 as the same hook failure. They have different root causes and different fixes.
• Switching --cov between module-name and file-path forms to "fix" a coverage red. Only the module-name form resolves correctly here.
• Fixing by analogy ("the other CLI sets this env var, so this one probably does"). This repo's rule is empirical probe first; see ai-agents-empirical-probe-toolkit.
• Shotgun-fixing after the first hypothesis fails instead of running one discriminating experiment per hypothesis.

Verification

Before declaring the failure triaged and fixed:

• The symptom was matched to a table row, and the FIRST command output (not a guess) identified the cause
• The discriminating experiment was run and its outcome recorded in the session log or PR description
• The exact failing check passes locally (same command, same file set as CI), not a proxy
• The check was proven to still fail with a defect injected, so its pass is about your change and not vacuous
• No escape hatch was used; or if one was, the PR description says which, why, and links the infrastructure issue
• If the fix touched a canonical surface with mirrors, the drift command for that surface passes and source plus generated are in the same commit

Provenance and Maintenance

Verified against the working tree on 2026-07-03. Retro-cited short SHAs do not resolve locally even with full history present (~1471 commits as of 2026-07-03); use .agents/retrospective/ and .serena/memories/ for archaeology, not git log.

Maintenance: when a new recurring failure earns a retro, add a table row here with all five columns filled, and update ai-agents-failure-archaeology with the history. When any cited line number drifts, fix it on contact using the re-verify command.