arndvs Error Audit
If running interactively (human present), output "Read Error Audit skill." to acknowledge. If running with --dangerously-skip-permissions (AFK/unattended), skip acknowledgement and proceed directly.
Scan every Claude Code session transcript for errors, cluster by root cause, surface the top offenders with suggested remediations.
When to invoke
- Periodically (every 5-10 sessions) as a health check
- When the user notices "this keeps happening"
- After a session with 3+ retry loops on the same error
- When proposing a new hook or rule (to justify it with data)
Usage
# Default: all sessions, top 20 clusters, human output
python3 skills/error-audit/error-audit.py
# Last 30 days only
python3 skills/error-audit/error-audit.py --since 30
# Show fewer clusters
python3 skills/error-audit/error-audit.py --top 10
# Machine-readable (for piping into other tools)
python3 skills/error-audit/error-audit.py --json
# Override the projects dir (useful for testing)
python3 skills/error-audit/error-audit.py --projects-dir /path/to/projects
# Override or disable suppressions
python3 skills/error-audit/error-audit.py --suppressions-path /path/to/suppressions.md
python3 skills/error-audit/error-audit.py --no-suppressions
python3 skills/error-audit/error-audit.py --show-suppressed
Steps (when invoking via /error-audit)
- Run
python3 <skill-path>/error-audit.pywith any arguments the user provided. - Display the output to the user exactly as printed (includes colour-coded counts and suggested remediation tiers).
- For the top 3 clusters, propose a concrete action:
- Tier 1 (settings allowlist): show the exact
~/.claude/settings.jsonentry to add, but do NOT auto-apply — user must review Bash allowlist changes. - Tier 2 (hook or script fix): locate the hook/script and propose the edit.
- Tier 3 (instruction/memory): draft the feedback or mechanism memory entry.
- Tier 1 (settings allowlist): show the exact
- Do NOT auto-apply any remediation. This skill surfaces and proposes; the user decides.
Error Classes
| Class | Signal | Example |
|---|---|---|
tool_error |
is_error:true on a tool result (excluding denials) |
Read/Write/Edit failures |
validation_error |
InputValidationError in tool result |
Wrong parameter types |
permission_denial |
"The user doesn't want to proceed" | Bash command rejected |
hook_block |
hook_failure attachment type |
git-workflow-gate blocks |
bash_fail |
Non-zero exitCode on Bash attachment |
Command failures |
retry_storm |
3+ overloaded_error records for the same tool on the same day |
API overload |
read_before_edit |
"File has not been read yet" | Edit without prior Read |
Remediation Tiers
| Tier | Fix type | Example |
|---|---|---|
| 1 | Settings allowlist | Add Bash(cmd:*) to allow list (review first) |
| 2 | Hook or script fix | Tune a hook to exclude a known-good pattern |
| 3 | Instruction/memory | Add behavioral memory covering the failure mode |
Configuration
| Env var | Purpose | Default |
|---|---|---|
CLAUDE_ERROR_AUDIT_SUPPRESSIONS |
Suppressions file path | <skill-dir>/suppressions.md |
Suppressions
The suppressions.md file (ships alongside error-audit.py) marks known "working-as-designed" cluster keys. Default human output hides them; --show-suppressed re-includes them; --json always includes them with a suppressed: true|false field.
Fallback Method (when Python is unavailable)
If python3 is not available, fall back to manual analysis:
Phase 1 — Gather error data
# Recent fix commits
git log --oneline --since="2 weeks ago" --grep="fix" --grep="revert" | head -30
# Find recent session files
find ~/.claude/projects/ -name "*.jsonl" -mtime -14 2>/dev/null | head -10
# HUD events
if [[ -f "$HOME/dotfiles/working/logs/events.jsonl" ]]; then
grep -i "error\|fail\|block" "$HOME/dotfiles/working/logs/events.jsonl" | tail -30
fi
Phase 2 — Classify
Group errors into: Tooling, Convention, Environment, Architecture, External.
Phase 3 — Score
Frequency × Cost × Fixability. Minimum 3 occurrences before calling "systemic."
Phase 4 — Recommend actions
One specific action per pattern. For each high-priority pattern, recommend ONE specific action:
| Action Type | When to use | Example |
|---|---|---|
| New hook | Repeated mechanical error that can be detected before it happens | "Add a hook that checks Node version on SessionStart" |
| New rule | Convention violation that the model keeps forgetting | "Add a rule that requires test file for every new module" |
| Architecture fix | Structural issue causing cascading failures | "Extract shared types to a package" |
| Documentation | Misunderstanding of how something works | "Document the auth flow in README" |
| Ignore | External/transient issue not worth automating | "GH API rate limits during peak hours" |
Report Format
## Error Audit Report
**Period**: Last 2 weeks (N sessions analyzed)
**Patterns found**: M
### High Priority
| # | Pattern | Freq | Cost | Action |
|---|---------|------|------|--------|
| 1 | TypeScript path alias misconfigured | 4 sessions | ~10 min each | Rule: verify tsconfig paths on project start |
| 2 | Commit to main without PR | 3 sessions | 5 min each | Hook: ✅ already fixed (git-workflow-gate) |
### Medium Priority
...
### Resolved by existing mechanisms
- [pattern] → resolved by [hook/rule name]
### Recommendations
1. [Specific action with a one-liner justification]
2. ...
Integration
After producing recommendations:
- If a new hook is recommended: Use the
hooks/PROPOSAL_TEMPLATE.mdto draft it - If a new rule is recommended: Create in
rules/following existing patterns - If resolved: Update
hooks/COVERAGE_MATRIX.mdto reflect new coverage
Constraints
- Only analyze data that actually exists — do NOT fabricate patterns from thin air
- If session data is unavailable or incomplete, say so explicitly
- Minimum 3 occurrences before calling something "systemic"
- Prefer conservative recommendations — one well-placed hook over five speculative ones