Chat Analysis

Chat Analysis — Recursive Self-Improvement

Analyze session transcripts from Claude (~/.claude/projects/) or Codex (~/.codex/sessions, ~/.codex/archived_sessions) to surface failure patterns, rule violations, and candidate improvements to agent instructions.

The pipeline:

1. Parse — read .jsonl session files
2. Filter — regex pre-filter drops clean sessions (no friction signals)
3. Label — survivors go to Sonnet with a strict JSON schema
4. Compile — aggregate into tables, proposed deltas written to a review file

When to use

• "Analyze my chats" / "run chat analysis"
• "What patterns are in my Claude/Codex sessions?"
• "What rules should I add to my agent instructions?"
• "How many sessions had verification failures?"
• Weekly improvement cadence: run on the last 50–100 sessions
• Monthly /insights cadence: review recent sessions for skills, memory, and automation candidates rather than one-off bug fixes

Setup (verify once)

# Check API key
echo $ANTHROPIC_API_KEY | head -c 10

# Install dependency (if needed)
pip install anthropic
# or: .venv/bin/pip install anthropic

If ANTHROPIC_API_KEY is not set, remind the user it's required and stop.

Workflow

Step 1 — Dry run first

Always do a dry run to confirm there are friction sessions worth labeling:

python3 scripts/chat_analysis.py --dry-run

This parses and pre-filters without spending any API tokens. Report what the filter found. If zero friction sessions: tell the user and stop — the sessions look clean.

Step 2 — Cost check

Estimate cost before running. Each surviving session is one labeling API call. The model is configurable with --model or CHAT_ANALYSIS_MODEL; check current provider pricing before quoting a dollar estimate. For the default run (last 50 sessions), proceed unless the user has flagged the project as sensitive or cost-constrained. For --all or --n values above 200, confirm with the user first.

Step 3 — Run the analysis

# Default: current project, last 50 sessions
python3 scripts/chat_analysis.py

# Codex sessions for the current working directory
python3 scripts/chat_analysis.py --source codex

# Last 100 sessions
python3 scripts/chat_analysis.py --n 100

# All projects
python3 scripts/chat_analysis.py --all

# Custom output path
python3 scripts/chat_analysis.py --output /tmp/review.md

# Heuristic-only — no API calls, nothing leaves the machine
python3 scripts/chat_analysis.py --no-label

Stream output so the user sees progress. Report the final count: X sessions parsed → Y passed filter → Z labeled.

Step 4 — Surface findings

After the script completes, read chat-analysis-review.md and summarize the top 3 findings to the user directly in the conversation:

# The report is self-contained, but surface the headline numbers
head -80 chat-analysis-review.md

Tell the user:

• The #1 failure mode by count
• The highest-cost failure mode (most rework cycles per incident)
• The most-violated rule (if any)
• How many candidate instruction deltas are in the review file

Step 5 — Review deltas (user decision)

The review file has a ## Proposed Agent-Instruction Deltas section with checkbox items. Walk the user through them one at a time if they want, or point them to the file. Never auto-apply any delta to agent instructions — the user decides.

If the user says "accept all" or "apply them", ask them to confirm which specific items they want and make only those edits. The model does not edit its own operating rules autonomously.

Monthly /insights cadence

In dans-brain, the Mac LaunchAgent launchd/com.dansbrain.monthly-agent-insights.plist runs this pipeline monthly against recent Codex sessions and writes:

state/monthly-agent-insights.md

Use that report as an improvement backlog for skills, memory notes, and automation candidates. Do not auto-apply proposed instruction deltas; Dan still decides which changes become durable operating rules.

Output file

chat-analysis-review.md (or the path passed via --output) contains:

• Summary stats table
• Top failure modes by count
• Highest-cost failure modes by avg rework cycles
• Communication antipatterns
• Rule violation rankings
• Proposed agent-instruction deltas (checkbox list, user must decide)
• Sample corrections and positive signals
• Per-session detail table

The file is ephemeral — not committed, not deployed. It's a scratchpad for the improvement session. Delete it when done.

What the pipeline looks for

Five friction patterns (from the blog post that describes this workflow):

1. Corrections — "no", "wrong", "stop", "that's not right"
2. Repeated instructions — same fix asked for across multiple turns
3. Praise of non-obvious choices — "yes, exactly, keep doing that"
4. Tool loops — same tool called 3+ times before switching
5. Clarifying questions — model had to ask because context was incomplete

Privacy & data security

Transcripts can contain code, paths, customer data, and secrets. The pipeline is hardened against leakage:

Automatic redaction before any data leaves the machine:

• API keys (Anthropic sk-ant-…, OpenAI sk-…, GitHub ghp_/gho_/ghs_, AWS AKIA…, Slack xox[baprs]-…)
• JWTs, Bearer tokens, Authorization: headers
• password=, secret=, api_key=, access_token= assignments
• Email addresses
• Absolute home paths (/Users/<name> → /Users/[REDACTED])
• Credit-card-shaped numbers

Redaction runs in redact() (scripts/chat_analysis.py). Update the REDACTION_RULES list to add patterns specific to your environment (internal hostnames, customer IDs, project codenames, etc.).

Other safeguards:

• Session IDs in the report are SHA-256 hashed (8 chars), not raw filenames
• Sample corrections / positives are redacted AND truncated to 160 chars
• Default scope is current project only — never --all without auditing
• --no-label produces a heuristic-only report with zero API calls; use this for any project with stricter confidentiality requirements
• The output report is gitignored by default

Before running on a sensitive project, do a dry run and skim the friction session list. If anything looks like it could leak, run with --no-label or extend REDACTION_RULES first.

Troubleshooting

"Could not auto-detect project dir" For Claude, the script maps CWD → ~/.claude/projects/ by hashing the path. If it fails, pass the path explicitly:

ls ~/.claude/projects/     # find the right directory name
python3 scripts/chat_analysis.py --project ~/.claude/projects/<dir>

For Codex, auto-detection reads recent sessions from ~/.codex/sessions and ~/.codex/archived_sessions, then filters by the session cwd. If the session is older than the default window, use --source codex --all --n <larger N> or pass a specific transcript directory with --project.

"ANTHROPIC_API_KEY not set"

export ANTHROPIC_API_KEY=sk-ant-...
python3 scripts/chat_analysis.py

"anthropic package not installed"

pip install anthropic

Labeling errors / JSON parse failures These are logged as warnings and skipped. The report includes only sessions that labeled successfully. A handful of failures is normal; if >20% fail, check the model response in debug mode.