Codex Sessions Skill Scan

Codex Sessions Skill Scan

Overview

This skill runs a daily scan over Codex session logs (global ~/.codex/sessions and per-repo sessions under ~/dev) to catch repeated “paper cuts” when using skills (broken file paths, missing scripts, validation commands that don’t run) and produces a short report + suggested fixes. It also highlights user words that often indicate complex tasks and adds a step-by-step reminder.

Scope and triggers

• “Scan yesterday’s Codex sessions for skill failures (personal skills only).”
• “Why does $some-skill keep failing? Look at recent session logs and tell me what’s wrong.”
• “Daily check: any broken skill references or validation commands in the last 24 hours?”

Quick start (daily)

Run:

cd ~/dev/agent-skills
python3 utilities/codex-sessions-skill-scan/scripts/scan_codex_sessions.py --days 1 --include-otel

Philosophy

• Single-threaded: scan → summarize → (optional) patch only the smallest diff.
• Evidence-led: cite the specific error snippets + point to the exact personal SKILL.md path when possible.
• Safety-first: redact secrets; do not copy full transcripts.

Required inputs

• --days <float>: how far back to scan (default 1).
• --sessions-root <path>: where to scan (default ~/.codex/sessions).
• --include-dev-project-sessions (enabled by default): also scan per-project .codex/sessions under ~/dev. Use --no-include-dev-project-sessions to disable.
• --max-samples-per-skill <int>: cap snippets per skill (default 3).
• --include-otel: include best-effort OTel signals (Codex OTLP endpoint listening status + repo-local OTLP-derived trace artifacts under .narrative/trace/).
• --include-otel-collector: include summary from ~/.agents/otel-collector/data/processed/stats.json.
• --otel-collector-stats <path>: override collector stats path (default ~/.agents/otel-collector/data/processed/stats.json).
• --codex-config-toml <path>: path to read Codex [otel] endpoints from (default ~/.codex/config.toml).

Deliverables

• A “daily skill health report” in Markdown (skills invoked, skills with issues, sample error snippets).
• A list of suggested fix patterns (no changes applied).

Constraints

• Personal skills only: do not patch skills-system/ or .system skills.
• Do not auto-install dependencies or change system settings as part of scanning.
• Keep output small (snippets only); avoid dumping raw logs.
• Redact secrets/sensitive data by default; never paste tokens/keys from logs into chat or files.

Reliability hardening (from recurring failures)

• rg/fd preflight: before any repo-wide search commands, run command -v rg and command -v fd. If missing, report the missing binary and stop (or use absolute paths such as /opt/homebrew/bin/rg and /opt/homebrew/bin/fd when available).
• No direct network curl: do not run external curl commands in this workflow. Use local scripts/MCP tools instead.
• TTY for interactive/streaming commands: if a command expects stdin or may run interactively (for example auth/login flows), run it with tty=true; otherwise avoid write_stdin follow-ups.
• Claude auth check: when claude_projects emits auth failures, verify with claude auth status and treat loggedIn=false as environment/auth state, not a skill regression.

Procedure

A) Scan (read-only)

1. Run the scan script (Quick start).
2. If exit code is 2, issues were found (see report).

B) Triage (human-in-the-loop)

For each flagged skill:

1. Identify if it’s a personal skill (lives under ~/dev/agent-skills/ and is not under skills-system/).
2. Confirm whether the failure is:
   • wrong file path in a SKILL.md reference
   • wrong interpreter for validation scripts (PyYAML required)
   • ~ not expanding (needs absolute path)
   • missing/moved file in the repo

C) Patch (optional; requires explicit user request)

If the user asks to apply fixes:

1. Make the smallest possible edits to the referenced personal skill files.
2. Re-run skill gates (fail fast on first failure):

cd ~/dev/agent-skills
~/.venvs/pyyaml/bin/python utilities/skill-creator/scripts/quick_validate.py <skill-folder>
~/.venvs/pyyaml/bin/python utilities/skill-creator/scripts/skill_gate.py <skill-folder>

Guardrails (non-negotiable)

• Personal skills only: do not patch skills-system/ or .system skills.
• Redact: do not paste secrets/tokens from session logs into chat, files, or issues.
• Do not auto-fix: always propose patches first; only apply after confirmation.

Anti-patterns

• Editing skills automatically just because the scan found an issue.
• “Fixing” system skills (skills-system/, .system) when the user asked for personal skills only.
• Copy/pasting full session logs into chat/issues (high risk for secrets/PII).

Examples

• “Scan my Codex sessions from the last day and tell me if any skills are failing. Personal skills only.”
• “Why does $product-spec keep referencing missing files? Scan yesterday’s sessions and suggest the smallest fix.”
• “Daily scan: any broken validation commands or missing paths from the last 24 hours?”

Validation

• Fail fast: stop at the first failed gate, fix it, then re-run.
• This skill’s scan script is stdlib-only; run it with python3.
• When changing skill files, validate with:
  • ~/.venvs/pyyaml/bin/python utilities/skill-creator/scripts/quick_validate.py utilities/codex-sessions-skill-scan
  • ~/.venvs/pyyaml/bin/python utilities/skill-creator/scripts/skill_gate.py utilities/codex-sessions-skill-scan

References used by skill-gate:

• references/contract.yaml
• references/evals.yaml

Philosophy and tradeoffs

• Use this skill when consistent decision-making matters more than one-off execution because project context should drive the approach.
• Principle and mindset: prioritize tradeoffs and constraints over rigid checklists; understand why each step exists.
• Ask this to keep outcomes robust: Why is this the right default, and what could change this outcome?
• How do we adapt if constraints shift?
• What evidence is needed before choosing one path over another?

Anti-patterns and caveats

• Avoid applying this playbook generically without checking repository-specific context.
• NEVER skip required validation gates when behavior changes.
• DO NOT use this skill as a rigid replacement for engineering judgment.
• DON'T ignore warnings or assume one pattern fits all repos.
• Common pitfall: treating anti-patterns as optional.
• Incorrect assumptions here can lead to fragile guidance.
• Warning: wrong sequencing can create avoidable regressions.

Variation and adaptation

• Vary the workflow by team size, risk, and deployment target.
• Use different strategies for small, medium, and large changes.
• Adapt recommendations to the specific environment and avoid repetitive templates.
• Avoid generic or cookie-cutter responses; craft context-specific alternatives.
• Keep outputs diverse and not repetitive.
• Converge on a custom path only after evidence review.
• Different constraints should produce different, non-generic recommendations.

Empowering execution style

• Be capable of exploring multiple options and enabling the team to make safe decisions.
• Unlock confidence by explaining options and tradeoffs clearly.
• Feel free to be creative while staying rigorous and precise.
• Push boundaries with practical alternatives when simple recipes fail.
• Enable outcomes-oriented problem solving.