Provider Session Ecosystem Audit

Provider Session Ecosystem Audit

Use when you need a cross-provider audit of actual AI work done in the repo, or when session-log observability is incomplete and needs to be strengthened.

When to use

• User asks for session log analysis across providers
• You need to compare Codex/Codex/Gemini/Hermes work patterns
• You need to explain missing-provider visibility gaps
• You are wiring recurring provider audit/report generation
• You need to export Gemini native sessions into repo-local orchestrator JSONL

Core approach

1. Start from repo-local orchestrator logs and existing saved audit artifacts.
2. Treat providers differently; do not assume one common native format.
3. Separate symbolic skill/tool reads from filesystem reads before reporting missing files.
4. Prefer raw logs over saved precomputed audit artifacts when both are available.
5. Add recurring wrappers/tests/docs in the same change so the audit becomes operational, not one-off.

Execution steps

1. Inventory actual data sources

Check these first:

• logs/orchestrator/README.md
• logs/orchestrator/<provider>/session_*.jsonl
• saved artifacts under analysis/ and docs/reports/
• existing exporters like:
  • scripts/cron/hermes-session-export.sh
  • scripts/cron/codex-session-export.sh
  • scripts/cron/comprehensive-learning-nightly.sh

Do not trust docs alone; compare against actual files in the checkout.

2. Build/patch the provider audit script

A reusable provider audit should:

• emit stable outputs:

  • analysis/provider-session-ecosystem-audit.json
  • docs/reports/provider-session-ecosystem-audit.md

• report per provider:

  • sessions
  • post-hook records
  • top tools
  • top repos
  • top reads
  • missing repo reads
  • remediation hints for stale repo reads
  • missing external reads
  • blank-read count
  • python3 vs uv run ... python

• emit an executive migration-debt summary derived from remediation-mapped stale reads

• treat provider-specific quirks correctly.

  • known migration debt per 1k records
  • rule-cluster count
  • top migration rule id / reads / share
  • provider rank by migration-debt density

• treat provider-specific quirks correctly.

Codex command normalization rule

Codex logs may encode commands as single characters separated by spaces. Do not flatten all whitespace. Use this rule instead:

• 1-2 spaces = encoding noise between characters
• 3+ spaces = actual token boundary

This preserves meaningful shell structure such as:

• &&
• pipes
• redirects like 2>/dev/null
• heredoc markers

If you flatten all whitespace, policy and usage analysis becomes misleading.

Required classification rules

• Hermes skill_view and session_search reads are often symbolic, not files
• slash-delimited symbolic names like coordination/workspace/repo-capability-map may be skills, not repo paths
• ~/.hermes/... and similar should be expanded and classified as external, not repo-local
• repo aliases like /mnt/workspace-hub/... may map to the current repo root and should be normalized
• Codex command logs may be stored with single-character spacing and must be normalized before policy checks
• If raw Codex logs exist at logs/orchestrator/Codex/session_*.jsonl, use them instead of stale saved Codex audit artifacts

3. Operationalize the audit

Add all three together:

• wrapper script, e.g. scripts/cron/provider-session-ecosystem-audit.sh
• scheduled task entry in config/scheduled-tasks/schedule-tasks.yaml
• lightweight tests for wrapper/schedule presence and output sanity

The wrapper should:

• run uv run --no-project python scripts/analysis/provider_session_ecosystem_audit.py
• verify JSON and Markdown outputs exist
• sanity-check required provider keys in the JSON output
• log to logs/quality/provider-session-ecosystem-audit-*.log

4. Update discoverability docs

4. Update discoverability docs

Update at least:

• logs/orchestrator/README.md

4. Update discoverability docs

Update at least:

• logs/orchestrator/README.md
• root README.md
• relevant AI docs index if one exists

Document:

• canonical command to run the audit
• canonical output paths
• scheduled task id
• input expectations per provider
• where stale-path redirects live (docs/ops/legacy-Codex-reference-map.md)
• which docs are intentionally allowed to mention legacy paths vs which should stay clean

When the audit reveals stale path drift, also patch the redirect surfaces that teach agents what to do instead. In this repo the highest-value targets were:

• docs/ops/legacy-Codex-reference-map.md
• docs/modules/ai/AGENT_EQUIVALENCE_ARCHITECTURE.md
• GEMINI.md
• .planning/templates/route-c-*.md
• .planning/skills/skill-knowledge-map.md

Then add docs regression tests so deleted paths cannot silently spread back into current planning/onboarding docs.

Gemini native session export

Gemini is the tricky provider.

Native sources

Do not assume only one path. Check both:

• ~/.gemini/tmp/<repo-name>/chats/session-*.json
• ~/.gemini/tmp/<projectHash>/chats/session-*.json

Filter by:

• projectHash == sha256(abs_repo_path)

This matters because docs may mention only the repo-name path while actual sessions can live in the hash-based path too.

Export target

Write to:

• logs/orchestrator/gemini/session_YYYYMMDD.jsonl

Export record shape

For each Gemini tool call, emit records with at least:

• ts
• hook: post
• tool
• gemini_tool
• project
• repo
• model
• session_id

Useful extras:

• project_hash
• tool_status
• native_session_file
• session_kind
• session_summary
• tool_call_id
• cmd / file / query / todo_count / error

Recommended Gemini tool mapping

• run_shell_command -> Bash
• read_file -> Read
• list_directory -> Read
• write_file -> Write
• replace -> Edit
• grep_search -> Grep
• glob -> Grep
• search_file_content -> Grep
• google_web_search -> Browser
• write_todos -> Write
• codebase_investigator -> ToolSearch
• cli_help -> ToolSearch
• leave interactive tools like ask_user unmapped unless you intentionally add a user-input category

Critical dedupe rule

Gemini native session JSON files are mutable and get rewritten as the conversation grows. Do NOT use a simple mtime-only append exporter.

Instead, persist state such as:

• logs/orchestrator/gemini/.export-state.json
• keyed by sessionId
• storing exported_tool_call_ids

Preferred dedupe key:

• native toolCall.id
• fallback to a fingerprint of session id + timestamp + tool name + args

Without this, repeated exports will duplicate historical Gemini tool calls.

Verification checklist

After implementing or patching:

1. Run targeted pytest suites for the audit and wrapper/export scripts.
2. Validate scheduled tasks with the repo validator.
3. Run exporter in --dry-run first.
4. Run real exporter.
5. If using --all, make sure the exporter clears old JSONL/state first so counts are not duplicated.
   • This rule applies to Hermes, Codex, and Gemini exporters.
6. Regenerate the provider audit.
7. Confirm the audit now shows real provider coverage and low/no blank-read noise.
8. Confirm the report now includes per-provider Bash command-family summaries, not just raw Bash totals.

Hermes exporter-specific rules

To avoid false blank reads:

• skill_view -> Read and export file + skill_name
• skills_list -> ToolSearch and export file + skill_category
• session_search -> Grep and export:
  • file=__session_history__
  • search_query
  • role_filter
  • limit
• Export session_id so Hermes runtime-session counts are meaningful
• When rerunning with --all, clear prior generated Hermes JSONL/state first so counts do not inflate

This change can reduce Hermes blank reads from hundreds to effectively zero.

Codex exporter-specific rules

Codex native rollout files are mutable and often get rewritten as the conversation grows. Do NOT rely on file mtime alone for incremental export.

Instead:

• use uv run --no-project python, not ad-hoc python3 -c
• persist state in logs/orchestrator/codex/.export-state.json
• key state by native rollout file path or native session id when available
• store exported_tool_call_ids
• prefer native payload ids such as call_id / id
• fall back to a fingerprint of session file + session id + timestamp + tool name + args

Export at least:

• native_session_file
• tool_call_id
• session_id
• codex_tool
• cmd / file / query / search_root where applicable

Without per-tool-call dedupe, rerunning the exporter after Codex appends new function calls will duplicate the already-exported history and distort provider comparisons.

Migration-debt remediation summaries

When stale repo reads cluster around known removed workflow surfaces, add redirect intelligence directly into the audit output rather than only listing missing files.

Recommended pattern:

• maintain a small static rule catalog in the audit script
• group top missing repo reads into reusable remediation clusters
• include for each cluster:
  • rule_id
  • matched_paths
  • total_count
  • canonical_targets
  • guidance
  • reference_doc
• expose this per provider as missing_repo_read_remediation_hints
• render a markdown subsection such as ### <provider> remediation hints for stale repo reads

High-value legacy clusters seen in practice:

• old scripts/work-queue/* transition scripts -> governance docs/hooks
• old generate-html-review.py -> scripts/review/cross-review.sh + current review evidence/template surfaces
• old work-queue lifecycle scripts -> scripts/refresh-agent-work-queue.*, notes/agent-work-queue.md, .planning/, GitHub issues
• removed work-queue skills -> AGENTS.md, GSD command/workflow surfaces
• removed scripts/agents/* wrapper tree -> AGENTS.md, current architecture docs, review/planning entrypoints
• local .Codex/work-queue/* item files -> GitHub issues + .planning/ as canonical source of truth

Also add an executive-summary migration-debt ranking using remediation-mapped stale reads:

• known_migration_debt_reads
• known_migration_debt_per_1k_records
• known_migration_debt_rule_count
• top_migration_rule_id
• top_migration_rule_reads
• top_migration_rule_share_pct
• migration_debt_rank

Rank providers primarily by known_migration_debt_per_1k_records, then by mapped debt volume. Label the scope clearly: this is a bounded/actionable migration-debt view derived from remediation-mapped entries in top missing repo reads, not a full census of all missing reads.

Migration-debt and stale-path remediation summaries

After computing top_missing_repo_reads, add a remediation layer:

• match known stale paths/prefixes to canonical redirect rules
• group matched stale paths into remediation clusters
• emit per-provider missing_repo_read_remediation_hints
• include:
  • rule_id
  • total_count
  • matched_paths
  • canonical_targets
  • guidance
  • reference_doc

Then build an executive-summary migration-debt ranking using remediation-mapped stale reads:

• known_migration_debt_reads
• known_migration_debt_per_1k_records
• known_migration_debt_rule_count
• top_migration_rule_id
• top_migration_rule_reads
• top_migration_rule_share_pct
• migration_debt_rank

Important scope note:

• this metric is intentionally bounded to remediation-mapped stale reads from the provider's top missing repo reads, not all missing-path drift

This turns the audit from a dead-file leaderboard into an actionable migration-debt report.

Doc guardrails to keep drift from returning

After cleaning high-value stale references in tracked planning/docs files, add regression tests such as:

• a strict banned-reference test for curated instructional files
• an allowlist-based test proving stale-path mentions remain confined to explicit legacy/reference docs only

Good first strict targets:

• .planning/templates/route-c-*.md
• docs/context-pipeline.md

Good allowlisted legacy/reference docs:

• GEMINI.md
• docs/work-queue-workflow.md
• docs/ops/legacy-Codex-reference-map.md
• docs/modules/ai/AGENT_EQUIVALENCE_ARCHITECTURE.md

Bash command-family summaries

After provider-specific command decoding, add a lightweight cleanup pass before prefix extraction:

• drop blank/comment-only leading lines
• strip simple cd ... && wrappers

Then bucket commands by family using multi-word prefixes such as:

• git diff
• git status
• uv run
• python -m
• python3 -m

Report per provider:

• top 8 Bash command families
• count
• share of Bash calls
• example command in JSON output

This is much more useful than a raw Bash tool count alone.

Example high-value verification sequence:

• bash scripts/cron/gemini-session-export.sh --dry-run
• uv run pytest tests/analysis/test_provider_session_ecosystem_audit.py tests/cron/test_provider_session_ecosystem_audit_wrapper.py tests/cron/test_gemini_session_export.py
• uv run --no-project python scripts/cron/validate-schedule.py
• bash scripts/cron/gemini-session-export.sh
• bash scripts/cron/provider-session-ecosystem-audit.sh

Pitfalls

• Docs may claim Codex raw logs are absent when they are actually present in the checkout
• Hermes symbolic skill names can massively inflate false missing-file counts if treated as paths
• Codex spaced-command logging can hide python3 policy violations unless normalized first
• Gemini cross-review .log files are not enough for parity analysis; you need exported native sessions
• Gemini exporter must scan both repo-name and project-hash native directories
• Gemini exporter must dedupe per tool call, not just per file timestamp

Regression guardrails for stale-path drift

After the audit starts surfacing recurring stale repo reads, add lightweight regression tests so cleaned docs/templates do not drift back toward deleted workflow paths.

Recommended pattern:

• create a shared helper under tests/helpers/ that centralizes banned stale-path regexes and a file scanner
• add a strict curated docs test for high-value instructional files (for example Route-C templates and current workflow docs)
• add an allowlist-based docs test proving stale-path mentions are confined to explicit legacy/reference docs only
• expand the allowlist-scan over time as more live docs are cleaned
• add an audit test proving remediation hints still cover the highest-value stale families:
  • scripts/work-queue/*
  • scripts/agents/*
  • .Codex/work-queue/*
  • old work-queue skill paths

This turns the audit from passive reporting into an active anti-regression system.

Executive migration-debt summary

Once remediation hints exist, add a compact executive-summary ranking so the worst provider drift rises to the top immediately.

Recommended derived fields:

• known_migration_debt_reads
• known_migration_debt_per_1k_records
• known_migration_debt_rule_count
• top_migration_rule_id
• top_migration_rule_reads
• top_migration_rule_share_pct
• migration_debt_rank
• migration_debt_status

Preferred ranking:

• sort by known_migration_debt_per_1k_records descending
• tie-break with total known mapped debt, then top-rule volume

Important scope note:

• label this as remediation-mapped / top-missing-read migration debt, not a full census of all missing-path drift

Outcome to aim for

A good implementation leaves the repo with:

• stable cross-provider audit artifacts
• recurring scheduled refresh
• corrected provider-specific read classification
• raw Gemini session coverage in repo-local orchestrator logs
• remediation hints for stale repo reads in both JSON and Markdown outputs
• executive migration-debt ranking across providers
• docs/tests guardrails that prevent cleaned workflow surfaces from reintroducing deleted paths
• tests proving the wrapper/export/report path remains intact