Provider Session Ecosystem Audit And Exporters

Provider Session Ecosystem Audit and Exporters

Use when you need to audit AI-provider activity across the workspace-hub repo, improve session-log observability, or add/fix exporter pipelines for Codex/Codex/Hermes/Gemini.

When to use

• Cross-provider session quality / drift analysis
• Building or fixing logs/orchestrator/<provider>/session_*.jsonl
• Reducing false positives in read-path audits
• Adding recurring audit jobs and report artifacts
• Verifying shell exporters with behavioral subprocess tests

Core approach

1. Prefer raw provider logs over saved precomputed reports when both exist.
2. Treat symbolic tool/skill reads separately from filesystem paths.
3. Normalize provider-specific command encodings before deriving Bash metrics.
4. Add exporter behavioral tests using temp repo + temp HOME + fake shims.
5. Regenerate audit artifacts after any exporter/schema change.

Key implementation rules

1. Codex

• Raw Codex orchestrator logs live at logs/orchestrator/Codex/session_*.jsonl.
• If raw logs exist, audit them directly; only fall back to saved Codex audit JSON when raw logs are absent.
• Historical Codex raw logs may not persist session_id, so unique_runtime_sessions can be unavailable on old corpus slices; report that limitation explicitly instead of inventing heuristics.
• The minimal safe producer-side fix is in .Codex/hooks/session-logger.sh:
  • parse .session_id // "" from hook stdin JSON
  • append optional top-level session_id to the emitted ENTRY object
  • keep existing fields and both write paths unchanged
• Once this patch is active, new Codex raw logs can support real unique_runtime_sessions; old logs remain legacy/no-session-id.

2. Codex

• Codex log commands are often stored in a spaced-character encoding.
• Do not strip all whitespace blindly.
• Decode only when the payload actually looks like spaced-character encoding.
• Preserve normal commands such as git status --short.
• Decode with this rule:
  • single/double spaces = encoding noise between characters
  • runs of 3+ spaces = one real token boundary
• After decoding, derive command families from the reconstructed shell string.
• For exporter tests, use a fake python3 shim and copy scripts/lib/python-resolver.sh into the temp repo.
• For --all behavior, clear old exported JSONL/state before re-exporting to avoid duplicate append inflation.
• Exporter must handle native Codex command payloads in all observed shapes:
  • command as string
  • cmd as string
  • cmd as list
• Map Codex MCP/GitHub fetch/search tool variants into semantic categories where possible so Codex is not falsely read-light:
  • file/resource fetch variants -> Read
  • search/query variants -> Grep
• Capture semantic fields such as file, query, search_root, or resource when present.

3. Hermes

• Hermes exporter false blank reads came primarily from:
  • session_search
  • skills_list
• Export them as semantic non-file search/discovery events:
  • session_search -> Grep
    • file = __session_history__
    • search_query, role_filter, limit
  • skills_list -> ToolSearch
    • file = category or __all__
    • skill_category
  • skill_view -> Read
    • file = skill name
    • skill_name
• Always export session_id from Hermes sessions.
• Hermes --all should remove prior session_*.jsonl, correction files, and state before rebuilding.
• Behavioral test should verify:
  • main JSONL output
  • correction JSONL output
  • repeated patch/write to same file creates one correction record

4. Gemini

• Native Gemini sessions may exist under both:
  • ~/.gemini/tmp/<repo-name>/chats/session-*.json
  • ~/.gemini/tmp/<sha256(abs_repo_path)>/chats/session-*.json
• Filter sessions by projectHash == sha256(real repo root path).
• Gemini session files are mutable, so dedupe by tool_call_id (or stable fingerprint fallback) in .export-state.json.
• Safe tool mappings discovered:
  • run_shell_command -> Bash
  • read_file, list_directory -> Read
  • write_file, write_todos -> Write
  • replace -> Edit
  • grep_search, glob, search_file_content -> Grep
  • google_web_search -> Browser
  • codebase_investigator, cli_help -> ToolSearch
• Do not force-map ask_user unless you intentionally introduce a user-input category.
• Gemini --all should clear exported JSONL/state before rebuilding.

Shared normalization helper

Create and reuse a shared helper for Bash family extraction, e.g. scripts/bash_command_prefixes.py.

It should provide:

• cleanup_bash_command(command)
  • remove blank/comment-only leading lines
  • strip cd ... && wrappers
  • strip wrapper/setup lines that are not the real command
  • strip leading and inline env-assignment prefixes such as WT=... or GIT_PAGER=cat
• normalize_command_to_prefix(command, cleanup=False)
  • multi-word prefixes like git diff, uv run, python -m, python3 -m

Use cleanup=True in the provider audit, but keep cleanup optional for permissions/allowlist tooling so behavior stays explicit.

Report design

Separate event-time activity from corpus-shape churn.

Executive summary blocks to maintain

Executive summary blocks to maintain

Keep these machine-readable summary blocks in analysis/provider-session-ecosystem-audit.json:

• recent_activity_since_previous_audit
• recent_activity_windows
• corpus_change_since_previous_audit
• provider_interpretation_summary
• executive_actions

As the executive layer matures, keep these compact sub-blocks inside provider_interpretation_summary and mirror them into executive_actions when they are intended for operators/automation:

• recommended_actions
• rank_movements
• health_overview
• watchlist
• change_alerts
• remediation_playbooks
• followup_issue_drafts

Within provider_interpretation_summary / executive_actions, maintain compact operator/automation surfaces:

• recommended_actions
• rank_movements
• health_overview
• watchlist
• change_alerts

Use them for different operator questions:

• recent_activity_since_previous_audit Use them for different operator questions:
• recent_activity_since_previous_audit
  • What actually happened since the last audit by event timestamp?
• recent_activity_windows
  • What happened in the rolling last_24h and last_7d windows, and is a provider bursty vs sustained?
• corpus_change_since_previous_audit
  • Did the exported corpus grow, shrink, or get rebuilt/backfilled?
• provider_interpretation_summary
  • Which provider needs attention and why?
• executive_actions
  • What should downstream automation or the operator do next?

For recent_activity_windows, keep at least:

• generated_at
• status
• windows.last_24h
• windows.last_7d

Each window should expose:

• window_start
• window_end
• status
• scope_note
• providers
• ranked_providers

Per-provider window stats should mirror the event-time recent-activity block where possible:

• post_records
• sessions
• top_tools
• top_missing_repo_reads
• top_bash_command_families

Provider interpretation summary

For each provider, include compact operator-facing fields such as:

• urgency_score
• urgency_tier
• urgency_rank
• previous_urgency_rank
• urgency_rank_delta
• urgency_rank_direction
• urgency_score_delta
• urgency_movement_summary
• activity_status
• activity_trend
• corpus_status
• debt_status
• debt_trend
• drift_trend
• python_hygiene_status
• python_hygiene_trend
• activity_window_profile
• last_24h_post_records
• last_24h_sessions
• last_7d_post_records
• last_7d_sessions
• health_status
• health_reasons
• health_summary
• primary_issue
• recommended_action
• recommended_actions
• missing_repo_reads_per_1k_records

For the operator-action layer, also expose:

• watchlist trigger metadata:
  • trigger_level
  • trigger_reason
  • suggested_followup
• change-detection metadata:
  • change_type
  • previous_trigger_level
  • current_trigger_level
  • previous_health_status
  • current_health_status
  • previous_urgency_rank
  • current_urgency_rank
  • summary
• remediation routing metadata:
  • inspect_paths
  • canonical_targets
  • first_steps
  • reference_doc
  • guidance
  • owner_surface
  • owner_team
  • preferred_fix_lane
• follow-up issue draft metadata:
  • title
  • severity
  • body

Sort provider rows by descending urgency_score so markdown and JSON agree on priority.

Markdown sections to render

Include these report sections:

• ## Provider interpretation summary
• Focus this week:
• Recommended actions:
• ## Recent activity since previous audit
• ## Corpus change since previous audit

Render provider rows compactly with inline trend annotations, for example:

• activity=active (stable)
• debt=high_debt (improving)
• python=uv_preferred (stable)

Report design

For each provider, keep these sections:

• top tools
• top repos
• top reads
• top symbolic reads
• top Bash command families
• top missing repo reads
• top missing external reads

For Bash command families, include:

• prefix
• count
• share_of_bash_calls
• example_command in JSON artifact (markdown can stay compact)

For the executive provider interpretation layer, expose both human-readable and machine-readable movement context across audits:

• per-provider fields:
  • urgency_rank
  • previous_urgency_rank
  • urgency_rank_delta
  • urgency_rank_direction
  • urgency_score_delta
  • urgency_movement_summary
  • activity_window_profile
  • last_24h_post_records
  • last_24h_sessions
  • last_7d_post_records
  • last_7d_sessions
  • health_status
  • health_reasons
  • health_summary
• compact executive block:
  • focus_this_week
  • recommended_actions
  • rank_movements
  • health_overview
  • watchlist
  • change_alerts
  • remediation_playbooks

Guidance:

• derive rank movement from the previous audit's provider_interpretation_summary.providers ordering, not from ad hoc score sorting outside the tracked artifact

• explain movement using the already-derived trends (activity_trend, debt_trend, drift_trend, python_hygiene_trend) plus corpus anomalies

• add rolling event-time windows for last_24h and last_7d using the same raw-log filtering logic as the since-previous-audit block; keep them separate from snapshot/corpus reconciliation logic

• derive a compact activity_window_profile such as dormant, burst_active, sustained_background, or light_recent

• derive health_status from urgency tier + debt/drift/python signals + rolling-window profile so the report has an immediate red/yellow/green operator surface

• derive watchlist from health/urgency/activity-profile conditions and keep levels compact (page, act_this_week, investigate, monitor)

• derive change_alerts only for meaningful state transitions versus the previous tracked audit: trigger escalation, health worsening, new watchlist entry, cleared watchlist, or urgency-rank improvement

• derive remediation_playbooks from provider-specific missing_repo_read_remediation_hints when a mapped rule exists; otherwise fall back to generic playbooks for drift/export anomalies/monitoring

• attach explicit routing metadata to remediation playbooks (owner_surface, owner_team, preferred_fix_lane) so downstream issue generation and triage do not need to rediscover ownership

• derive followup_issue_drafts from watchlist + remediation playbooks for every non-monitor provider; include at least title, severity, owner_team, preferred_fix_lane, and an issue-ready body

• add draft dedupe metadata so recurring audits can detect repeated issue seeds without spam:

  • draft_key
  • dedupe_scope
  • duplicate_of_previous_draft_key

• compute draft_key from a stable fingerprint of provider, primary issue, preferred fix lane, owner team, severity, and trigger level

• dedupe_scope should be human-readable, e.g. provider:primary_issue:preferred_fix_lane

• compare against the prior audit's followup_issue_drafts to populate duplicate_of_previous_draft_key when the draft fingerprint is unchanged

• add follow-up draft lifecycle metadata so downstream automation can distinguish stable recurring work from net-new or changed work:

  • draft_state (new, unchanged, changed)
  • previous_title
  • previous_severity
  • previous_owner_team

• derive draft_state by matching the prior audit's draft set first by provider identity and then by draft_key

• emit cleared_followup_issue_drafts for providers that had a prior actionable draft but no longer produce one in the current audit; each cleared row should carry the prior draft identity and use draft_state = cleared

• add issue-posting readiness metadata so downstream automation can safely decide whether to open a GitHub issue now:

  • per draft: minimum_evidence_present, should_open_issue, issue_open_reason, blocker_reason, evidence_gaps
  • executive block: issue_posting_readiness

• compute minimum_evidence_present from the presence of concrete remediation evidence such as inspect_paths, canonical_targets, first_steps, reference_doc, guidance, trigger_reason, suggested_followup, owner_team, and preferred_fix_lane

• if evidence is incomplete, set should_open_issue = false and populate blocker_reason with the missing fields

• if evidence is complete but draft_state == unchanged, keep should_open_issue = false and explain that duplicate issue creation should wait until linked-issue state is known

• only mark drafts ready / should_open_issue = true when evidence is complete and the draft is materially new or changed

• add live GitHub issue linkage awareness before the final open/no-open decision:

  • load a repo-local issue index via gh issue list --state all --json number,title,state,url
  • enrich readiness rows with linked_issue_number, linked_issue_url, linked_issue_state, linkage_confidence, should_open_issue_final, final_posting_status, final_open_reason, and final_blocker_reason
  • final decision policy:
    • incomplete evidence -> blocked
    • linked open issue -> blocked (final_blocker_reason should cite the issue number)
    • linked closed issue -> ready to open a fresh follow-up
    • unchanged + unlinked -> ready once (do not keep blocking merely because the draft is unchanged)

• make issue linkage resilient to title drift by emitting deterministic draft linkage identity:

  • per draft: linkage_key, linkage_aliases
  • linkage_key should be stable and compact, e.g. provider:primary_issue:severity:preferred_fix_lane
  • linkage_aliases should include the current title, severity-stripped title, previous title when present, severity-stripped previous title, linkage_key, and dedupe_scope

• match linked issues in this order and expose provenance in readiness rows:

  1. exact current title
  2. exact previous title
  3. exact alias match
  • emit matched_on and linked_issue_match_reason so operators/automation can see why a link was chosen

• when readiness is called on synthetic/minimal draft rows (for tests or future consumers), derive fallback linkage_key and linkage_aliases from title/provider/severity/lane rather than assuming they were precomputed by draft generation

• render state=... inline in markdown draft summaries, add a compact Issue posting readiness: section, and include matched_on=... plus the final open/block reason

• include movement text, watchlist text, change-alert text, remediation routing text, issue-posting readiness, and issue-draft summaries in markdown so operators can triage without opening the JSON

  • expose linked_issue_number, linked_issue_url, linked_issue_state, linkage_confidence
  • derive should_open_issue_final, final_posting_status, final_open_reason, final_blocker_reason

• final decision rules that worked well in practice:

  • missing evidence -> final blocked
  • linked OPEN issue -> final blocked (linked open issue already exists: #<n>)
  • linked CLOSED issue -> final ready (safe to open a fresh follow-up)
  • no linked issue + new/changed draft -> final ready
  • no linked issue + unchanged draft -> final ready once (unchanged draft has no linked issue; safe to open once)

• keep both layers in the JSON: draft-layer readiness (should_open_issue) and linkage-aware final readiness (should_open_issue_final) so operators can distinguish evidence sufficiency from duplicate-prevention logic

• render state=... inline in markdown draft summaries, add a compact Issue posting readiness: section including final_should_open and linked_issue=..., and add a compact cleared-drafts section when non-empty

• include movement text, watchlist text, change-alert text, remediation routing text, issue-posting readiness, and issue-draft summaries in markdown so operators can triage without opening the JSON

  • page -> critical
  • act_this_week -> high
  • investigate -> medium
  • monitor -> low

• skip follow-up issue draft creation for monitor providers so recurring audits do not seed low-value issue spam

• include issue-draft summaries in markdown so operators can scan draftable work without opening the JSON

• include movement text, watchlist text, change-alert text, remediation routing text, and issue-draft text in markdown so operators can triage without opening the JSON

• keep rank_movements filtered to meaningful entries (up, down, new) so the machine-readable executive block stays compact even when all providers are stable

• keep health_overview, watchlist, and remediation_playbooks unfiltered so dashboards and cron consumers always have a compact per-provider status table

• derive health_status from urgency tier + debt/drift/python signals + rolling-window profile so the report has an immediate red/yellow/green operator surface

• derive a watchlist with trigger_level, trigger_reason, and suggested_followup so the audit becomes actionable rather than purely descriptive

• emit change_alerts only for meaningful threshold crossings relative to the previous tracked audit:

  • trigger escalation
  • health-color worsening
  • new watchlist entry
  • cleared watchlist
  • meaningful urgency-rank rise

• build remediation_playbooks from rule-backed remediation hints when available (missing_repo_read_remediation_hints / LEGACY_REMEDIATION_RULES), otherwise fall back to generic playbooks for drift/backfill/corpus-anomaly cases

• include exact inspect_paths, canonical_targets, first_steps, reference_doc, and guidance in the remediation playbooks so operators can start work immediately

• include both movement text and health text in markdown so operators can triage without opening the JSON

• keep rank_movements filtered to meaningful entries (up, down, new) so the machine-readable executive block stays compact even when all providers are stable

• keep health_overview unfiltered so dashboards and cron consumers always have a compact per-provider status table

• keep health_overview unfiltered so dashboards and cron consumers always have a compact per-provider status table

• allow change_alerts to be empty when no threshold crossed; that is the desired noise-reduction behavior, not a data-loss bug

  • trigger_reason
  • suggested_followup
  • health_status
  • urgency_tier
  • primary_issue

• keep watchlist sorted by escalation severity so cron consumers can act without recomputing policy

• include movement text, health text, and watchlist trigger text in markdown so operators can triage without opening the JSON

• keep rank_movements filtered to meaningful entries (up, down, new) so the machine-readable executive block stays compact even when all providers are stable

• keep health_overview unfiltered so dashboards and cron consumers always have a compact per-provider status table

• keep watchlist unfiltered as the canonical escalation surface for downstream notifications and alert-threshold logic

Behavioral test pattern

For shell exporters, prefer subprocess tests over string-only tests.

Common pattern

1. Copy the exporter script into a temp repo so it derives repo root from that temp location.
2. Create a temp HOME with native session fixtures.
3. Prepend a fake shim to PATH:
   • fake uv for Gemini/Hermes inline-python exporters
   • fake python3 for Codex resolver-based exporter
4. Run the real shell script with subprocess.run(..., cwd=temp_repo, env=env, capture_output=True, text=True).
5. Assert:
   • exit code is 0
   • output JSONL exists
   • state file exists
   • mapped fields are correct
   • rerun dedup/skip behavior works where applicable
6. Also add one hook-level behavioral test for Codex logger changes:
   • run .Codex/hooks/session-logger.sh directly in a temp repo
   • pass stdin JSON containing session_id, tool_name, and tool_input
   • verify both .Codex/state/sessions/session_YYYYMMDD.jsonl and logs/orchestrator/Codex/session_YYYYMMDD.jsonl receive the new field without breaking existing fields

Operator closeout workflow for audit-refresh tasks

When the user asks to review provider work / strengthen the repository ecosystem, the reusable closeout pattern is:

1. Regenerate the audit and run the targeted provider-audit tests.
2. Read the current executive_summary.provider_interpretation_summary, especially watchlist, followup_issue_drafts, and issue_posting_readiness.
3. Before creating any new GitHub issue from a ready draft, run targeted gh issue list --state all --search ... queries for the draft's provider, primary issue, key stale path families, and generic terms like provider-session / migration debt.
4. If existing open issues cover the finding, do not create duplicates. Add a concise audit-refresh comment to the most relevant umbrella/remediation issues with:
   • generated audit timestamp and artifact paths
   • current provider priority metrics
   • exact verification commands and pass/fail results
   • recommendation to reconcile issue mapping before opening more tickets
5. Commit only durable regenerated artifacts such as analysis/provider-session-ecosystem-audit.json and docs/reports/provider-session-ecosystem-audit.md. Leave transient .Codex/state/* session/correction churn uncommitted unless the task explicitly asks to preserve it.
6. If git add or git commit fails on .git/index.lock, first check for live git/gh processes with ps; if none are active, remove the stale lock and retry.
7. Verify origin/main matches HEAD after push, then report commit SHA, tests, issue comments, and any intentionally uncommitted transient state.

Drift-classification expansion pattern

When provider audits show large missing_repo_reads for Codex/Hermes, first classify the path drift before creating remediation work. The #2333 pattern that worked:

1. Sample top missing reads and separate true actionable repo drift from benign/generated/cross-repo noise.
2. Add narrow taxonomy buckets before changing urgency policy. For generated/static-site paths, use a non_repo_artifact bucket for attested examples such as content/demos/, content/partials/, examples/demos/gtm/output/, build.js, vercel.json, and root package.json when they are absent from the repo.
3. Preserve precedence: existing repo files remain repo, known sibling workspace repos remain sibling_repo, absolute external/transient paths remain external/transient; only missing repo-relative attested artifacts become non_repo_artifact.
4. Exclude non-repo artifacts from actionable missing_repo_reads, top_missing_repo_reads, and remediation hints, but expose them separately in JSON/Markdown as top_non_repo_artifact_reads and non_repo_artifact_read_total so noise reduction remains auditable.
5. Keep schema parity for raw providers and fallback/precomputed summaries by adding empty/default values for new fields.
6. Add tests for classifier precedence, summary exclusion, markdown rendering, fallback schema defaults, positive/zero/negative corpus reconciliation gaps, and JSON scope notes before regenerating artifacts.
7. Re-run the provider audit wrapper after tests, but if the only subsequent changes are timestamp/recent-activity churn from re-running the wrapper after commit/review, restore generated artifacts to the committed verified snapshot rather than creating noisy follow-up commits.

Verification checklist

After any exporter/audit change:

1. Run targeted tests for the changed exporter and provider audit.
2. Re-run the exporter with --all if schema changed materially.
3. Re-run bash scripts/cron/provider-session-ecosystem-audit.sh.
4. Confirm in the report/JSON:
   • false blank reads dropped as expected
   • tool buckets look sane
   • command-family summaries render correctly
   • recent event-time activity and corpus-change sections do not contradict each other
   • provider interpretation ordering matches urgency sorting
   • trend annotations render and compare against the previous tracked audit
   • watchlist and change-alert outputs are present and consistent with prior-audit deltas
   • empty change_alerts on a stable run is acceptable and should not be treated as a failure
   • Codex limitation remains explicit if session ids are still absent

Pitfalls

• Do not count symbolic skill/tool names as missing files.
• Do not append --all exports on top of existing JSONL files.
• Do not rely on only the repo-name Gemini directory; hashed project directories also matter.
• Do not estimate Codex runtime sessions from time gaps; report unavailability unless logger schema changes.
• Do not flatten Codex commands by removing all whitespace.
• Do not confuse event-time recent activity with corpus backfill/rebuild churn.
• Do not prioritize providers from raw counts alone once urgency scores/tiers exist; use the interpretation summary.