Codex History Repair

Codex History Repair

Repair Codex task history from its durable rollout while preserving recoverable evidence. Never sync raw task databases or session logs between computers.

Safety boundary

• Start read-only. Identify the exact task ID, current rollout, state row, projection row, and lineage source before changing anything.
• Run mutating repairs through scripts/run-with-codex-stopped.sh. It quits Codex, stops the app-server daemon, temporarily unloads a running Threadripper LaunchAgent, requires five consecutive holder-free database checks, restores Threadripper on every exit, and reopens Codex only after repair and watcher restoration succeed. The underlying repair scripts retain their own lsof refusal checks as a second safety layer.
• Preserve dropped encrypted reasoning under $CODEX_HOME/repairs/stashed-reasoning.
• Treat SQLite turns/items/projection rows as derived state. Preserve rollout JSONL as the durable source.
• Back up the exact rollout and SQLite databases before a one-off repair. The sanitizer is idempotent and journals unfinished reprojection in $CODEX_HOME/repairs.
• Do not copy state_5.sqlite, thread_history_*.sqlite, raw sessions, credentials, or provider configuration into a skill bundle or Git repository.
• Do not delete repair backups without explicit approval.

Diagnose

Set the home explicitly when testing another profile:

codex_home=${CODEX_HOME:-"$HOME/.codex"}
python3 scripts/codex-sanitize-reasoning.py --codex-home "$codex_home" --dry-run --verbose

Classify the failure:

• encrypted-reasoning failure: provider-bound response_item reasoning contains encrypted_content that the active provider cannot verify;
• projection-only failure: state and rollout say paginated, but projection/turn/item rows are missing;
• lineage failure: a rollout's history_base.end_byte_offset no longer points to the byte boundary represented by end_ordinal_exclusive in its source rollout;
• combined failure: sanitizing a source rollout shortened it, which invalidated descendants and their derived projections.
• split-rollout failure: one task's chronological history was written to two or more standalone rollout segments, while the state row and paginated projection expose only the newest segment;
• subagent-boundary failure: a subagent has owned item_completed events, but subagent_history_start_ordinal is at EOF and its projection is empty. Codex 0.152.1's legacy migrator reproduces this even when an earlier boundary is supplied; do not repeatedly reset such a rollout to legacy or accept an empty projection as healthy.

Use the terminal UUID in a rollout filename as its rollout/session ID when resolving history_base.thread_id. Do not match arbitrary UUID substrings or assume an underscore separator.

Prevent provider-switch breakage

Install the tools:

sh scripts/install-tools.sh

Run the standalone read-only split audit at any time, including while Codex is open:

codex-auth history-check

codex-auth status and codex-auth recovery run the same audit. It scans active and archived rollout roots, accepts repeated --rollout-root values when invoked directly through codex-sanitize-reasoning.py --check-splits-only, groups segments by rollout metadata rather than requiring the observed filename suffix, and does not mutate rollout or SQLite state.

Run the sanitizer through the lifecycle wrapper:

"$HOME/.local/bin/run-codex-history-repair" -- \
  python3 "$HOME/.local/bin/codex-sanitize-reasoning.py" --verbose

The sanitizer must complete all of these before a provider config changes:

1. detect top-level tasks whose current standalone rollout hides older segments and fail before mutating any rollout or derived state;
2. stash and remove provider-locked encrypted reasoning;
3. rebase every resolvable stale history_base offset to a fixed point;
4. mark affected top-level current rollouts and state rows legacy, keeping subagents out of that migration path;
5. clear only affected top-level derived projection rows;
6. run Codex's own migrate-rollouts --apply for those tasks, and rebuild affected canonical subagent projections through the isolated native projection path;
7. verify every affected task is paginated and projected again;
8. advance idempotency state only after verification.

If any step fails, cancel the provider switch and leave the pending-reprojection journal intact for the next run.

Both the slim and full provider-switch paths invoke this split-rollout preflight. Detection is automatic, but combining segments remains an explicit lifecycle-wrapped repair because it replaces live derived history state. The failure output must list every detected task ID and print the corresponding batch-repair command.

The subagent path backs up the batch databases and rollouts, proves ownership from canonical item_completed and task_started events, and migrates a temporary projection-only copy as a top-level session. It requires event-for-event preservation, complete owned turn/item coverage, and a recognized database schema, then maps native ordinals and byte offsets back to the original rollout. Only the original history mode, visible-history boundary, and derived projection rows change; source/parent/role/provider identity and event content stay intact. Inherited prefixes are excluded. Missing ownership anchors, foreign suffix events, history_base, or unfamiliar schemas fail closed for separate diagnosis. Do not use the one-task legacy projection helper below for a subagent; use the sanitizer's subagent path through the lifecycle wrapper.

One-task repair

For a missing projection with valid lineage:

scripts/run-with-codex-stopped.sh -- \
  python3 scripts/repair-paginated-projection.py --thread <task-uuid>

For a stale lineage cutoff plus projection repair:

scripts/run-with-codex-stopped.sh -- \
  python3 scripts/repair-paginated-lineage.py --thread <task-uuid>

These helpers create timestamped backups under $CODEX_HOME/repairs/backups and refuse to run while required files are open.

Split-rollout repair

When a task has two or more non-overlapping standalone rollout segments, first run the read-only discovery check with every exact task ID and every nonstandard session root:

python3 "$HOME/.codex/repairs/repair-split-rollouts.py" \
  --thread <task-uuid> \
  --rollout-root "/Volumes/Dan SSD/Codex Active/sessions" \
  --check

Review the exact segment count, current rollout, canonical destination, archived state, and older verification turn for every task. Then run the unchanged task/root packet through the lifecycle wrapper:

"$HOME/.local/bin/run-codex-history-repair" -- \
  python3 "$HOME/.codex/repairs/repair-split-rollouts.py" \
  --thread <task-uuid> \
  --rollout-root "/Volumes/Dan SSD/Codex Active/sessions"

Repeat --thread and --rollout-root for a batch. The tool deduplicates mirrored segments by rollout identity and timestamp; requires strict chronological, session-identity, and context-window continuity; creates one canonical combined rollout; preserves archived state; backs up both databases and every source segment; verifies the canonical native migration outcome by path; verifies projection counts and an older turn through native app-server pagination; and only then quarantines superseded segments under the backup directory. Mixed same-thread migration results are allowed only when the canonical path is exactly migrated and every additional result is migrated or already_paginated for the requested task.

Long stages print flushed progress with elapsed times. Native migrations emit a still running message every ten seconds by default; set CODEX_HISTORY_PROGRESS_INTERVAL to another positive number of seconds when testing. A successful repair writes verification.json inside its timestamped backup directory with the canonical rollout hash, projection counts, native migration outcome, and older-turn pagination proof.

On failure, rollback restores source segments first, atomically installs the database backups after moving WAL/SHM sidecars aside, requires PRAGMA quick_check(1) = ok, and verifies every restored state row points to an existing source rollout before moving a canonical rollout into failed-artifact preservation. Do not delete either successful or failed backup directories without explicit approval.

The wrapper manages Threadripper only when its LaunchAgent plist exists and the service was running. Override nonstandard installations with --threadripper-plist and --threadripper-label. Pass --no-reopen to leave Codex closed after success. It never starts a watcher that was stopped before the repair.

When the repair command starts with python3, python, or a .py path, the wrapper resolves Python 3 from PATH and common Homebrew, system, and Python.org locations before stopping Codex. Set CODEX_HISTORY_PYTHON to an absolute interpreter path for another installation.

Validation

Run the synthetic regression suite without touching the live Codex home:

python3 -m unittest discover -s tests -v

After updating the installed lifecycle wrapper, exercise that exact file rather than only the source copy. This suite uses mocked app/watcher controls and temporary database placeholders; it does not stop Codex or run a live repair:

CODEX_TEST_WRAPPER=/Users/dan/.local/bin/run-codex-history-repair python3 -m unittest discover -s tests -p test_lifecycle_wrapper.py -v

For the native subagent migration regressions, supply an absolute Codex binary. These tests use synthetic temporary homes with no credentials or live histories:

CODEX_TEST_BIN=/Applications/ChatGPT.app/Contents/Resources/codex python3 -m unittest discover -s tests -v

That opt-in suite also runs a complete synthetic two-segment repair through the bundled native migrator, database verification, app-server pagination, quarantine, and durable receipt path. It never uses the live Codex profile.

After an approved live repair, reopen Codex and verify the exact task:

• it opens without encrypted-content or lineage errors;
• older and newer visible history are present;
• a new turn succeeds;
• provider switching does not repeat the failure;
• pending reprojection state is empty for the repaired task.

Do not claim restored history merely because migration exited zero. The task must pass both database verification and direct UI/open-turn verification.