Sse Lines Round Extractor

SSE Lines Round Extractor

[Created by Codex: 019b8ca8-c958-7a11-8077-4afb8b1df18e]

Overview

Freeze the moving sse_lines.jsonl logs, extract one {round}.jsonl per round (preserving original event order), then compose a session index and validation artifacts so future work never re-parses 12GB+ of raw logs.

Code Discovery (Use This First)

Canonical extraction code (Python, this repo)

Run these entrypoints from the repo root:

• Freeze sources: extract-from-sse-lines/harness/scripts/freeze_sources.py
• Discover rounds (fresh runs only): extract-from-sse-lines/harness/scripts/discover_rounds.py
• Chunked extractor (resume + time-boxing): extract-from-sse-lines/harness/scripts/extract_rounds.py
• Build sessions index: extract-from-sse-lines/harness/scripts/build_sessions_index.py
• Validate outputs + generate report: extract-from-sse-lines/harness/scripts/validate.py

Key state files:

• Frozen metadata: extract-from-sse-lines/harness/frozen_sources.json
• Provider checkpoints: extract-from-sse-lines/output/{provider}/_checkpoint.json
• Provider manifests: extract-from-sse-lines/output/{provider}/_manifest.jsonl
• Session index: extract-from-sse-lines/output/_sessions_index.jsonl

Consumer/replay code (JS/TS/TSX/Python, any repo)

Locate readers/UIs by searching for these stable “API” strings:

• round-export-v1 (round file header version)
• _sessions_index.jsonl
• output/codex/rounds/ or output/claude/rounds/

Prefer adding new consumers that only parse the envelope JSON (never regex sid/round from content).

Output Structure (Expected)

extract-from-sse-lines/
├── harness/
│   ├── codex_sse_lines.jsonl
│   ├── claude_sse_lines.jsonl
│   ├── frozen_sources.json
│   └── scripts/
│       ├── sse_rounds_lib.py
│       ├── freeze_sources.py
│       ├── discover_rounds.py
│       ├── extract_rounds.py
│       ├── build_sessions_index.py
│       └── validate.py
├── output/
│   ├── _sessions_index.jsonl
│   ├── codex/
│   │   ├── _manifest.jsonl
│   │   ├── _checkpoint.json
│   │   ├── _progress.jsonl
│   │   └── rounds/
│   │       └── {round_id}.jsonl
│   └── claude/
│       ├── _manifest.jsonl
│       ├── _checkpoint.json
│       ├── _progress.jsonl
│       └── rounds/
│           └── {round_id}.jsonl
└── validation/
    ├── checksums.json
    ├── round_counts.json
    └── test_report.md

Workflows

Workflow A — Fresh Full Build (new machine / new output root)

Run this when extract-from-sse-lines/output/ is empty, untrusted, or intentionally discarded.

cd /Users/sotola/AgenticProjects/agent-box-v1 && \
  python extract-from-sse-lines/harness/scripts/freeze_sources.py --force && \
  python extract-from-sse-lines/harness/scripts/discover_rounds.py --provider codex --overwrite && \
  python extract-from-sse-lines/harness/scripts/discover_rounds.py --provider claude --overwrite && \
  python extract-from-sse-lines/harness/scripts/extract_rounds.py --provider codex --chunk-size 500000 --reset-output && \
  python extract-from-sse-lines/harness/scripts/extract_rounds.py --provider claude --chunk-size 500000 --reset-output && \
  python extract-from-sse-lines/harness/scripts/build_sessions_index.py && \
  python extract-from-sse-lines/harness/scripts/validate.py

Workflow B — Incremental Update (new data only; logs grew)

Run this when sse_lines.jsonl has appended new events and existing outputs should be preserved.

1. Freeze new snapshots (overwrite harness copies).
2. Resume extraction (no --reset-output).
3. Rebuild index + validate.

cd /Users/sotola/AgenticProjects/agent-box-v1 && \
  python extract-from-sse-lines/harness/scripts/freeze_sources.py --force && \
  python extract-from-sse-lines/harness/scripts/extract_rounds.py --provider codex --chunk-size 500000 && \
  python extract-from-sse-lines/harness/scripts/extract_rounds.py --provider claude --chunk-size 500000 && \
  python extract-from-sse-lines/harness/scripts/build_sessions_index.py && \
  python extract-from-sse-lines/harness/scripts/validate.py

Interpret results:

• If there is no new data, extract_rounds.py returns quickly (checkpoint already at EOF).
• If the frozen source is smaller than the prior checkpoint offset, treat as truncation/rotation and rerun Workflow A.

Workflow C — Time-Boxed Runs (< 5 minutes)

Use this when running under a strict budget or when iterating. Run repeatedly until the checkpoint marks completed=true.

cd /Users/sotola/AgenticProjects/agent-box-v1 && \
  python extract-from-sse-lines/harness/scripts/extract_rounds.py --provider codex --chunk-size 500000 --max-chunks 2 && \
  python extract-from-sse-lines/harness/scripts/extract_rounds.py --provider claude --chunk-size 500000 --max-chunks 2

Then, only after extraction is complete for both providers:

cd /Users/sotola/AgenticProjects/agent-box-v1 && \
  python extract-from-sse-lines/harness/scripts/build_sessions_index.py && \
  python extract-from-sse-lines/harness/scripts/validate.py

Validation Contract (Proof Step)

Treat extract-from-sse-lines/harness/scripts/validate.py as the gate:

• Line parity: sum(events in all rounds) == source_lines - bad_json - missing_round
• Round count: manifest completed rounds == round files on disk
• No duplicates (best-effort): monotonic (round, event_seq) within each round
• Parseability: every output .jsonl parses, header count matches actual events
• Session completeness: every extracted round appears exactly once in the sessions index

Artifacts:

• extract-from-sse-lines/validation/test_report.md
• extract-from-sse-lines/validation/checksums.json
• extract-from-sse-lines/validation/round_counts.json

Safe Change Requirements (If Code Must Change)

When modifying anything, keep changes confined to extract-from-sse-lines/harness/scripts/ and preserve determinism.

Minimum requirements:

• Keep extraction append-only while reading; only rewrite round files in “finalize header” step.
• Never regex-match sid/round from conversation content; only parse top-level envelope JSON.
• Keep crash-safety: if a chunk crashes mid-write, be able to re-run without corrupting rounds (use checkpoint + chunk journal semantics).
• Maintain resumability by byte offset; treat source truncation as a hard error requiring a reset.
• Extend validation first when introducing new behavior (make failures loud and actionable).