# Sse Lines Round Extractor > This skill should be used when freezing centralized Codex/Claude sse_lines.jsonl logs and extracting per-round NDJSON files, composing a sessions index, validating outputs, and re-running incrementally for newly appended data using checkpoints. - Skill: `tankygranny05/sse-lines-round-extractor` (Agent Skill) - Install (CLI): `npx skillmds@latest add tankygranny05/sse-lines-round-extractor` - Raw SKILL.md: https://api.skillmd.com/api/skills/tankygranny05/sse-lines-round-extractor/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: tankygranny05 (https://skillmd.com/u/tankygranny05) - Updated: 2026-09-10 - Page: https://skillmd.com/skills/tankygranny05/sse-lines-round-extractor --- # SSE Lines Round Extractor *[Created by Codex: 019b8ca8-c958-7a11-8077-4afb8b1df18e]* ## Overview Freeze the moving `sse_lines.jsonl` logs, extract one `{round}.jsonl` per round (preserving original event order), then compose a session index and validation artifacts so future work never re-parses 12GB+ of raw logs. ## Code Discovery (Use This First) ### Canonical extraction code (Python, this repo) Run these entrypoints from the repo root: - Freeze sources: `extract-from-sse-lines/harness/scripts/freeze_sources.py` - Discover rounds (fresh runs only): `extract-from-sse-lines/harness/scripts/discover_rounds.py` - Chunked extractor (resume + time-boxing): `extract-from-sse-lines/harness/scripts/extract_rounds.py` - Build sessions index: `extract-from-sse-lines/harness/scripts/build_sessions_index.py` - Validate outputs + generate report: `extract-from-sse-lines/harness/scripts/validate.py` Key state files: - Frozen metadata: `extract-from-sse-lines/harness/frozen_sources.json` - Provider checkpoints: `extract-from-sse-lines/output/{provider}/_checkpoint.json` - Provider manifests: `extract-from-sse-lines/output/{provider}/_manifest.jsonl` - Session index: `extract-from-sse-lines/output/_sessions_index.jsonl` ### Consumer/replay code (JS/TS/TSX/Python, any repo) Locate readers/UIs by searching for these stable “API” strings: - `round-export-v1` (round file header version) - `_sessions_index.jsonl` - `output/codex/rounds/` or `output/claude/rounds/` Prefer adding new consumers that only parse the envelope JSON (never regex `sid`/`round` from content). ## Output Structure (Expected) ```text extract-from-sse-lines/ ├── harness/ │ ├── codex_sse_lines.jsonl │ ├── claude_sse_lines.jsonl │ ├── frozen_sources.json │ └── scripts/ │ ├── sse_rounds_lib.py │ ├── freeze_sources.py │ ├── discover_rounds.py │ ├── extract_rounds.py │ ├── build_sessions_index.py │ └── validate.py ├── output/ │ ├── _sessions_index.jsonl │ ├── codex/ │ │ ├── _manifest.jsonl │ │ ├── _checkpoint.json │ │ ├── _progress.jsonl │ │ └── rounds/ │ │ └── {round_id}.jsonl │ └── claude/ │ ├── _manifest.jsonl │ ├── _checkpoint.json │ ├── _progress.jsonl │ └── rounds/ │ └── {round_id}.jsonl └── validation/ ├── checksums.json ├── round_counts.json └── test_report.md ``` ## Workflows ### Workflow A — Fresh Full Build (new machine / new output root) Run this when `extract-from-sse-lines/output/` is empty, untrusted, or intentionally discarded. ```bash cd /Users/sotola/AgenticProjects/agent-box-v1 && \ python extract-from-sse-lines/harness/scripts/freeze_sources.py --force && \ python extract-from-sse-lines/harness/scripts/discover_rounds.py --provider codex --overwrite && \ python extract-from-sse-lines/harness/scripts/discover_rounds.py --provider claude --overwrite && \ python extract-from-sse-lines/harness/scripts/extract_rounds.py --provider codex --chunk-size 500000 --reset-output && \ python extract-from-sse-lines/harness/scripts/extract_rounds.py --provider claude --chunk-size 500000 --reset-output && \ python extract-from-sse-lines/harness/scripts/build_sessions_index.py && \ python extract-from-sse-lines/harness/scripts/validate.py ``` ### Workflow B — Incremental Update (new data only; logs grew) Run this when `sse_lines.jsonl` has appended new events and existing outputs should be preserved. 1) Freeze new snapshots (overwrite harness copies). 2) Resume extraction (no `--reset-output`). 3) Rebuild index + validate. ```bash cd /Users/sotola/AgenticProjects/agent-box-v1 && \ python extract-from-sse-lines/harness/scripts/freeze_sources.py --force && \ python extract-from-sse-lines/harness/scripts/extract_rounds.py --provider codex --chunk-size 500000 && \ python extract-from-sse-lines/harness/scripts/extract_rounds.py --provider claude --chunk-size 500000 && \ python extract-from-sse-lines/harness/scripts/build_sessions_index.py && \ python extract-from-sse-lines/harness/scripts/validate.py ``` Interpret results: - If there is no new data, `extract_rounds.py` returns quickly (checkpoint already at EOF). - If the frozen source is smaller than the prior checkpoint offset, treat as truncation/rotation and rerun Workflow A. ### Workflow C — Time-Boxed Runs (< 5 minutes) Use this when running under a strict budget or when iterating. Run repeatedly until the checkpoint marks `completed=true`. ```bash cd /Users/sotola/AgenticProjects/agent-box-v1 && \ python extract-from-sse-lines/harness/scripts/extract_rounds.py --provider codex --chunk-size 500000 --max-chunks 2 && \ python extract-from-sse-lines/harness/scripts/extract_rounds.py --provider claude --chunk-size 500000 --max-chunks 2 ``` Then, only after extraction is complete for both providers: ```bash cd /Users/sotola/AgenticProjects/agent-box-v1 && \ python extract-from-sse-lines/harness/scripts/build_sessions_index.py && \ python extract-from-sse-lines/harness/scripts/validate.py ``` ## Validation Contract (Proof Step) Treat `extract-from-sse-lines/harness/scripts/validate.py` as the gate: - Line parity: `sum(events in all rounds) == source_lines - bad_json - missing_round` - Round count: `manifest completed rounds == round files on disk` - No duplicates (best-effort): monotonic `(round, event_seq)` within each round - Parseability: every output `.jsonl` parses, header count matches actual events - Session completeness: every extracted round appears exactly once in the sessions index Artifacts: - `extract-from-sse-lines/validation/test_report.md` - `extract-from-sse-lines/validation/checksums.json` - `extract-from-sse-lines/validation/round_counts.json` ## Safe Change Requirements (If Code Must Change) When modifying anything, keep changes confined to `extract-from-sse-lines/harness/scripts/` and preserve determinism. Minimum requirements: - Keep extraction append-only while reading; only rewrite round files in “finalize header” step. - Never regex-match `sid`/`round` from conversation content; only parse top-level envelope JSON. - Keep crash-safety: if a chunk crashes mid-write, be able to re-run without corrupting rounds (use checkpoint + chunk journal semantics). - Maintain resumability by byte offset; treat source truncation as a hard error requiring a reset. - Extend validation first when introducing new behavior (make failures loud and actionable).