Jsonl Core

JSONL Core — Conversation Parsing Infrastructure

Architecture

All parsing logic lives in ${CLAUDE_PLUGIN_ROOT}/scripts/echolib/ — a Python package (stdlib only, Python 3.6+). User-facing tools are sd-recall.py (search/list/stats), index-builder.py (FTS index), topic-segmenter.py (segmentation), trend-engine.py (longitudinal aggregation), skill-gap-finder.py (pain-point mining), format-detector.py (unknown agent format detection), topic_classify.py (topic routing), chat-profiles.py (group chat participant extraction), and remember.py (auto memory generation).

Data Locations

• Session index (fast path): ~/.claude/projects/<encoded-path>/sessions-index.json
• Fallback index (built by session-digger): ~/.claude/projects/<encoded-path>/.session-digger-index.json
• SQLite unified index: ~/.claude/.session-digger/index.db — auto-maintained by index-builder.py, stores session stats, tool usage JSON, timestamps for all environments
• Full conversations: ~/.claude/projects/<encoded-path>/<uuid>.jsonl
• Subagent conversations: ~/.claude/projects/<encoded-path>/<uuid>/subagents/agent-<id>.jsonl
• Global prompt history: ~/.claude/history.jsonl

Important: Only ~10% of projects have sessions-index.json. The scripts automatically build a fallback index from raw .jsonl files for the remaining 90%, cached in .session-digger-index.json.

Strategy: Fast Path First

Always start with the index before opening any .jsonl file:

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/sd-recall.py sessions --scope current --limit 20
python3 ${CLAUDE_PLUGIN_ROOT}/scripts/sd-recall.py search "search term" --limit 10
python3 ${CLAUDE_PLUGIN_ROOT}/scripts/sd-recall.py sessions --scope path --limit 20

Output is tab-separated: SESSION_ID CREATED MODIFIED MSG_COUNT BRANCH SUMMARY FIRST_PROMPT PROJECT_PATH FULL_PATH

The FULL_PATH field (9th column) is the absolute path to the .jsonl file. Use this to pass to other scripts.

Only open the full .jsonl when you need message-level detail.

Primary Tools

All parsing lives in ${CLAUDE_PLUGIN_ROOT}/scripts/echolib/. Use these front-ends:

# Build fast FTS search index (one-time, ~1-2s)
python3 ${CLAUDE_PLUGIN_ROOT}/scripts/index-builder.py build

# Search across all sessions (FTS, <50ms with index)
python3 ${CLAUDE_PLUGIN_ROOT}/scripts/sd-recall.py search "keyword" --limit 10

# List sessions
python3 ${CLAUDE_PLUGIN_ROOT}/scripts/sd-recall.py sessions --scope all --limit 20

# Aggregate statistics
python3 ${CLAUDE_PLUGIN_ROOT}/scripts/sd-recall.py stats

# Topic segmentation per session
python3 ${CLAUDE_PLUGIN_ROOT}/scripts/topic-segmenter.py <file.jsonl>

# Pattern analysis (retries, errors, user corrections)
bash ${CLAUDE_PLUGIN_ROOT}/scripts/analyze-session.sh <file.jsonl>

Direct Python API

import echolib
for msg in echolib.extract_messages("file.jsonl", role="user", limit=20):
    print(msg["text"])
for tool in echolib.extract_tools("file.jsonl", errors_only=True):
    print(tool["name"], tool["status"])
stats = echolib.session_stats("file.jsonl")

Subagent Discovery

Sessions with subagent work have a <session-uuid>/subagents/ directory. Check for it:

ls "$(dirname <full_path>)/$(basename <full_path> .jsonl)/subagents/" 2>/dev/null

Subagent files follow the same JSONL format and can be parsed with the same scripts.

Performance Notes

• With FTS index: keyword search <50ms regardless of total sessions
• index-builder.py incremental update: only re-parses changed files
• extract_messages(limit=N) enables early exit — near-instant for small N
• iter_records(skip_noise=True) avoids json.loads on noise lines
• For files > 10MB: json.loads is the CPU bottleneck (63% of time), not I/O
• extract_files_changed() uses reverse-read for large files
• session_stats() single-pass: counts errors, tokens, compactions together

When Grep Is Enough

For targeted searches when no FTS index is built yet:

# Find user messages containing a keyword
Grep pattern='"type":"user"' path="<file.jsonl>" output_mode="content"

# Find error results
Grep pattern='"is_error"\s*:\s*true'

# Find specific tool usage
Grep pattern='"name"\s*:\s*"ToolName"'

# Find decisions (AskUserQuestion usage)
Grep pattern='"name"\s*:\s*"AskUserQuestion"'

Record Type Quick Reference

See references/record-types.md for the complete schema. For WorkBuddy's tree-structured format, see references/workbuddy-record-types.md. The essential types:

Finding the Right Session Directory

To map a project path to its Claude session directory:

1. Take the absolute project path (e.g., /Users/joker/github/myproject)
2. Replace all / with - → Users-joker-github-myproject
3. Prepend - → -Users-joker-github-myproject
4. Look in ~/.claude/projects/-Users-joker-github-myproject/

Use sd-recall.py sessions which handles lookup automatically:

Noise Filtering

When reading raw .jsonl, skip these:

• Records where type is progress or queue-operation (streaming/internal bookkeeping)
• User records with isMeta: true (slash command injection)
• User records with isCompactSummary: true (auto-generated context, not human input)
• Assistant records with model: "<synthetic>" (passthrough, not real inference)
• User records where content is an array of tool_result blocks (tool outputs, not human messages)

This is handled automatically by echolib/ functions (skip_noise=True by default).

Schema Evolution Awareness

Claude Code evolves rapidly. The JSONL format has changed across versions:

• New record types appear silently (e.g., progress at v2.1.14, pr-link later)
• New optional fields are added to existing records (~3-5 per minor version)
• Some record types (summary, pr-link, file-history-snapshot) lack common fields like version or uuid
• The directory encoding is lossy for Unicode paths — use sessions-index.json's originalPath field as ground truth

When parsing results look unexpected, use echolib.detect_schema("file.jsonl") or the schema-scout agent to check for unknown record types.

Multi-Agent Support

Session-digger supports parsing sessions from multiple agents. Use detect_agent_type() to identify the agent:

python3 -c "from echolib import detect_agent_type; print(detect_agent_type('/path/to/session'))"

Grok Build Session Format

Grok stores sessions under ~/.grok/sessions/<url-encoded-cwd>/<session-id>/ with a different file layout:

$GROK_HOME/sessions/<encoded-cwd-or-slug>/<session-id>/   # GROK_HOME defaults to ~/.grok
  summary.json            # metadata: title, timestamps, model, message count
  chat_history.jsonl      # raw messages sent to the model
  updates.jsonl           # authoritative ACP session/update stream (resume source)
  events.jsonl            # internal event stream (tool_started/completed, turns)
  signals.json            # pre-aggregated stats (toolFailureCount, toolCallCount, …)
  plan.json               # in-session TODO state (when present)
  rewind_points.jsonl     # file snapshots for /rewind
  compaction_checkpoints/ # auto-compact saved state
  subagents/              # child session metadata; child transcripts live in sessions tree
  .cwd                    # (group dir only) original cwd when path encoding is rewritten

Key differences from Claude Code:

• Data root: honour GROK_HOME (official CLI), not only ~/.grok.
• Tool calls in chat_history.jsonl: embedded tool_calls on assistant messages + tool_result rows. Use grok_extract_tools() (events.jsonl supplements timestamps/outcome).
• Stats source of truth: signals.json for activity counters (tools/messages/errors); updates.jsonl → params.update.usage for billable tokens (inputTokens/outputTokens/cachedReadTokens). Usage snapshots are run-cumulative; a drop in modelCalls starts a new run — digger sums last-of-each-run. Per-model legs from usage.modelUsage use the same rule and land in stats["model_usage"] (sum of models == session totals). Do not use signals.contextTokensUsed as billable input (that is window occupancy).
• 主会话 / 子代理分账: grok_family_usage_report(session_dir) — 读 subagents/*/meta.json；自动判定 rollup / separate；轻量 token profile + updates 单次流式聚合 + mtime 缓存。跨会话汇总 grok_aggregate_model_usage(mode="family")：有子代理的父会话只计家族一次（rollup 用父、separate 用父+子），子会话不双计；mode="raw" 才允许调试双计。
• No file-history-snapshot: use rewind_points.jsonl for file change tracking.
• FTS5 search: session_search.sqlite under the sessions root. Use grok_list_sessions() with keyword.
• Outcome field: tool_completed uses outcome: "success"|"error" (also accept "failure").
• No gitBranch field: Grok sessions don't track git branch info.

Grok adapter functions in echolib/:

• detect_agent_type(path) — returns "claude", "grok", "both", or "unknown"
• grok_list_sessions(cwd, limit, keyword) — list/search Grok sessions
• grok_session_stats(session_dir) — read pre-aggregated stats
• grok_extract_messages(session_dir, role, limit) — extract from chat_history.jsonl
• grok_extract_tools(session_dir, tool_filter, errors_only, limit) — extract tool calls from chat_history.jsonl
• grok_session_path(cwd, session_id) — find a Grok session directory

ZCode Session Format

• Agents root: ~/.zcode/cli/agents/sess_*/agent_*/transcript.jsonl
• Official billable source: ~/.zcode/cli/db/db.sqlite table model_usage (one row per model call: model_id, input/output/cache/reasoning tokens).
• digger prefers SQL SUM + GROUP BY model_id → stats["model_usage"]; falls back to summing transcript model_complete.usage.
• Subagents use distinct session_id (sess_subagent_agent_*, also metadata.childSessionId); no parent rollup double-count in this table.
• Global model totals: zcode_aggregate_model_usage().
• Cache hit rate (Grok + ZCode): cache_hit_rate = cache_read_tokens / input_tokens on session stats and each model_usage leg (None if no input). Helper: compute_cache_hit_rate / attach_cache_hit_rates.

Kimi Code Session Format

Kimi Code stores sessions under ~/.kimi/sessions/<project-hash>/<session-uuid>/:

~/.kimi/sessions/<project-hash>/<session-uuid>/
  wire.jsonl       # raw event stream (TurnBegin, ToolCall, ToolResult, ContentPart, ...)
  context.jsonl    # context management records (_checkpoint, compaction boundaries)
  metadata.json    # session_id, title, wire_mtime
  state.json       # optional: model, agent state
  context_sub_N.jsonl  # sub-context files from compaction
  subagents/       # child session directories

Key differences from Claude Code:

• Top-level type field: Kimi stores message type in message.type (e.g. TurnBegin, ToolCall, ContentPart), not in the record-level type.
• User input in TurnBegin: User messages are nested in TurnBegin.payload.user_input[].text. Use kimi_extract_messages().
• Tool calls by ID: ToolCall and ToolResult are matched by tool_call_id (exact match, not proximity). Use kimi_extract_tools().
• wire_mtime for timestamps: Session timestamp comes from metadata.json's wire_mtime field (Unix float), not from individual message timestamps.
• No is_error field: Kimi's ToolResult doesn't expose error status. Tool failures are not distinguishable from successes in the wire format.
• No gitBranch field: Kimi sessions don't track git branch info.

Kimi adapter functions in echolib/:

• kimi_list_sessions(cwd, limit, keyword) — list/search Kimi sessions
• kimi_session_stats(session_dir) — count messages, tools from wire.jsonl
• kimi_extract_messages(session_dir, role, limit) — extract from wire.jsonl
• kimi_extract_tools(session_dir, tool_filter, errors_only, limit) — extract tool calls from wire.jsonl
• kimi_session_path(cwd, session_id) — find a Kimi session directory

Cross-Tool Unified Interface

For analysis across all agents, use the unified functions:

• cross_tool_list_sessions(limit, keyword, agent_filter) — returns merged list of dicts with agent, session_id, created, summary, first_prompt, msg_count, full_path
• cross_tool_session_stats(session_path) — auto-detects agent type and dispatches
• scan_all_environments_parallel() — probes all known agent directories in parallel, returns status per environment
• dispatch_resolve_agent(path) — maps a path to its agent type (claude, grok, kimi_code, codex, workbuddy, trae_cn, zcode, dim, reasonix, universal)

Use sd-recall.py:

# List across all agents
python3 ${CLAUDE_PLUGIN_ROOT}/scripts/sd-recall.py sessions --scope all --agent cross --limit 20

# List only Kimi sessions
python3 ${CLAUDE_PLUGIN_ROOT}/scripts/sd-recall.py sessions --scope all --agent kimi --limit 20

下游协作

DO NOT

• 从零创建新 skill → skill-creator
• 需要提炼经验/教训 → experience-synthesis（jsonl-core 只做解析，不做分析）
• 需要管理 MEMORY.md → memory-management