Structural Code Index Builder
This is an optional first-class stage in the Pass Lifecycle Contract. It
runs immediately after the snapshot is pinned (Block D), before the first
code-reading analysis stage (summarize/architecture). It only needs CODE_ROOT
+ SNAPSHOT_ID and must not depend on architecture/KB.
System Goal
Structural Code Index Builder. Builds a content-addressed semantic-unit index
from source code using capability-based per-partition backend selection,
degrading gracefully to grep. Provides find_callers(symbol),
get_function_boundary(file, line), and call-site awareness to improve LLM
reasoning quality during discovery — supplementing (never replacing) grep-based
call-site discovery with structural data. The index is composed from independent
semantic compilation units, persisted as a manifest + SQLite catalog, and
exposed through a bounded query helper.
Command Definition
- Command:
/mantis-structural-index
- Description: Build a content-addressed semantic-unit index from source
code under
CODE_ROOT.
- Arguments (optional; supplied by the orchestrator, consumed by Block A):
--snapshot_root/--snapshot_id/--state_root. All absent → MODE-OFF/legacy
mode (reads source from the current directory, writes index to
./workspace/kb/structural_index/).
Input/Output Contract
- Reads:
workspace/.mantis_state.json (to read active_snapshot for provenance
checking and snapshot-aware rebuild logic).
workspace/kb/structural_index/manifest.json (to check snapshot_id for
reuse-on-match idempotency — primary check).
workspace/kb/structural_index.jsonl (backward-compat provenance check if
manifest absent).
workspace/kb/structural_index/units/ (content-addressed cache for
incremental unit reuse).
workspace/kb/structural_index/native/ and sidecar provenance.json files
(prebuilt index attachments and metadata manifests).
- CODE_ROOT source files (via generated helper script — the script parses all
source files under
CODE_ROOT).
- Writes:
workspace/kb/structural_index/manifest.json (STATE-RELATIVE — atomic
commit point, written LAST).
workspace/kb/structural_index/catalog.sqlite (STATE-RELATIVE —
query-optimized serving store).
workspace/kb/structural_index/units/ (STATE-RELATIVE — content-addressed
immutable unit outputs).
workspace/kb/structural_index/shards/ (STATE-RELATIVE — partitioned
serving data for large corpora).
workspace/kb/structural_index/native/ (STATE-RELATIVE — prebuilt index
attachments: SCIP, Kythe, LSIF).
workspace/kb/structural_index/tmp/ (STATE-RELATIVE — temporary objects
during build).
workspace/kb/structural_index.jsonl (STATE-RELATIVE — compatibility
pointer; full export below threshold).
workspace/helpers/build_structural_index.py (STATE-RELATIVE — the builder
helper script).
workspace/helpers/query_structural_index.py (STATE-RELATIVE — the query
helper script).
- Preconditions:
- Source files must exist under
CODE_ROOT. If CODE_ROOT is not resolved
(MODE-OFF and no readable active_snapshot), build against the current
directory with snapshot_id set to "unknown". Do NOT skip — this is the
standalone-efficiency case.
- Inert until wired: This skill returns an empty index until a caller
invokes it (the harness,
mantis-plan, or mantis-researcher). It never
fails — it simply returns an empty index if tools are unavailable or source
cannot be parsed.
- Idempotency Guarantee:
- Read-only on
CODE_ROOT. Writes only to STATE-RELATIVE paths. Re-running
with the same CODE_ROOT and SNAPSHOT_ID reuses the existing index
(manifest snapshot_id match) rather than rebuilding — except in MODE-OFF,
where it always rebuilds. Individual semantic units are reused across
snapshots when their content-addressed cache keys match (incremental
rebuild).
Instructions
Step 0: Locator Resolution (run first)
LOCATOR RESOLUTION (before reading ANY target code or artifact):
0. ROLE: If this skill NEVER reads target source (report, calibrate, reflect),
you are a FINDINGS-ONLY stage: skip steps 2-6; still read active_snapshot from
state for provenance/annotation; NEVER stop merely because a code root is unset.
1. Determine CODE_ROOT, in this priority order:
a. If --target_root is passed on THIS invocation, CODE_ROOT = --target_root.
It is AUTHORITATIVE and OVERRIDES SNAPSHOT_ROOT and the state fallback
(used when a caller hands you a prepared tree, e.g. a patched shadow).
b. Else if --snapshot_root (or SNAPSHOT_ROOT) is passed, use it.
c. Else read state_root/workspace/.mantis_state.json (state_root from
--state_root if passed, else ./workspace/... relative to the current dir)
-> active_snapshot.root / .snapshot_id / .snapshot_pinned.
d. Else (no arg AND no readable active_snapshot): CODE_ROOT = current directory,
treat snapshot_pinned = false (MODE-OFF). Do NOT stop.
2. SENTINEL CHECK (only if snapshot_pinned is true AND you did NOT take path 1a):
verify CODE_ROOT/.mantis_snapshot_id exists and equals SNAPSHOT_ID. If missing
or different -> STOP "snapshot sentinel mismatch". (A --target_root tree (1a) is
deliberately mutated and is sentinel-EXEMPT.)
3. PATH FIELDS:
- SNAPSHOT-RELATIVE (read under CODE_ROOT): code_paths entries; plan target_files
that are file paths. Strip ONLY a trailing ":<digits>". A code_paths entry
containing "://" is a URL/endpoint, NOT a file read. A code_paths entry that is
NOT of the form <existing-path>:<integer> is a non-source LOCATOR
(symbol/offset/endpoint): only check that the artifact/symbol exists; skip ALL
line-range and line-existence logic.
- STATE-RELATIVE (read/write under state_root/workspace, NEVER prefix CODE_ROOT):
kb_references, repro_file_path, reattack_file_path, helper scripts, report
files, and all state/findings JSON.
4. Never WRITE under CODE_ROOT when snapshot_pinned is true. Any command that
compiles, generates, or writes artifacts MUST run in a PRIVATE SHADOW copy
(mktemp -d from CODE_ROOT), never with cwd=CODE_ROOT. Read-only inspection may
cd into CODE_ROOT.
5. VCS-METADATA CARVE-OUT: history-log extraction and any VCS diff/blame command
run in the LIVE repository root (which still has .git/.hg/.repo), NOT CODE_ROOT
(the snapshot copy strips VCS metadata). Do NOT stop merely because CODE_ROOT
lacks .git/.hg/.repo.
6. Every shell command uses ABSOLUTE paths and sets its own working directory on
that call. Do NOT assume the working directory persists between calls.
This is a CODE-READING stage — it reads target source files under CODE_ROOT via
the helper script. Block A step 0's findings-only skip does NOT apply.
Step 1: Idempotency / Freshness Check (MANDATORY FIRST)
MANDATORY FIRST STEP: Before writing any build scripts, probing backends, or
extracting symbols, check for MODE-OFF or an existing manifest:
- MODE-OFF check (FIRST): In MODE-OFF (
SNAPSHOT_ID is "unknown" or
absent) → always rebuild. Do NOT reuse a previous "unknown" index,
because the live tree is mutable and "unknown" is a constant (not a
freshness signal). Skip directly to Step 2 (individual units whose content
has not changed may still hit the content-addressed cache).
- If
workspace/kb/structural_index/manifest.json exists, read its
snapshot_id field.
- If
snapshot_id matches the current SNAPSHOT_ID (and SNAPSHOT_ID is NOT
"unknown") → reuse the index immediately and STOP. Do NOT invoke Step
2, probe backends, or write build scripts. This bounds cost across
retries/crash-resume.
- If
manifest.json is absent but workspace/kb/structural_index.jsonl exists
(backward compat), read its provenance header (_provenance, snapshot_id
keys). If snapshot_id matches (and is NOT "unknown") → reuse and
STOP. Otherwise proceed to rebuild.
- If
SNAPSHOT_ID differs → proceed to rebuild (Steps 2–5). For incremental
reuse: before rebuilding a semantic unit, check units/ for a
content-addressed cache hit (see Content-Addressed Cache Key below). A cache
hit reuses the unit output without re-extraction.
Step 2: Select Backend per Partition
Partition the codebase into semantic units (see Per-Language Semantic Units
below). For each unit, select the most precise backend available in this
environment using a capability-based, per-partition decision — not a single
global ladder. The following are examples, not an exhaustive enum. The
helper probes each tier per partition and selects the highest available:
- Snapshot-matched SCIP / LSIF / Kythe / clangd-static / remote index
(precision:
semantic) — if a pre-built index matching the current
snapshot_id or root_fingerprint is available. Most precise: full
type-aware cross-reference, call hierarchy, and hover/signature data.
- Compiler / typechecker-backed extractor (precision:
typecheck) — if
compile_commands.json, build context, or typechecker is available.
Type-accurate symbol resolution and call edges.
- Language-aware AST extraction (precision:
ast) — if tree-sitter,
ast-grep, or a language-specific parser is available. Full AST parsing:
function boundaries, call expressions, signatures.
- Symbol-only extraction (precision:
symbol-only) — if ctags or
equivalent is on PATH. Symbol table only (function definitions, locations —
no call graph). Call-site extraction uses a lightweight regex pass within
known function boundaries.
- Heuristic fallback (precision:
heuristic) — Python stdlib regex pass
over source files. Identifies function definitions and call patterns using
language-agnostic heuristics. Less precise but zero-dependency.
- Coverage-only manifest + lexical fallback (precision:
coverage-only) —
grep; no structural index is written. Manifest records status: "empty".
Consumers fall back to grep-based discovery (today's behavior byte-for-byte).
Native Index Probing & Resolution Rules:
- Probe instruction: Before evaluating per-partition backends, probe
workspace/kb/structural_index/native/ and subdirectories
native/{scip,lsif,kythe}/ for prebuilt index files.
- Snapshot-declaration convention: Because native formats (SCIP, LSIF,
Kythe) do not embed snapshot identity directly in their binary payload,
prebuilt indexes MUST declare their target snapshot using a sidecar
provenance.json manifest located at
workspace/kb/structural_index/native/provenance.json or
native/<kind>/provenance.json. The manifest contains an array of
attachments:
[{"kind": "scip|lsif|kythe", "path": "...", "snapshot_id": "...", "root_fingerprint": "...", "language": "...", "indexer": "...", "precision": "semantic", "files": [...]}].
A native index is matched if its declared snapshot_id equals SNAPSHOT_ID
(when SNAPSHOT_ID != "unknown") or its root_fingerprint matches the
workspace's calculated root fingerprint. If the files array is absent or
empty, treat the native index as covering no individual files directly (record
in manifest.native_indexes but do not update coverage rows; fall through
to lower tiers for all files).
- Record-and-Defer Ingestion Rule: Parsing raw binary native indexes (e.g.
SCIP protobuf) in pure Python without dependencies is costly and complex.
Option A builder scripts MUST detect matching prebuilt native indexes, record
their entries in
manifest.native_indexes, and set the coverage table
backend (e.g., "scip" or "scip-clangd") for all files listed in the
provenance manifest. If catalog.sqlite is NOT populated with symbols from
the native index (raw binary deferred to harness/MCP readers), set
coverage.status = "deferred" and precision = "deferred" (with
indexed_files = 0). This prevents the query helper from claiming an
un-ingested partition is "authoritative empty" at semantic precision,
ensuring consumers run the mandatory grep fallback. When catalog.sqlite IS
populated (e.g., via Option B pre-ingestion or scip-to-sqlite), set
precision = "semantic" and status = "indexed".
LSP is NOT equivalent to SCIP / LSIF. LSP is an interactive protocol whose
workspace state may be partial or mutable. Use it only when the server can
demonstrate snapshot identity AND complete workspace coverage. A running
language server does not automatically qualify as a semantic backend.
SCIP merging. SCIP explicitly permits merging complementary information from
indexers with different precision levels. Its format records indexer version and
per-document language metadata. The builder MAY merge results from multiple
indexers (e.g., a SCIP index for Go + tree-sitter for Python) within a single
catalog.
The determinism lives in a runtime-generated versioned helper
(build_structural_index.py, # MANTIS_HELPER_VERSION = 5, grep-and-regenerate
on reuse) that probes and selects backends per partition. No shipped binaries;
air-gapped-safe.
Deterministic partial coverage. On very large source trees, a full rebuild
can dominate stage-0 wall-clock. Replace discovery-order truncation with a
deterministic priority queue:
- Explicit target files and symbols (from
plan.json, if available).
- Changed units and known reverse dependencies (from Block E diff +
reverse-dependency edges in the existing catalog).
- Containing packages/modules and direct imports of tier 1+2 units.
- Remaining units in normalized path order.
Apply deterministic max_units (default: 10000) or max_source_bytes (default:
500MB) bounds to that queue. Persist deferred units in the manifest so another
invocation can resume. Statuses: complete, partial, empty, failed.
A consumer should only interpret "no callers" strongly when the relevant
partition is complete, snapshot-matched, covered, and semantically precise.
Otherwise the answer is "no indexed callers", followed by the existing lexical
fallback. The index is HINT-only, so partial coverage is safe (grep remains
authoritative).
Step 3: Write and Run Helper Scripts
Two runtime-generated helpers are used. Both follow the grep-and-regenerate
pattern: before reuse, grep the first line for the version marker; if absent or
a different integer, REGENERATE.
Builder: build_structural_index.py
- Write the builder to
workspace/helpers/build_structural_index.py. The FIRST
LINE MUST be exactly # MANTIS_HELPER_VERSION = 5. Before reusing an
existing helper, grep its first lines for MANTIS_HELPER_VERSION = 5; if
that marker is absent or a different integer, REGENERATE the helper.
- The builder partitions the codebase into semantic units (see Per-Language
Semantic Units), computes content-addressed cache keys, checks
units/ for
reuse, selects a backend per partition (Step 2), extracts symbols + edges,
and writes results to catalog.sqlite.
- The builder must extract:
- Symbols:
symbol_id, name, qualified_name, namespace,
language, file_path, start_line, end_line, kind, signature,
backend, precision.
- Call edges:
caller_id, callee_id, callee_name, file_path,
line, edge_kind.
- Function boundaries:
symbol_id, file_path, start_line,
end_line, signature, language.
- Coverage:
file_path, indexed, backend, precision,
unit_cache_key, status.
- The builder writes the manifest LAST (Step 4) and the compatibility pointer.
- The builder MUST use ABSOLUTE paths and set its own working directory (Block
A step 6). It MUST NOT write anything under
CODE_ROOT when
snapshot_pinned is true (Block A step 4). Backends that produce sidecar
files (ctags tags, cscope cscope.out, clangd cache) MUST be redirected to
STATE-RELATIVE paths: ctags -f <state>/helpers/tags,
cscope -f <state>/helpers/cscope.out,
CLANGD_INDEX_STORAGE=<state>/helpers/. Read-only LSP/SCIP queries to a
running server need no redirect.
Query helper: query_structural_index.py
- Write the query helper to
workspace/helpers/query_structural_index.py. The
FIRST LINE MUST be exactly # MANTIS_HELPER_VERSION = 5. Before reusing an
existing helper, grep its first lines for MANTIS_HELPER_VERSION = 5; if
absent or a different integer, REGENERATE.
- The query helper provides bounded, paginated operations against
catalog.sqlite (or a remote endpoint — identical API). It IS the
consumption contract (see Query Interface below).
- If
catalog.sqlite is absent but structural_index.jsonl exists, the query
helper falls back to linear scanning of the JSONL file (slower but
functional). If both are absent, it returns empty results with
coverage.partition_status = "empty".
Step 4: Write Manifest and Commit
- Write the manifest to
workspace/kb/structural_index/tmp/manifest.json
first. Then atomically rename it to
workspace/kb/structural_index/manifest.json. This is the atomic commit
point — the manifest is written LAST, after all units, catalog, and coverage
data are written.
- Write a compatibility pointer to
workspace/kb/structural_index.jsonl
(STATE-RELATIVE — NEVER under CODE_ROOT). Below a configurable threshold
(default: 10K records), emit a complete JSONL export (provenance header + all
records). Above the threshold, emit only a provenance header with
compat_pointer.full_export = false and compat_pointer.symbol_count set.
Large consumers MUST use the query interface.
- The manifest
provider.backend_versions field records which backend was used
per language (e.g.,
{"go": {"backend_name": "scip-clangd", "precision": "semantic"}, "python": {"backend_name": "tree-sitter", "precision": "ast"}}),
so consumers know the precision level.
- Interrupted builds leave unreferenced temp objects in
tmp/ without
corrupting the last published index. On resume, check manifest.json status
and snapshot_id (Step 1).
Step 5: Return Results / Notify Caller
- Return the path to
manifest.json, catalog.sqlite, and the query helper.
Include a summary (unit count, symbol count, call-edge count, backends used,
coverage status).
- If the index is empty (no tools available or no source files found), notify
the caller: "Structural index is empty — structural context unavailable."
- Do not notify the user directly — this skill is invoked as a sub-agent by the
harness, planner, or researcher.
Index Schema and On-Disk Contract
On-Disk Layout
workspace/kb/structural_index/
├── manifest.json # Atomic commit point — written LAST
├── catalog.sqlite # Query-optimized serving store (both directions indexed)
├── units/ # Content-addressed immutable unit outputs
│ └── ab/cd/abcdef... # sha256 prefix sharding (2+2 hex dirs)
├── shards/ # Partitioned serving data (large corpora)
│ └── shard_0000.sqlite
├── native/ # Prebuilt index attachments (SCIP, Kythe, LSIF)
│ ├── provenance.json # Prebuilt index provenance manifest
│ ├── scip/
│ └── kythe/
└── tmp/ # Temporary objects during build
workspace/helpers/
├── build_structural_index.py # Builder (MANTIS_HELPER_VERSION = 5)
└── query_structural_index.py # Query helper (MANTIS_HELPER_VERSION = 5)
workspace/kb/structural_index.jsonl # Compatibility pointer
Manifest Schema (manifest.json)
{
"schema_version": 1,
"snapshot_id": "<SNAPSHOT_ID or 'unknown'>",
"root_fingerprint": "<sha256 of sorted (path, content_sha256) for all source files>",
"status": "complete|partial|empty|failed",
"provider": {
"kind": "local-build|baseline+overlay|remote",
"catalog": "catalog.sqlite",
"backend_versions": {
"cpp": {"backend_name": "tree-sitter", "backend_version": "0.20.8", "precision": "ast"},
"go": {"backend_name": "scip-clangd", "backend_version": "0.2.3", "precision": "semantic"}
}
},
"units": {"total": 0, "reused": 0, "rebuild": 0, "failed": 0},
"coverage": {"total_files": 0, "indexed_files": 0, "failed_files": 0, "deferred_files": 0},
"shards": [{"id": "", "path": "", "checksum": "", "partition_key": "", "symbol_count": 0, "edge_count": 0}],
"deferred_units": [{"unit_id": "", "language": "", "files": [], "priority": 4, "reason": ""}],
"native_indexes": [{"kind": "scip", "path": "", "snapshot_id": "", "root_fingerprint": "", "language": "", "indexer": "", "precision": ""}],
"baseline": {"source": "ci|local|none", "snapshot_id": "", "manifest_path": ""},
"overlay": {"units_added": 0, "units_modified": 0, "files": []},
"compat_pointer": {"path": "structural_index.jsonl", "full_export": true, "symbol_count": 0},
"created_at": "<ISO 8601>",
"build_duration_ms": 0
}
Atomic commit: The manifest is written LAST (atomic rename from tmp/).
Interrupted builds leave unreferenced temp objects without corrupting the last
published index.
SQLite Catalog Schema (catalog.sqlite)
CREATE TABLE IF NOT EXISTS schema_meta (
key TEXT PRIMARY KEY, value TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS symbols (
symbol_id TEXT PRIMARY KEY,
name TEXT NOT NULL,
qualified_name TEXT NOT NULL,
namespace TEXT,
language TEXT NOT NULL,
file_path TEXT NOT NULL,
start_line INTEGER NOT NULL,
end_line INTEGER,
kind TEXT NOT NULL,
signature TEXT,
backend TEXT NOT NULL,
precision TEXT NOT NULL CHECK (precision IN ('semantic','typecheck','ast','symbol-only','heuristic','deferred','coverage-only')),
corpus TEXT DEFAULT 'default',
partition_key TEXT,
unit_cache_key TEXT,
source_layer TEXT NOT NULL DEFAULT 'baseline'
);
CREATE TABLE IF NOT EXISTS call_edges (
edge_id INTEGER PRIMARY KEY AUTOINCREMENT,
caller_id TEXT NOT NULL,
callee_id TEXT,
callee_name TEXT NOT NULL,
file_path TEXT NOT NULL,
line INTEGER NOT NULL,
edge_kind TEXT NOT NULL CHECK (edge_kind IN ('direct','indirect','virtual','macro','unresolved')),
corpus TEXT DEFAULT 'default',
partition_key TEXT,
source_layer TEXT NOT NULL DEFAULT 'baseline',
FOREIGN KEY (caller_id) REFERENCES symbols(symbol_id),
FOREIGN KEY (callee_id) REFERENCES symbols(symbol_id)
);
CREATE TABLE IF NOT EXISTS function_boundaries (
symbol_id TEXT PRIMARY KEY,
file_path TEXT NOT NULL,
start_line INTEGER NOT NULL,
end_line INTEGER NOT NULL,
signature TEXT,
language TEXT NOT NULL,
FOREIGN KEY (symbol_id) REFERENCES symbols(symbol_id)
);
CREATE TABLE IF NOT EXISTS coverage (
file_path TEXT NOT NULL,
indexed INTEGER NOT NULL DEFAULT 0,
backend TEXT,
precision TEXT,
unit_cache_key TEXT,
status TEXT NOT NULL DEFAULT 'pending',
corpus TEXT DEFAULT 'default',
partition_key TEXT,
PRIMARY KEY (file_path, corpus)
);
CREATE TABLE IF NOT EXISTS unit_cache (
cache_key TEXT PRIMARY KEY,
unit_id TEXT NOT NULL,
language TEXT NOT NULL,
extractor_name TEXT NOT NULL,
extractor_version TEXT NOT NULL,
file_count INTEGER NOT NULL,
symbol_count INTEGER NOT NULL,
edge_count INTEGER NOT NULL,
source_bytes INTEGER NOT NULL,
created_at TEXT NOT NULL,
snapshot_id TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS deferred_units (
unit_id TEXT PRIMARY KEY,
language TEXT NOT NULL,
files TEXT NOT NULL,
priority INTEGER NOT NULL,
reason TEXT NOT NULL,
cache_key TEXT,
created_at TEXT NOT NULL
);
-- Indexes: both directions!
CREATE INDEX IF NOT EXISTS idx_symbols_name ON symbols(name);
CREATE INDEX IF NOT EXISTS idx_symbols_qualified ON symbols(qualified_name);
CREATE INDEX IF NOT EXISTS idx_symbols_file ON symbols(file_path, start_line);
CREATE INDEX IF NOT EXISTS idx_symbols_lang ON symbols(language);
CREATE INDEX IF NOT EXISTS idx_edges_caller ON call_edges(caller_id);
CREATE INDEX IF NOT EXISTS idx_edges_callee ON call_edges(callee_id);
CREATE INDEX IF NOT EXISTS idx_edges_callee_name ON call_edges(callee_name);
CREATE INDEX IF NOT EXISTS idx_edges_file ON call_edges(file_path, line);
CREATE INDEX IF NOT EXISTS idx_boundaries_file ON function_boundaries(file_path, start_line, end_line);
CREATE INDEX IF NOT EXISTS idx_coverage_status ON coverage(status);
CREATE INDEX IF NOT EXISTS idx_deferred_priority ON deferred_units(priority);
Partitioning: When symbols exceed 500K or edges exceed 2M, partition into
shard SQLite files under shards/ by lang:{language}:bucket:{NN} (stable
hash). Each shard has the same schema. The query helper routes queries to the
correct shard(s) based on the manifest.
Content-Addressed Cache Key
cache_key = sha256(
schema_version +
extractor_name@version +
language +
compile_context_digest +
ordered_input_content_digests +
dependency_or_interface_digest
)
Critical: snapshot_id is NOT in the cache key. It goes in provenance only.
This permits reuse across commits, branches, and snapshots with identical units.
Canonical Symbol IDs
Native IDs (when semantic backend is available):
- SCIP:
scip:{symbol}
- Kythe:
kythe:{uri}
- clangd:
clangd:{usr}
Fallback IDs (when no semantic backend):
fallback:{language}:{file_path}:{sha256(qualified_name|start_line|signature)[:16]}
This distinguishes namespaces, overloads, methods, local functions, and
duplicate names across languages or repositories.
Edge Kinds
Edge kinds are kept explicitly separate — never silently merged:
direct — statically resolved call to a known symbol
indirect — function pointer, closure, callback
virtual — virtual method dispatch (runtime-resolved)
macro — call introduced by macro expansion
unresolved — callee name found but no symbol_id resolved
Per-Language Semantic Units
| Language |
Unit Type |
Compile Context |
Dependency Digest |
| C/C++ |
Compilation unit |
compile_commands.json entry |
sha256 of transitive header interfaces |
| Go |
Package |
go.mod + build tags |
sha256 of imported packages' exported API |
| Rust |
Crate |
Cargo.toml + features |
sha256 of extern crate signatures |
| Java/Kotlin |
Compiler batch |
module path + classpath |
sha256 of imported class signatures |
| TypeScript |
Project |
tsconfig.json |
sha256 of imported module type declarations |
| Fallback |
Individual file |
None |
Empty string |
Fallback Granularity Rule: For the fallback tier (regex, AST parser, or
lightweight crawler without a language build system), each source file MUST be
its own independent semantic unit. Never bundle multiple source files into a
single fallback unit. Per-file unit isolation is essential for content-addressed
cache efficiency — when one file changes in a future snapshot, only that single
file's unit invalidates while all unchanged files hit the cache
(reused = N-1).
Baseline + Delta Overlay
- Static baseline: Built once in CI for the complete snapshot.
- Incremental cache: Immutable unit outputs reused across snapshots.
- Delta overlay: Changed/target files indexed locally, layered over
baseline.
- Remote serving: Optional when global index is too costly locally.
At query time, the query helper merges baseline + overlay results. Overlay
symbols supersede baseline symbols for the same file. The manifest records
provider.kind = "baseline+overlay" with baseline manifest reference and
overlay unit count.
Compatibility JSONL Export
The old structural_index.jsonl format is retained as a compatibility export
only — it is NOT the query contract. Below the threshold (default 10K records),
a complete JSONL export is emitted. Above the threshold, only a provenance
pointer is written. Large consumers MUST use the query interface.
Line 1 — provenance header:
{"_provenance": true, "snapshot_id": "abc123", "tool": "tree-sitter"}
Lines 2+ — structural records (one per line, _type discriminator):
{"_type": "function", "key": "src/parser.c:parse_input", "start_line": 45, "end_line": 120, "signature": "int parse_input(char *buf, size_t len)", "calls": ["malloc", "validate_input", "memcpy"]}
{"_type": "call_edge", "caller": "parse_input", "callee": "malloc", "file": "src/parser.c", "line": 78}
{"_type": "call_edge", "caller": "main", "callee": "parse_input", "file": "src/main.c", "line": 203}
Provenance header fields:
| Field |
Type |
Description |
_provenance |
bool |
Always true — marks this as the provenance header line |
snapshot_id |
string |
SNAPSHOT_ID the index was built against |
tool |
string |
Backend used (e.g. "lsp-clangd", "tree-sitter", "ctags", "regex", "grep") |
Record types:
_type |
Description |
Key fields |
function |
Function definition with boundary and signature |
key, start_line, end_line, signature, calls |
call_edge |
A call from caller to callee at file:line |
caller, callee, file, line |
Snapshot Safety
- Build from
CODE_ROOT, not live tree (when pinned). When the snapshot is
pinned, the structural index is built from the pinned CODE_ROOT, ensuring
it reflects the exact bytes the pipeline is analyzing.
- Reuse-on-match. If
manifest.json already carries the current
SNAPSHOT_ID (and SNAPSHOT_ID is NOT "unknown" — in MODE-OFF, always
rebuild), reuse it — do not rebuild. Rebuild when SNAPSHOT_ID differs or is
"unknown". Individual units may still be reused from the content-addressed
cache. This bounds cost across retries/crash-resume.
- Manifest atomicity. The manifest is written LAST via atomic rename from
tmp/. Interrupted builds leave unreferenced temp objects without corrupting
the last published index. On resume, check manifest.json status and
snapshot_id.
- STALE flag in HALT mode. When
snapshot_pinned is false (HALT), the
index may be built from the unpinned CODE_ROOT but is marked as potentially
stale. Consumers treat structural hints as advisory.
- MODE-OFF: build, do not skip. When
active_snapshot is absent
(MODE-OFF), build against the current directory (cwd) with provenance
snapshot_id set to "unknown". This is the standalone-efficiency case —
the index is still useful for the researcher/planner even without snapshot
pinning. Do NOT return an empty index merely because the snapshot is absent.
Consumption Contract
These are runtime instructions for callers — they define how consumers use the
structural index. The structural index is a HINT-only enhancement; skills that
do not use it behave exactly as they do today.
Query Interface
The query helper IS the contract. All consumers use
query_structural_index.py (or a compatible remote endpoint). The JSONL file is
NOT the query contract — it is a compatibility export only.
Operations:
resolve_symbol(name, language?, file?, namespace?) →
{results, total, ambiguous, coverage}
- Returns ALL matches — no silent selection among ambiguous symbols.
- Caller MUST disambiguate before calling
find_callers / find_callees.
find_callers(symbol_id, limit=100, offset=0) →
{results, total, has_more, coverage}
- Bounded, paginated caller lookup.
- Each result carries
precision, backend, edge_kind.
- Empty results carry
coverage.partition_status.
find_callees(symbol_id, limit=100, offset=0) →
{results, total, has_more, coverage}
- Same shape as
find_callers.
get_function_boundary(file, line) →
{symbol_id, start_line, end_line, signature, precision, backend}
get_coverage(file?) →
{total_files, indexed_files, failed, deferred, partition_status, backends_used}
Key properties:
- Bounded results and pagination (prevents loading entire index into memory).
- Explicit name resolution before graph traversal (no silent selection among
ambiguous symbols).
- Precision and backend attached to every result.
- Coverage attached to empty results.
- Identical operations for local and remote providers (remote via
MANTIS_STRUCTURAL_INDEX_URL env var or manifest provider.kind = "remote").
Coverage on empty results:
partition_status |
Meaning |
Consumer action |
complete |
All files indexed |
"No indexed callers" (still run grep per HINT-only rule) |
partial |
Some files deferred or failed |
"Not fully indexed" — MUST run grep fallback |
empty |
No backend available |
"Not indexed" — MUST run grep fallback |
failed |
Backend attempted but failed |
"Index failed" — MUST run grep fallback |
mantis-plan
- Use the structural index query helper for function-level dependency fan-out.
When planning investigations, call
resolve_symbol() then find_callers() to
identify all functions that call into a target — this broadens the audit set
beyond single-file analysis.
- The structural index decides ORDER of investigations (which functions to audit
first based on call-graph centrality), never MEMBERSHIP. It may broaden the
audit set (safe over-reporting), but must never REMOVE or drop a file; the
planner's existing logic remains the membership floor.
mantis-researcher
- Wave 1 (Rapid Triage): Run a repo-wide grep for the function name to build
the exhaustive set of candidate call-sites — this is the mandatory floor. Then
use the structural index query helper (
resolve_symbol then find_callers)
to RANK and prioritize which call-sites to audit first (the index
distinguishes actual calls from comments/strings/variable names). Audit the
union of both result sets — the structural index may miss macro-based calls,
function pointers, and dynamic dispatch, so grep remains the floor.
- Wave 2 (Deep Audit): Use
get_function_boundary(file, line) to start with
the enclosing function, expanding to callers/callees/file as needed for
cross-function context — this saves context while preserving coverage.
- If the structural index is absent or empty, fall back to grep-based discovery
(today's behavior). The structural index is a coverage HINT only — it improves
audit quality but is never required.
Safety
Non-negotiable invariants:
- Agnostic. Nothing is ever required; no-tool / parse-fail / not-invoked →
empty index → grep fallback = today's behavior byte-for-byte. Optional in the
Pass Lifecycle Contract; a non-conformant harness simply skips it.
- HINT-only / union / never MEMBERSHIP. Consumers audit the union with
grep; a symbol the index misses must still be reachable by the exhaustive
sweep. The structural index decides ORDER, never MEMBERSHIP. The query
contract enforces this:
find_callers returns HINTs, never authoritative
membership — consumers MUST union with grep.
- No verdicts, touches no findings. Cannot violate INV-1 and cannot itself
drop a finding.
- Narrow scope: source cross-reference only. Binary/build-derived
reachability ("is it compiled into production") is explicitly out of scope —
a dev customization, not part of this skill, because absence-from-a-build can
hide a real finding (INV-2). Do not fold build-derived reachability into this
skill.
A reference blueprint is available at
mantis-pipeline-adapter/references/mantis-structural-index.md.
It is a stub that points to this SKILL.md as the single source of truth — do not
duplicate spec content there.
1---
2name: mantis-structural-index
3description: >-
4 Builds a content-addressed semantic-unit index from source code for
5 structural context. Use when a pinned or live codebase is available and
6 structural cross-reference data would improve research quality. Don't use for
7 findings analysis, patching, or reporting.
8metadata:
9 source: https://skillmd.com/skills/google/mantis-structural-index
10---
11
12# Structural Code Index Builder
13
14This is an **optional first-class stage** in the Pass Lifecycle Contract. It
15runs immediately after the snapshot is pinned (Block D), before the first
16code-reading analysis stage (summarize/architecture). It only needs `CODE_ROOT`
17\+ `SNAPSHOT_ID` and must not depend on architecture/KB.
18
19## System Goal
20
21Structural Code Index Builder. Builds a content-addressed semantic-unit index
22from source code using capability-based per-partition backend selection,
23degrading gracefully to grep. Provides `find_callers(symbol)`,
24`get_function_boundary(file, line)`, and call-site awareness to improve LLM
25reasoning quality during discovery — supplementing (never replacing) grep-based
26call-site discovery with structural data. The index is composed from independent
27semantic compilation units, persisted as a manifest + SQLite catalog, and
28exposed through a bounded query helper.
29
30## Command Definition
31
32- **Command:** `/mantis-structural-index`
33- **Description:** Build a content-addressed semantic-unit index from source
34 code under `CODE_ROOT`.
35- **Arguments (optional; supplied by the orchestrator, consumed by Block A):**
36 `--snapshot_root`/`--snapshot_id`/`--state_root`. All absent → MODE-OFF/legacy
37 mode (reads source from the current directory, writes index to
38 `./workspace/kb/structural_index/`).
39
40## Input/Output Contract
41
42- **Reads**:
43 - `workspace/.mantis_state.json` (to read `active_snapshot` for provenance
44 checking and snapshot-aware rebuild logic).
45 - `workspace/kb/structural_index/manifest.json` (to check `snapshot_id` for
46 reuse-on-match idempotency — primary check).
47 - `workspace/kb/structural_index.jsonl` (backward-compat provenance check if
48 manifest absent).
49 - `workspace/kb/structural_index/units/` (content-addressed cache for
50 incremental unit reuse).
51 - `workspace/kb/structural_index/native/` and sidecar `provenance.json` files
52 (prebuilt index attachments and metadata manifests).
53 - CODE_ROOT source files (via generated helper script — the script parses all
54 source files under `CODE_ROOT`).
55- **Writes**:
56 - `workspace/kb/structural_index/manifest.json` (STATE-RELATIVE — atomic
57 commit point, written LAST).
58 - `workspace/kb/structural_index/catalog.sqlite` (STATE-RELATIVE —
59 query-optimized serving store).
60 - `workspace/kb/structural_index/units/` (STATE-RELATIVE — content-addressed
61 immutable unit outputs).
62 - `workspace/kb/structural_index/shards/` (STATE-RELATIVE — partitioned
63 serving data for large corpora).
64 - `workspace/kb/structural_index/native/` (STATE-RELATIVE — prebuilt index
65 attachments: SCIP, Kythe, LSIF).
66 - `workspace/kb/structural_index/tmp/` (STATE-RELATIVE — temporary objects
67 during build).
68 - `workspace/kb/structural_index.jsonl` (STATE-RELATIVE — compatibility
69 pointer; full export below threshold).
70 - `workspace/helpers/build_structural_index.py` (STATE-RELATIVE — the builder
71 helper script).
72 - `workspace/helpers/query_structural_index.py` (STATE-RELATIVE — the query
73 helper script).
74- **Preconditions**:
75 - Source files must exist under `CODE_ROOT`. If `CODE_ROOT` is not resolved
76 (MODE-OFF and no readable `active_snapshot`), build against the current
77 directory with `snapshot_id` set to `"unknown"`. Do NOT skip — this is the
78 standalone-efficiency case.
79- **Inert until wired:** This skill returns an empty index until a caller
80 invokes it (the harness, `mantis-plan`, or `mantis-researcher`). It never
81 fails — it simply returns an empty index if tools are unavailable or source
82 cannot be parsed.
83- **Idempotency Guarantee**:
84 - Read-only on `CODE_ROOT`. Writes only to STATE-RELATIVE paths. Re-running
85 with the same `CODE_ROOT` and `SNAPSHOT_ID` reuses the existing index
86 (manifest `snapshot_id` match) rather than rebuilding — except in MODE-OFF,
87 where it always rebuilds. Individual semantic units are reused across
88 snapshots when their content-addressed cache keys match (incremental
89 rebuild).
90
91## Instructions
92
93### Step 0: Locator Resolution (run first)
94
95```
96LOCATOR RESOLUTION (before reading ANY target code or artifact):
970. ROLE: If this skill NEVER reads target source (report, calibrate, reflect),
98 you are a FINDINGS-ONLY stage: skip steps 2-6; still read active_snapshot from
99 state for provenance/annotation; NEVER stop merely because a code root is unset.
1001. Determine CODE_ROOT, in this priority order:
101 a. If --target_root is passed on THIS invocation, CODE_ROOT = --target_root.
102 It is AUTHORITATIVE and OVERRIDES SNAPSHOT_ROOT and the state fallback
103 (used when a caller hands you a prepared tree, e.g. a patched shadow).
104 b. Else if --snapshot_root (or SNAPSHOT_ROOT) is passed, use it.
105 c. Else read state_root/workspace/.mantis_state.json (state_root from
106 --state_root if passed, else ./workspace/... relative to the current dir)
107 -> active_snapshot.root / .snapshot_id / .snapshot_pinned.
108 d. Else (no arg AND no readable active_snapshot): CODE_ROOT = current directory,
109 treat snapshot_pinned = false (MODE-OFF). Do NOT stop.
1102. SENTINEL CHECK (only if snapshot_pinned is true AND you did NOT take path 1a):
111 verify CODE_ROOT/.mantis_snapshot_id exists and equals SNAPSHOT_ID. If missing
112 or different -> STOP "snapshot sentinel mismatch". (A --target_root tree (1a) is
113 deliberately mutated and is sentinel-EXEMPT.)
1143. PATH FIELDS:
115 - SNAPSHOT-RELATIVE (read under CODE_ROOT): code_paths entries; plan target_files
116 that are file paths. Strip ONLY a trailing ":<digits>". A code_paths entry
117 containing "://" is a URL/endpoint, NOT a file read. A code_paths entry that is
118 NOT of the form <existing-path>:<integer> is a non-source LOCATOR
119 (symbol/offset/endpoint): only check that the artifact/symbol exists; skip ALL
120 line-range and line-existence logic.
121 - STATE-RELATIVE (read/write under state_root/workspace, NEVER prefix CODE_ROOT):
122 kb_references, repro_file_path, reattack_file_path, helper scripts, report
123 files, and all state/findings JSON.
1244. Never WRITE under CODE_ROOT when snapshot_pinned is true. Any command that
125 compiles, generates, or writes artifacts MUST run in a PRIVATE SHADOW copy
126 (mktemp -d from CODE_ROOT), never with cwd=CODE_ROOT. Read-only inspection may
127 cd into CODE_ROOT.
1285. VCS-METADATA CARVE-OUT: history-log extraction and any VCS diff/blame command
129 run in the LIVE repository root (which still has .git/.hg/.repo), NOT CODE_ROOT
130 (the snapshot copy strips VCS metadata). Do NOT stop merely because CODE_ROOT
131 lacks .git/.hg/.repo.
1326. Every shell command uses ABSOLUTE paths and sets its own working directory on
133 that call. Do NOT assume the working directory persists between calls.
134```
135
136This is a CODE-READING stage — it reads target source files under CODE_ROOT via
137the helper script. Block A step 0's findings-only skip does NOT apply.
138
139### Step 1: Idempotency / Freshness Check (MANDATORY FIRST)
140
141**MANDATORY FIRST STEP:** Before writing any build scripts, probing backends, or
142extracting symbols, check for MODE-OFF or an existing manifest:
143
1441. **MODE-OFF check (FIRST):** In MODE-OFF (`SNAPSHOT_ID` is `"unknown"` or
145 absent) → **always rebuild**. Do NOT reuse a previous `"unknown"` index,
146 because the live tree is mutable and `"unknown"` is a constant (not a
147 freshness signal). Skip directly to Step 2 (individual units whose content
148 has not changed may still hit the content-addressed cache).
1492. If `workspace/kb/structural_index/manifest.json` exists, read its
150 `snapshot_id` field.
1513. If `snapshot_id` matches the current `SNAPSHOT_ID` (and `SNAPSHOT_ID` is NOT
152 `"unknown"`) → **reuse the index immediately and STOP**. Do NOT invoke Step
153 2, probe backends, or write build scripts. This bounds cost across
154 retries/crash-resume.
1554. If `manifest.json` is absent but `workspace/kb/structural_index.jsonl` exists
156 (backward compat), read its provenance header (`_provenance`, `snapshot_id`
157 keys). If `snapshot_id` matches (and is NOT `"unknown"`) → **reuse and
158 STOP**. Otherwise proceed to rebuild.
1595. If `SNAPSHOT_ID` differs → proceed to rebuild (Steps 2–5). For incremental
160 reuse: before rebuilding a semantic unit, check `units/` for a
161 content-addressed cache hit (see Content-Addressed Cache Key below). A cache
162 hit reuses the unit output without re-extraction.
163
164### Step 2: Select Backend per Partition
165
166Partition the codebase into semantic units (see Per-Language Semantic Units
167below). For each unit, select the most precise backend available in this
168environment using a **capability-based, per-partition decision** — not a single
169global ladder. The following are **examples, not an exhaustive enum**. The
170helper probes each tier per partition and selects the highest available:
171
1721. **Snapshot-matched SCIP / LSIF / Kythe / clangd-static / remote index**
173 (precision: `semantic`) — if a pre-built index matching the current
174 `snapshot_id` or `root_fingerprint` is available. Most precise: full
175 type-aware cross-reference, call hierarchy, and hover/signature data.
1762. **Compiler / typechecker-backed extractor** (precision: `typecheck`) — if
177 `compile_commands.json`, build context, or typechecker is available.
178 Type-accurate symbol resolution and call edges.
1793. **Language-aware AST extraction** (precision: `ast`) — if `tree-sitter`,
180 `ast-grep`, or a language-specific parser is available. Full AST parsing:
181 function boundaries, call expressions, signatures.
1824. **Symbol-only extraction** (precision: `symbol-only`) — if `ctags` or
183 equivalent is on `PATH`. Symbol table only (function definitions, locations —
184 no call graph). Call-site extraction uses a lightweight regex pass within
185 known function boundaries.
1865. **Heuristic fallback** (precision: `heuristic`) — Python stdlib regex pass
187 over source files. Identifies function definitions and call patterns using
188 language-agnostic heuristics. Less precise but zero-dependency.
1896. **Coverage-only manifest + lexical fallback** (precision: `coverage-only`) —
190 grep; no structural index is written. Manifest records `status: "empty"`.
191 Consumers fall back to grep-based discovery (today's behavior byte-for-byte).
192
193**Native Index Probing & Resolution Rules**:
194
195- **Probe instruction**: Before evaluating per-partition backends, probe
196 `workspace/kb/structural_index/native/` and subdirectories
197 `native/{scip,lsif,kythe}/` for prebuilt index files.
198- **Snapshot-declaration convention**: Because native formats (SCIP, LSIF,
199 Kythe) do not embed snapshot identity directly in their binary payload,
200 prebuilt indexes MUST declare their target snapshot using a sidecar
201 `provenance.json` manifest located at
202 `workspace/kb/structural_index/native/provenance.json` or
203 `native/<kind>/provenance.json`. The manifest contains an array of
204 attachments:
205 `[{"kind": "scip|lsif|kythe", "path": "...", "snapshot_id": "...", "root_fingerprint": "...", "language": "...", "indexer": "...", "precision": "semantic", "files": [...]}]`.
206 A native index is matched if its declared `snapshot_id` equals `SNAPSHOT_ID`
207 (when `SNAPSHOT_ID != "unknown"`) or its `root_fingerprint` matches the
208 workspace's calculated root fingerprint. If the `files` array is absent or
209 empty, treat the native index as covering no individual files directly (record
210 in `manifest.native_indexes` but do not update `coverage` rows; fall through
211 to lower tiers for all files).
212- **Record-and-Defer Ingestion Rule**: Parsing raw binary native indexes (e.g.
213 SCIP protobuf) in pure Python without dependencies is costly and complex.
214 Option A builder scripts MUST detect matching prebuilt native indexes, record
215 their entries in `manifest.native_indexes`, and set the `coverage` table
216 `backend` (e.g., `"scip"` or `"scip-clangd"`) for all files listed in the
217 provenance manifest. If `catalog.sqlite` is NOT populated with symbols from
218 the native index (raw binary deferred to harness/MCP readers), set
219 `coverage.status = "deferred"` and `precision = "deferred"` (with
220 `indexed_files = 0`). This prevents the query helper from claiming an
221 un-ingested partition is "authoritative empty" at `semantic` precision,
222 ensuring consumers run the mandatory grep fallback. When `catalog.sqlite` IS
223 populated (e.g., via Option B pre-ingestion or `scip-to-sqlite`), set
224 `precision = "semantic"` and `status = "indexed"`.
225
226**LSP is NOT equivalent to SCIP / LSIF.** LSP is an interactive protocol whose
227workspace state may be partial or mutable. Use it only when the server can
228demonstrate snapshot identity AND complete workspace coverage. A running
229language server does not automatically qualify as a `semantic` backend.
230
231**SCIP merging.** SCIP explicitly permits merging complementary information from
232indexers with different precision levels. Its format records indexer version and
233per-document language metadata. The builder MAY merge results from multiple
234indexers (e.g., a SCIP index for Go + tree-sitter for Python) within a single
235catalog.
236
237The determinism lives in a runtime-generated versioned helper
238(`build_structural_index.py`, `# MANTIS_HELPER_VERSION = 5`, grep-and-regenerate
239on reuse) that probes and selects backends per partition. No shipped binaries;
240air-gapped-safe.
241
242**Deterministic partial coverage.** On very large source trees, a full rebuild
243can dominate stage-0 wall-clock. Replace discovery-order truncation with a
244deterministic priority queue:
245
2461. **Explicit target files and symbols** (from `plan.json`, if available).
2472. **Changed units and known reverse dependencies** (from Block E diff +
248 reverse-dependency edges in the existing catalog).
2493. **Containing packages/modules and direct imports** of tier 1+2 units.
2504. **Remaining units in normalized path order**.
251
252Apply deterministic `max_units` (default: 10000) or `max_source_bytes` (default:
253500MB) bounds to that queue. Persist deferred units in the manifest so another
254invocation can resume. Statuses: `complete`, `partial`, `empty`, `failed`.
255
256A consumer should only interpret "no callers" strongly when the relevant
257partition is complete, snapshot-matched, covered, and semantically precise.
258Otherwise the answer is "no indexed callers", followed by the existing lexical
259fallback. The index is HINT-only, so partial coverage is safe (grep remains
260authoritative).
261
262### Step 3: Write and Run Helper Scripts
263
264Two runtime-generated helpers are used. Both follow the grep-and-regenerate
265pattern: before reuse, grep the first line for the version marker; if absent or
266a different integer, REGENERATE.
267
268#### Builder: `build_structural_index.py`
269
2701. Write the builder to `workspace/helpers/build_structural_index.py`. The FIRST
271 LINE MUST be exactly `# MANTIS_HELPER_VERSION = 5`. Before reusing an
272 existing helper, grep its first lines for `MANTIS_HELPER_VERSION = 5`; if
273 that marker is absent or a different integer, REGENERATE the helper.
2742. The builder partitions the codebase into semantic units (see Per-Language
275 Semantic Units), computes content-addressed cache keys, checks `units/` for
276 reuse, selects a backend per partition (Step 2), extracts symbols + edges,
277 and writes results to `catalog.sqlite`.
2783. The builder must extract:
279 - **Symbols**: `symbol_id`, `name`, `qualified_name`, `namespace`,
280 `language`, `file_path`, `start_line`, `end_line`, `kind`, `signature`,
281 `backend`, `precision`.
282 - **Call edges**: `caller_id`, `callee_id`, `callee_name`, `file_path`,
283 `line`, `edge_kind`.
284 - **Function boundaries**: `symbol_id`, `file_path`, `start_line`,
285 `end_line`, `signature`, `language`.
286 - **Coverage**: `file_path`, `indexed`, `backend`, `precision`,
287 `unit_cache_key`, `status`.
2884. The builder writes the manifest LAST (Step 4) and the compatibility pointer.
2895. The builder MUST use ABSOLUTE paths and set its own working directory (Block
290 A step 6). It MUST NOT write anything under `CODE_ROOT` when
291 `snapshot_pinned` is true (Block A step 4). Backends that produce sidecar
292 files (ctags `tags`, cscope `cscope.out`, clangd cache) MUST be redirected to
293 STATE-RELATIVE paths: `ctags -f <state>/helpers/tags`,
294 `cscope -f <state>/helpers/cscope.out`,
295 `CLANGD_INDEX_STORAGE=<state>/helpers/`. Read-only LSP/SCIP queries to a
296 running server need no redirect.
297
298#### Query helper: `query_structural_index.py`
299
3001. Write the query helper to `workspace/helpers/query_structural_index.py`. The
301 FIRST LINE MUST be exactly `# MANTIS_HELPER_VERSION = 5`. Before reusing an
302 existing helper, grep its first lines for `MANTIS_HELPER_VERSION = 5`; if
303 absent or a different integer, REGENERATE.
3042. The query helper provides bounded, paginated operations against
305 `catalog.sqlite` (or a remote endpoint — identical API). It IS the
306 consumption contract (see Query Interface below).
3073. If `catalog.sqlite` is absent but `structural_index.jsonl` exists, the query
308 helper falls back to linear scanning of the JSONL file (slower but
309 functional). If both are absent, it returns empty results with
310 `coverage.partition_status = "empty"`.
311
312### Step 4: Write Manifest and Commit
313
3141. Write the manifest to `workspace/kb/structural_index/tmp/manifest.json`
315 first. Then atomically rename it to
316 `workspace/kb/structural_index/manifest.json`. This is the atomic commit
317 point — the manifest is written LAST, after all units, catalog, and coverage
318 data are written.
3192. Write a compatibility pointer to `workspace/kb/structural_index.jsonl`
320 (STATE-RELATIVE — NEVER under `CODE_ROOT`). Below a configurable threshold
321 (default: 10K records), emit a complete JSONL export (provenance header + all
322 records). Above the threshold, emit only a provenance header with
323 `compat_pointer.full_export = false` and `compat_pointer.symbol_count` set.
324 Large consumers MUST use the query interface.
3253. The manifest `provider.backend_versions` field records which backend was used
326 per language (e.g.,
327 `{"go": {"backend_name": "scip-clangd", "precision": "semantic"}, "python": {"backend_name": "tree-sitter", "precision": "ast"}}`),
328 so consumers know the precision level.
3294. Interrupted builds leave unreferenced temp objects in `tmp/` without
330 corrupting the last published index. On resume, check `manifest.json` status
331 and `snapshot_id` (Step 1).
332
333### Step 5: Return Results / Notify Caller
334
3351. Return the path to `manifest.json`, `catalog.sqlite`, and the query helper.
336 Include a summary (unit count, symbol count, call-edge count, backends used,
337 coverage status).
3382. If the index is empty (no tools available or no source files found), notify
339 the caller: "Structural index is empty — structural context unavailable."
3403. Do not notify the user directly — this skill is invoked as a sub-agent by the
341 harness, planner, or researcher.
342
343## Index Schema and On-Disk Contract
344
345### On-Disk Layout
346
347```
348workspace/kb/structural_index/
349├── manifest.json # Atomic commit point — written LAST
350├── catalog.sqlite # Query-optimized serving store (both directions indexed)
351├── units/ # Content-addressed immutable unit outputs
352│ └── ab/cd/abcdef... # sha256 prefix sharding (2+2 hex dirs)
353├── shards/ # Partitioned serving data (large corpora)
354│ └── shard_0000.sqlite
355├── native/ # Prebuilt index attachments (SCIP, Kythe, LSIF)
356│ ├── provenance.json # Prebuilt index provenance manifest
357│ ├── scip/
358│ └── kythe/
359└── tmp/ # Temporary objects during build
360
361workspace/helpers/
362├── build_structural_index.py # Builder (MANTIS_HELPER_VERSION = 5)
363└── query_structural_index.py # Query helper (MANTIS_HELPER_VERSION = 5)
364
365workspace/kb/structural_index.jsonl # Compatibility pointer
366```
367
368### Manifest Schema (`manifest.json`)
369
370```json
371{
372 "schema_version": 1,
373 "snapshot_id": "<SNAPSHOT_ID or 'unknown'>",
374 "root_fingerprint": "<sha256 of sorted (path, content_sha256) for all source files>",
375 "status": "complete|partial|empty|failed",
376 "provider": {
377 "kind": "local-build|baseline+overlay|remote",
378 "catalog": "catalog.sqlite",
379 "backend_versions": {
380 "cpp": {"backend_name": "tree-sitter", "backend_version": "0.20.8", "precision": "ast"},
381 "go": {"backend_name": "scip-clangd", "backend_version": "0.2.3", "precision": "semantic"}
382 }
383 },
384 "units": {"total": 0, "reused": 0, "rebuild": 0, "failed": 0},
385 "coverage": {"total_files": 0, "indexed_files": 0, "failed_files": 0, "deferred_files": 0},
386 "shards": [{"id": "", "path": "", "checksum": "", "partition_key": "", "symbol_count": 0, "edge_count": 0}],
387 "deferred_units": [{"unit_id": "", "language": "", "files": [], "priority": 4, "reason": ""}],
388 "native_indexes": [{"kind": "scip", "path": "", "snapshot_id": "", "root_fingerprint": "", "language": "", "indexer": "", "precision": ""}],
389 "baseline": {"source": "ci|local|none", "snapshot_id": "", "manifest_path": ""},
390 "overlay": {"units_added": 0, "units_modified": 0, "files": []},
391 "compat_pointer": {"path": "structural_index.jsonl", "full_export": true, "symbol_count": 0},
392 "created_at": "<ISO 8601>",
393 "build_duration_ms": 0
394}
395```
396
397**Atomic commit**: The manifest is written LAST (atomic rename from `tmp/`).
398Interrupted builds leave unreferenced temp objects without corrupting the last
399published index.
400
401### SQLite Catalog Schema (`catalog.sqlite`)
402
403```sql
404CREATE TABLE IF NOT EXISTS schema_meta (
405 key TEXT PRIMARY KEY, value TEXT NOT NULL
406);
407
408CREATE TABLE IF NOT EXISTS symbols (
409 symbol_id TEXT PRIMARY KEY,
410 name TEXT NOT NULL,
411 qualified_name TEXT NOT NULL,
412 namespace TEXT,
413 language TEXT NOT NULL,
414 file_path TEXT NOT NULL,
415 start_line INTEGER NOT NULL,
416 end_line INTEGER,
417 kind TEXT NOT NULL,
418 signature TEXT,
419 backend TEXT NOT NULL,
420 precision TEXT NOT NULL CHECK (precision IN ('semantic','typecheck','ast','symbol-only','heuristic','deferred','coverage-only')),
421 corpus TEXT DEFAULT 'default',
422 partition_key TEXT,
423 unit_cache_key TEXT,
424 source_layer TEXT NOT NULL DEFAULT 'baseline'
425);
426
427CREATE TABLE IF NOT EXISTS call_edges (
428 edge_id INTEGER PRIMARY KEY AUTOINCREMENT,
429 caller_id TEXT NOT NULL,
430 callee_id TEXT,
431 callee_name TEXT NOT NULL,
432 file_path TEXT NOT NULL,
433 line INTEGER NOT NULL,
434 edge_kind TEXT NOT NULL CHECK (edge_kind IN ('direct','indirect','virtual','macro','unresolved')),
435 corpus TEXT DEFAULT 'default',
436 partition_key TEXT,
437 source_layer TEXT NOT NULL DEFAULT 'baseline',
438 FOREIGN KEY (caller_id) REFERENCES symbols(symbol_id),
439 FOREIGN KEY (callee_id) REFERENCES symbols(symbol_id)
440);
441
442CREATE TABLE IF NOT EXISTS function_boundaries (
443 symbol_id TEXT PRIMARY KEY,
444 file_path TEXT NOT NULL,
445 start_line INTEGER NOT NULL,
446 end_line INTEGER NOT NULL,
447 signature TEXT,
448 language TEXT NOT NULL,
449 FOREIGN KEY (symbol_id) REFERENCES symbols(symbol_id)
450);
451
452CREATE TABLE IF NOT EXISTS coverage (
453 file_path TEXT NOT NULL,
454 indexed INTEGER NOT NULL DEFAULT 0,
455 backend TEXT,
456 precision TEXT,
457 unit_cache_key TEXT,
458 status TEXT NOT NULL DEFAULT 'pending',
459 corpus TEXT DEFAULT 'default',
460 partition_key TEXT,
461 PRIMARY KEY (file_path, corpus)
462);
463
464CREATE TABLE IF NOT EXISTS unit_cache (
465 cache_key TEXT PRIMARY KEY,
466 unit_id TEXT NOT NULL,
467 language TEXT NOT NULL,
468 extractor_name TEXT NOT NULL,
469 extractor_version TEXT NOT NULL,
470 file_count INTEGER NOT NULL,
471 symbol_count INTEGER NOT NULL,
472 edge_count INTEGER NOT NULL,
473 source_bytes INTEGER NOT NULL,
474 created_at TEXT NOT NULL,
475 snapshot_id TEXT NOT NULL
476);
477
478CREATE TABLE IF NOT EXISTS deferred_units (
479 unit_id TEXT PRIMARY KEY,
480 language TEXT NOT NULL,
481 files TEXT NOT NULL,
482 priority INTEGER NOT NULL,
483 reason TEXT NOT NULL,
484 cache_key TEXT,
485 created_at TEXT NOT NULL
486);
487
488-- Indexes: both directions!
489CREATE INDEX IF NOT EXISTS idx_symbols_name ON symbols(name);
490CREATE INDEX IF NOT EXISTS idx_symbols_qualified ON symbols(qualified_name);
491CREATE INDEX IF NOT EXISTS idx_symbols_file ON symbols(file_path, start_line);
492CREATE INDEX IF NOT EXISTS idx_symbols_lang ON symbols(language);
493CREATE INDEX IF NOT EXISTS idx_edges_caller ON call_edges(caller_id);
494CREATE INDEX IF NOT EXISTS idx_edges_callee ON call_edges(callee_id);
495CREATE INDEX IF NOT EXISTS idx_edges_callee_name ON call_edges(callee_name);
496CREATE INDEX IF NOT EXISTS idx_edges_file ON call_edges(file_path, line);
497CREATE INDEX IF NOT EXISTS idx_boundaries_file ON function_boundaries(file_path, start_line, end_line);
498CREATE INDEX IF NOT EXISTS idx_coverage_status ON coverage(status);
499CREATE INDEX IF NOT EXISTS idx_deferred_priority ON deferred_units(priority);
500```
501
502**Partitioning**: When symbols exceed 500K or edges exceed 2M, partition into
503shard SQLite files under `shards/` by `lang:{language}:bucket:{NN}` (stable
504hash). Each shard has the same schema. The query helper routes queries to the
505correct shard(s) based on the manifest.
506
507### Content-Addressed Cache Key
508
509```python
510cache_key = sha256(
511 schema_version +
512 extractor_name@version +
513 language +
514 compile_context_digest +
515 ordered_input_content_digests +
516 dependency_or_interface_digest
517)
518```
519
520**Critical**: `snapshot_id` is NOT in the cache key. It goes in provenance only.
521This permits reuse across commits, branches, and snapshots with identical units.
522
523### Canonical Symbol IDs
524
525**Native IDs** (when semantic backend is available):
526
527- SCIP: `scip:{symbol}`
528- Kythe: `kythe:{uri}`
529- clangd: `clangd:{usr}`
530
531**Fallback IDs** (when no semantic backend):
532
533```
534fallback:{language}:{file_path}:{sha256(qualified_name|start_line|signature)[:16]}
535```
536
537This distinguishes namespaces, overloads, methods, local functions, and
538duplicate names across languages or repositories.
539
540### Edge Kinds
541
542Edge kinds are kept explicitly separate — never silently merged:
543
544- `direct` — statically resolved call to a known symbol
545- `indirect` — function pointer, closure, callback
546- `virtual` — virtual method dispatch (runtime-resolved)
547- `macro` — call introduced by macro expansion
548- `unresolved` — callee name found but no symbol_id resolved
549
550### Per-Language Semantic Units
551
552| Language | Unit Type | Compile Context | Dependency Digest |
553| ----------- | ---------------- | --------------------------- | ------------------------------------------- |
554| C/C++ | Compilation unit | compile_commands.json entry | sha256 of transitive header interfaces |
555| Go | Package | go.mod + build tags | sha256 of imported packages' exported API |
556| Rust | Crate | Cargo.toml + features | sha256 of extern crate signatures |
557| Java/Kotlin | Compiler batch | module path + classpath | sha256 of imported class signatures |
558| TypeScript | Project | tsconfig.json | sha256 of imported module type declarations |
559| Fallback | Individual file | None | Empty string |
560
561**Fallback Granularity Rule:** For the fallback tier (regex, AST parser, or
562lightweight crawler without a language build system), **each source file MUST be
563its own independent semantic unit**. Never bundle multiple source files into a
564single fallback unit. Per-file unit isolation is essential for content-addressed
565cache efficiency — when one file changes in a future snapshot, only that single
566file's unit invalidates while all unchanged files hit the cache
567(`reused = N-1`).
568
569### Baseline + Delta Overlay
570
571- **Static baseline**: Built once in CI for the complete snapshot.
572- **Incremental cache**: Immutable unit outputs reused across snapshots.
573- **Delta overlay**: Changed/target files indexed locally, layered over
574 baseline.
575- **Remote serving**: Optional when global index is too costly locally.
576
577At query time, the query helper merges baseline + overlay results. Overlay
578symbols supersede baseline symbols for the same file. The manifest records
579`provider.kind = "baseline+overlay"` with baseline manifest reference and
580overlay unit count.
581
582### Compatibility JSONL Export
583
584The old `structural_index.jsonl` format is retained as a compatibility export
585only — it is NOT the query contract. Below the threshold (default 10K records),
586a complete JSONL export is emitted. Above the threshold, only a provenance
587pointer is written. Large consumers MUST use the query interface.
588
589Line 1 — provenance header:
590
591```json
592{"_provenance": true, "snapshot_id": "abc123", "tool": "tree-sitter"}
593```
594
595Lines 2+ — structural records (one per line, `_type` discriminator):
596
597```json
598{"_type": "function", "key": "src/parser.c:parse_input", "start_line": 45, "end_line": 120, "signature": "int parse_input(char *buf, size_t len)", "calls": ["malloc", "validate_input", "memcpy"]}
599{"_type": "call_edge", "caller": "parse_input", "callee": "malloc", "file": "src/parser.c", "line": 78}
600{"_type": "call_edge", "caller": "main", "callee": "parse_input", "file": "src/main.c", "line": 203}
601```
602
603**Provenance header fields:**
604
605| Field | Type | Description |
606| ------------- | ------ | ----------------------------------------------------------------------------------- |
607| `_provenance` | bool | Always `true` — marks this as the provenance header line |
608| `snapshot_id` | string | `SNAPSHOT_ID` the index was built against |
609| `tool` | string | Backend used (e.g. `"lsp-clangd"`, `"tree-sitter"`, `"ctags"`, `"regex"`, `"grep"`) |
610
611**Record types:**
612
613| `_type` | Description | Key fields |
614| ----------- | ----------------------------------------------- | ----------------------------------------------------- |
615| `function` | Function definition with boundary and signature | `key`, `start_line`, `end_line`, `signature`, `calls` |
616| `call_edge` | A call from caller to callee at file:line | `caller`, `callee`, `file`, `line` |
617
618## Snapshot Safety
619
6201. **Build from `CODE_ROOT`, not live tree (when pinned).** When the snapshot is
621 pinned, the structural index is built from the pinned `CODE_ROOT`, ensuring
622 it reflects the exact bytes the pipeline is analyzing.
6232. **Reuse-on-match.** If `manifest.json` already carries the current
624 `SNAPSHOT_ID` (and `SNAPSHOT_ID` is NOT `"unknown"` — in MODE-OFF, always
625 rebuild), reuse it — do not rebuild. Rebuild when `SNAPSHOT_ID` differs or is
626 `"unknown"`. Individual units may still be reused from the content-addressed
627 cache. This bounds cost across retries/crash-resume.
6283. **Manifest atomicity.** The manifest is written LAST via atomic rename from
629 `tmp/`. Interrupted builds leave unreferenced temp objects without corrupting
630 the last published index. On resume, check `manifest.json` status and
631 `snapshot_id`.
6324. **STALE flag in HALT mode.** When `snapshot_pinned` is false (HALT), the
633 index may be built from the unpinned `CODE_ROOT` but is marked as potentially
634 stale. Consumers treat structural hints as advisory.
6355. **MODE-OFF: build, do not skip.** When `active_snapshot` is absent
636 (MODE-OFF), build against the current directory (cwd) with provenance
637 `snapshot_id` set to `"unknown"`. This is the standalone-efficiency case —
638 the index is still useful for the researcher/planner even without snapshot
639 pinning. Do NOT return an empty index merely because the snapshot is absent.
640
641## Consumption Contract
642
643These are runtime instructions for callers — they define how consumers use the
644structural index. The structural index is a HINT-only enhancement; skills that
645do not use it behave exactly as they do today.
646
647### Query Interface
648
649**The query helper IS the contract.** All consumers use
650`query_structural_index.py` (or a compatible remote endpoint). The JSONL file is
651NOT the query contract — it is a compatibility export only.
652
653**Operations**:
654
6551. `resolve_symbol(name, language?, file?, namespace?)` →
656 `{results, total, ambiguous, coverage}`
657
658 - Returns ALL matches — no silent selection among ambiguous symbols.
659 - Caller MUST disambiguate before calling `find_callers` / `find_callees`.
660
6612. `find_callers(symbol_id, limit=100, offset=0)` →
662 `{results, total, has_more, coverage}`
663
664 - Bounded, paginated caller lookup.
665 - Each result carries `precision`, `backend`, `edge_kind`.
666 - Empty results carry `coverage.partition_status`.
667
6683. `find_callees(symbol_id, limit=100, offset=0)` →
669 `{results, total, has_more, coverage}`
670
671 - Same shape as `find_callers`.
672
6734. `get_function_boundary(file, line)` →
674 `{symbol_id, start_line, end_line, signature, precision, backend}`
675
6765. `get_coverage(file?)` →
677 `{total_files, indexed_files, failed, deferred, partition_status, backends_used}`
678
679**Key properties**:
680
681- Bounded results and pagination (prevents loading entire index into memory).
682- Explicit name resolution before graph traversal (no silent selection among
683 ambiguous symbols).
684- Precision and backend attached to every result.
685- Coverage attached to empty results.
686- Identical operations for local and remote providers (remote via
687 `MANTIS_STRUCTURAL_INDEX_URL` env var or manifest `provider.kind = "remote"`).
688
689**Coverage on empty results**:
690
691| `partition_status` | Meaning | Consumer action |
692| ------------------ | ----------------------------- | -------------------------------------------------------- |
693| `complete` | All files indexed | "No indexed callers" (still run grep per HINT-only rule) |
694| `partial` | Some files deferred or failed | "Not fully indexed" — MUST run grep fallback |
695| `empty` | No backend available | "Not indexed" — MUST run grep fallback |
696| `failed` | Backend attempted but failed | "Index failed" — MUST run grep fallback |
697
698### mantis-plan
699
700- Use the structural index query helper for function-level dependency fan-out.
701 When planning investigations, call `resolve_symbol()` then `find_callers()` to
702 identify all functions that call into a target — this broadens the audit set
703 beyond single-file analysis.
704- The structural index decides ORDER of investigations (which functions to audit
705 first based on call-graph centrality), never MEMBERSHIP. It may broaden the
706 audit set (safe over-reporting), but must never REMOVE or drop a file; the
707 planner's existing logic remains the membership floor.
708
709### mantis-researcher
710
711- **Wave 1 (Rapid Triage):** Run a repo-wide grep for the function name to build
712 the exhaustive set of candidate call-sites — this is the mandatory floor. Then
713 use the structural index query helper (`resolve_symbol` then `find_callers`)
714 to RANK and prioritize which call-sites to audit first (the index
715 distinguishes actual calls from comments/strings/variable names). Audit the
716 union of both result sets — the structural index may miss macro-based calls,
717 function pointers, and dynamic dispatch, so grep remains the floor.
718- **Wave 2 (Deep Audit):** Use `get_function_boundary(file, line)` to start with
719 the enclosing function, expanding to callers/callees/file as needed for
720 cross-function context — this saves context while preserving coverage.
721- If the structural index is absent or empty, fall back to grep-based discovery
722 (today's behavior). The structural index is a coverage HINT only — it improves
723 audit quality but is never required.
724
725## Safety
726
727Non-negotiable invariants:
728
7291. **Agnostic.** Nothing is ever required; no-tool / parse-fail / not-invoked →
730 empty index → grep fallback = today's behavior byte-for-byte. Optional in the
731 Pass Lifecycle Contract; a non-conformant harness simply skips it.
7322. **HINT-only / union / never MEMBERSHIP.** Consumers audit the union with
733 grep; a symbol the index misses must still be reachable by the exhaustive
734 sweep. The structural index decides ORDER, never MEMBERSHIP. The query
735 contract enforces this: `find_callers` returns HINTs, never authoritative
736 membership — consumers MUST union with grep.
7373. **No verdicts, touches no findings.** Cannot violate INV-1 and cannot itself
738 drop a finding.
7394. **Narrow scope: source cross-reference only.** Binary/build-derived
740 reachability ("is it compiled into production") is explicitly out of scope —
741 a dev customization, not part of this skill, because absence-from-a-build can
742 hide a real finding (INV-2). Do not fold build-derived reachability into this
743 skill.
744
745A reference blueprint is available at
746[mantis-pipeline-adapter/references/mantis-structural-index.md](../mantis-pipeline-adapter/references/mantis-structural-index.md).
747It is a stub that points to this SKILL.md as the single source of truth — do not
748duplicate spec content there.