Mantis Structural Index

Structural Code Index Builder

This is an optional first-class stage in the Pass Lifecycle Contract. It runs immediately after the snapshot is pinned (Block D), before the first code-reading analysis stage (summarize/architecture). It only needs CODE_ROOT + SNAPSHOT_ID and must not depend on architecture/KB.

System Goal

Structural Code Index Builder. Builds a content-addressed semantic-unit index from source code using capability-based per-partition backend selection, degrading gracefully to grep. Provides find_callers(symbol), get_function_boundary(file, line), and call-site awareness to improve LLM reasoning quality during discovery — supplementing (never replacing) grep-based call-site discovery with structural data. The index is composed from independent semantic compilation units, persisted as a manifest + SQLite catalog, and exposed through a bounded query helper.

Command Definition

• Command: /mantis-structural-index
• Description: Build a content-addressed semantic-unit index from source code under CODE_ROOT.
• Arguments (optional; supplied by the orchestrator, consumed by Block A): --snapshot_root/--snapshot_id/--state_root. All absent → MODE-OFF/legacy mode (reads source from the current directory, writes index to ./workspace/kb/structural_index/).

Input/Output Contract

• Reads:
  • workspace/.mantis_state.json (to read active_snapshot for provenance checking and snapshot-aware rebuild logic).
  • workspace/kb/structural_index/manifest.json (to check snapshot_id for reuse-on-match idempotency — primary check).
  • workspace/kb/structural_index.jsonl (backward-compat provenance check if manifest absent).
  • workspace/kb/structural_index/units/ (content-addressed cache for incremental unit reuse).
  • workspace/kb/structural_index/native/ and sidecar provenance.json files (prebuilt index attachments and metadata manifests).
  • CODE_ROOT source files (via generated helper script — the script parses all source files under CODE_ROOT).
• Writes:
  • workspace/kb/structural_index/manifest.json (STATE-RELATIVE — atomic commit point, written LAST).
  • workspace/kb/structural_index/catalog.sqlite (STATE-RELATIVE — query-optimized serving store).
  • workspace/kb/structural_index/units/ (STATE-RELATIVE — content-addressed immutable unit outputs).
  • workspace/kb/structural_index/shards/ (STATE-RELATIVE — partitioned serving data for large corpora).
  • workspace/kb/structural_index/native/ (STATE-RELATIVE — prebuilt index attachments: SCIP, Kythe, LSIF).
  • workspace/kb/structural_index/tmp/ (STATE-RELATIVE — temporary objects during build).
  • workspace/kb/structural_index.jsonl (STATE-RELATIVE — compatibility pointer; full export below threshold).
  • workspace/helpers/build_structural_index.py (STATE-RELATIVE — the builder helper script).
  • workspace/helpers/query_structural_index.py (STATE-RELATIVE — the query helper script).
• Preconditions:
  • Source files must exist under CODE_ROOT. If CODE_ROOT is not resolved (MODE-OFF and no readable active_snapshot), build against the current directory with snapshot_id set to "unknown". Do NOT skip — this is the standalone-efficiency case.
• Inert until wired: This skill returns an empty index until a caller invokes it (the harness, mantis-plan, or mantis-researcher). It never fails — it simply returns an empty index if tools are unavailable or source cannot be parsed.
• Idempotency Guarantee:
  • Read-only on CODE_ROOT. Writes only to STATE-RELATIVE paths. Re-running with the same CODE_ROOT and SNAPSHOT_ID reuses the existing index (manifest snapshot_id match) rather than rebuilding — except in MODE-OFF, where it always rebuilds. Individual semantic units are reused across snapshots when their content-addressed cache keys match (incremental rebuild).

Instructions

Step 0: Locator Resolution (run first)

LOCATOR RESOLUTION (before reading ANY target code or artifact):
0. ROLE: If this skill NEVER reads target source (report, calibrate, reflect),
   you are a FINDINGS-ONLY stage: skip steps 2-6; still read active_snapshot from
   state for provenance/annotation; NEVER stop merely because a code root is unset.
1. Determine CODE_ROOT, in this priority order:
   a. If --target_root is passed on THIS invocation, CODE_ROOT = --target_root.
      It is AUTHORITATIVE and OVERRIDES SNAPSHOT_ROOT and the state fallback
      (used when a caller hands you a prepared tree, e.g. a patched shadow).
   b. Else if --snapshot_root (or SNAPSHOT_ROOT) is passed, use it.
   c. Else read state_root/workspace/.mantis_state.json (state_root from
      --state_root if passed, else ./workspace/... relative to the current dir)
      -> active_snapshot.root / .snapshot_id / .snapshot_pinned.
   d. Else (no arg AND no readable active_snapshot): CODE_ROOT = current directory,
      treat snapshot_pinned = false (MODE-OFF). Do NOT stop.
2. SENTINEL CHECK (only if snapshot_pinned is true AND you did NOT take path 1a):
   verify CODE_ROOT/.mantis_snapshot_id exists and equals SNAPSHOT_ID. If missing
   or different -> STOP "snapshot sentinel mismatch". (A --target_root tree (1a) is
   deliberately mutated and is sentinel-EXEMPT.)
3. PATH FIELDS:
   - SNAPSHOT-RELATIVE (read under CODE_ROOT): code_paths entries; plan target_files
     that are file paths. Strip ONLY a trailing ":<digits>". A code_paths entry
     containing "://" is a URL/endpoint, NOT a file read. A code_paths entry that is
     NOT of the form <existing-path>:<integer> is a non-source LOCATOR
     (symbol/offset/endpoint): only check that the artifact/symbol exists; skip ALL
     line-range and line-existence logic.
   - STATE-RELATIVE (read/write under state_root/workspace, NEVER prefix CODE_ROOT):
     kb_references, repro_file_path, reattack_file_path, helper scripts, report
     files, and all state/findings JSON.
4. Never WRITE under CODE_ROOT when snapshot_pinned is true. Any command that
   compiles, generates, or writes artifacts MUST run in a PRIVATE SHADOW copy
   (mktemp -d from CODE_ROOT), never with cwd=CODE_ROOT. Read-only inspection may
   cd into CODE_ROOT.
5. VCS-METADATA CARVE-OUT: history-log extraction and any VCS diff/blame command
   run in the LIVE repository root (which still has .git/.hg/.repo), NOT CODE_ROOT
   (the snapshot copy strips VCS metadata). Do NOT stop merely because CODE_ROOT
   lacks .git/.hg/.repo.
6. Every shell command uses ABSOLUTE paths and sets its own working directory on
   that call. Do NOT assume the working directory persists between calls.

This is a CODE-READING stage — it reads target source files under CODE_ROOT via the helper script. Block A step 0's findings-only skip does NOT apply.

Step 1: Idempotency / Freshness Check (MANDATORY FIRST)

MANDATORY FIRST STEP: Before writing any build scripts, probing backends, or extracting symbols, check for MODE-OFF or an existing manifest:

1. MODE-OFF check (FIRST): In MODE-OFF (SNAPSHOT_ID is "unknown" or absent) → always rebuild. Do NOT reuse a previous "unknown" index, because the live tree is mutable and "unknown" is a constant (not a freshness signal). Skip directly to Step 2 (individual units whose content has not changed may still hit the content-addressed cache).
2. If workspace/kb/structural_index/manifest.json exists, read its snapshot_id field.
3. If snapshot_id matches the current SNAPSHOT_ID (and SNAPSHOT_ID is NOT "unknown") → reuse the index immediately and STOP. Do NOT invoke Step 2, probe backends, or write build scripts. This bounds cost across retries/crash-resume.
4. If manifest.json is absent but workspace/kb/structural_index.jsonl exists (backward compat), read its provenance header (_provenance, snapshot_id keys). If snapshot_id matches (and is NOT "unknown") → reuse and STOP. Otherwise proceed to rebuild.
5. If SNAPSHOT_ID differs → proceed to rebuild (Steps 2–5). For incremental reuse: before rebuilding a semantic unit, check units/ for a content-addressed cache hit (see Content-Addressed Cache Key below). A cache hit reuses the unit output without re-extraction.

Step 2: Select Backend per Partition

Partition the codebase into semantic units (see Per-Language Semantic Units below). For each unit, select the most precise backend available in this environment using a capability-based, per-partition decision — not a single global ladder. The following are examples, not an exhaustive enum. The helper probes each tier per partition and selects the highest available:

1. Snapshot-matched SCIP / LSIF / Kythe / clangd-static / remote index (precision: semantic) — if a pre-built index matching the current snapshot_id or root_fingerprint is available. Most precise: full type-aware cross-reference, call hierarchy, and hover/signature data.
2. Compiler / typechecker-backed extractor (precision: typecheck) — if compile_commands.json, build context, or typechecker is available. Type-accurate symbol resolution and call edges.
3. Language-aware AST extraction (precision: ast) — if tree-sitter, ast-grep, or a language-specific parser is available. Full AST parsing: function boundaries, call expressions, signatures.
4. Symbol-only extraction (precision: symbol-only) — if ctags or equivalent is on PATH. Symbol table only (function definitions, locations — no call graph). Call-site extraction uses a lightweight regex pass within known function boundaries.
5. Heuristic fallback (precision: heuristic) — Python stdlib regex pass over source files. Identifies function definitions and call patterns using language-agnostic heuristics. Less precise but zero-dependency.
6. Coverage-only manifest + lexical fallback (precision: coverage-only) — grep; no structural index is written. Manifest records status: "empty". Consumers fall back to grep-based discovery (today's behavior byte-for-byte).

Native Index Probing & Resolution Rules:

• Probe instruction: Before evaluating per-partition backends, probe workspace/kb/structural_index/native/ and subdirectories native/{scip,lsif,kythe}/ for prebuilt index files.
• Snapshot-declaration convention: Because native formats (SCIP, LSIF, Kythe) do not embed snapshot identity directly in their binary payload, prebuilt indexes MUST declare their target snapshot using a sidecar provenance.json manifest located at workspace/kb/structural_index/native/provenance.json or native/<kind>/provenance.json. The manifest contains an array of attachments: [{"kind": "scip|lsif|kythe", "path": "...", "snapshot_id": "...", "root_fingerprint": "...", "language": "...", "indexer": "...", "precision": "semantic", "files": [...]}]. A native index is matched if its declared snapshot_id equals SNAPSHOT_ID (when SNAPSHOT_ID != "unknown") or its root_fingerprint matches the workspace's calculated root fingerprint. If the files array is absent or empty, treat the native index as covering no individual files directly (record in manifest.native_indexes but do not update coverage rows; fall through to lower tiers for all files).
• Record-and-Defer Ingestion Rule: Parsing raw binary native indexes (e.g. SCIP protobuf) in pure Python without dependencies is costly and complex. Option A builder scripts MUST detect matching prebuilt native indexes, record their entries in manifest.native_indexes, and set the coverage table backend (e.g., "scip" or "scip-clangd") for all files listed in the provenance manifest. If catalog.sqlite is NOT populated with symbols from the native index (raw binary deferred to harness/MCP readers), set coverage.status = "deferred" and precision = "deferred" (with indexed_files = 0). This prevents the query helper from claiming an un-ingested partition is "authoritative empty" at semantic precision, ensuring consumers run the mandatory grep fallback. When catalog.sqlite IS populated (e.g., via Option B pre-ingestion or scip-to-sqlite), set precision = "semantic" and status = "indexed".

LSP is NOT equivalent to SCIP / LSIF. LSP is an interactive protocol whose workspace state may be partial or mutable. Use it only when the server can demonstrate snapshot identity AND complete workspace coverage. A running language server does not automatically qualify as a semantic backend.

SCIP merging. SCIP explicitly permits merging complementary information from indexers with different precision levels. Its format records indexer version and per-document language metadata. The builder MAY merge results from multiple indexers (e.g., a SCIP index for Go + tree-sitter for Python) within a single catalog.

The determinism lives in a runtime-generated versioned helper (build_structural_index.py, # MANTIS_HELPER_VERSION = 5, grep-and-regenerate on reuse) that probes and selects backends per partition. No shipped binaries; air-gapped-safe.

Deterministic partial coverage. On very large source trees, a full rebuild can dominate stage-0 wall-clock. Replace discovery-order truncation with a deterministic priority queue:

1. Explicit target files and symbols (from plan.json, if available).
2. Changed units and known reverse dependencies (from Block E diff + reverse-dependency edges in the existing catalog).
3. Containing packages/modules and direct imports of tier 1+2 units.
4. Remaining units in normalized path order.

Apply deterministic max_units (default: 10000) or max_source_bytes (default: 500MB) bounds to that queue. Persist deferred units in the manifest so another invocation can resume. Statuses: complete, partial, empty, failed.

A consumer should only interpret "no callers" strongly when the relevant partition is complete, snapshot-matched, covered, and semantically precise. Otherwise the answer is "no indexed callers", followed by the existing lexical fallback. The index is HINT-only, so partial coverage is safe (grep remains authoritative).

Step 3: Write and Run Helper Scripts

Two runtime-generated helpers are used. Both follow the grep-and-regenerate pattern: before reuse, grep the first line for the version marker; if absent or a different integer, REGENERATE.

Builder: build_structural_index.py

1. Write the builder to workspace/helpers/build_structural_index.py. The FIRST LINE MUST be exactly # MANTIS_HELPER_VERSION = 5. Before reusing an existing helper, grep its first lines for MANTIS_HELPER_VERSION = 5; if that marker is absent or a different integer, REGENERATE the helper.
2. The builder partitions the codebase into semantic units (see Per-Language Semantic Units), computes content-addressed cache keys, checks units/ for reuse, selects a backend per partition (Step 2), extracts symbols + edges, and writes results to catalog.sqlite.
3. The builder must extract:
   • Symbols: symbol_id, name, qualified_name, namespace, language, file_path, start_line, end_line, kind, signature, backend, precision.
   • Call edges: caller_id, callee_id, callee_name, file_path, line, edge_kind.
   • Function boundaries: symbol_id, file_path, start_line, end_line, signature, language.
   • Coverage: file_path, indexed, backend, precision, unit_cache_key, status.
4. The builder writes the manifest LAST (Step 4) and the compatibility pointer.
5. The builder MUST use ABSOLUTE paths and set its own working directory (Block A step 6). It MUST NOT write anything under CODE_ROOT when snapshot_pinned is true (Block A step 4). Backends that produce sidecar files (ctags tags, cscope cscope.out, clangd cache) MUST be redirected to STATE-RELATIVE paths: ctags -f <state>/helpers/tags, cscope -f <state>/helpers/cscope.out, CLANGD_INDEX_STORAGE=<state>/helpers/. Read-only LSP/SCIP queries to a running server need no redirect.

Query helper: query_structural_index.py

1. Write the query helper to workspace/helpers/query_structural_index.py. The FIRST LINE MUST be exactly # MANTIS_HELPER_VERSION = 5. Before reusing an existing helper, grep its first lines for MANTIS_HELPER_VERSION = 5; if absent or a different integer, REGENERATE.
2. The query helper provides bounded, paginated operations against catalog.sqlite (or a remote endpoint — identical API). It IS the consumption contract (see Query Interface below).
3. If catalog.sqlite is absent but structural_index.jsonl exists, the query helper falls back to linear scanning of the JSONL file (slower but functional). If both are absent, it returns empty results with coverage.partition_status = "empty".

Step 4: Write Manifest and Commit

1. Write the manifest to workspace/kb/structural_index/tmp/manifest.json first. Then atomically rename it to workspace/kb/structural_index/manifest.json. This is the atomic commit point — the manifest is written LAST, after all units, catalog, and coverage data are written.
2. Write a compatibility pointer to workspace/kb/structural_index.jsonl (STATE-RELATIVE — NEVER under CODE_ROOT). Below a configurable threshold (default: 10K records), emit a complete JSONL export (provenance header + all records). Above the threshold, emit only a provenance header with compat_pointer.full_export = false and compat_pointer.symbol_count set. Large consumers MUST use the query interface.
3. The manifest provider.backend_versions field records which backend was used per language (e.g., {"go": {"backend_name": "scip-clangd", "precision": "semantic"}, "python": {"backend_name": "tree-sitter", "precision": "ast"}}), so consumers know the precision level.
4. Interrupted builds leave unreferenced temp objects in tmp/ without corrupting the last published index. On resume, check manifest.json status and snapshot_id (Step 1).

Step 5: Return Results / Notify Caller

1. Return the path to manifest.json, catalog.sqlite, and the query helper. Include a summary (unit count, symbol count, call-edge count, backends used, coverage status).
2. If the index is empty (no tools available or no source files found), notify the caller: "Structural index is empty — structural context unavailable."
3. Do not notify the user directly — this skill is invoked as a sub-agent by the harness, planner, or researcher.

Index Schema and On-Disk Contract

On-Disk Layout

workspace/kb/structural_index/
├── manifest.json          # Atomic commit point — written LAST
├── catalog.sqlite         # Query-optimized serving store (both directions indexed)
├── units/                 # Content-addressed immutable unit outputs
│   └── ab/cd/abcdef...    # sha256 prefix sharding (2+2 hex dirs)
├── shards/                # Partitioned serving data (large corpora)
│   └── shard_0000.sqlite
├── native/                # Prebuilt index attachments (SCIP, Kythe, LSIF)
│   ├── provenance.json    # Prebuilt index provenance manifest
│   ├── scip/
│   └── kythe/
└── tmp/                   # Temporary objects during build

workspace/helpers/
├── build_structural_index.py    # Builder (MANTIS_HELPER_VERSION = 5)
└── query_structural_index.py    # Query helper (MANTIS_HELPER_VERSION = 5)

workspace/kb/structural_index.jsonl  # Compatibility pointer

Manifest Schema (manifest.json)

{
  "schema_version": 1,
  "snapshot_id": "<SNAPSHOT_ID or 'unknown'>",
  "root_fingerprint": "<sha256 of sorted (path, content_sha256) for all source files>",
  "status": "complete|partial|empty|failed",
  "provider": {
    "kind": "local-build|baseline+overlay|remote",
    "catalog": "catalog.sqlite",
    "backend_versions": {
      "cpp": {"backend_name": "tree-sitter", "backend_version": "0.20.8", "precision": "ast"},
      "go": {"backend_name": "scip-clangd", "backend_version": "0.2.3", "precision": "semantic"}
    }
  },
  "units": {"total": 0, "reused": 0, "rebuild": 0, "failed": 0},
  "coverage": {"total_files": 0, "indexed_files": 0, "failed_files": 0, "deferred_files": 0},
  "shards": [{"id": "", "path": "", "checksum": "", "partition_key": "", "symbol_count": 0, "edge_count": 0}],
  "deferred_units": [{"unit_id": "", "language": "", "files": [], "priority": 4, "reason": ""}],
  "native_indexes": [{"kind": "scip", "path": "", "snapshot_id": "", "root_fingerprint": "", "language": "", "indexer": "", "precision": ""}],
  "baseline": {"source": "ci|local|none", "snapshot_id": "", "manifest_path": ""},
  "overlay": {"units_added": 0, "units_modified": 0, "files": []},
  "compat_pointer": {"path": "structural_index.jsonl", "full_export": true, "symbol_count": 0},
  "created_at": "<ISO 8601>",
  "build_duration_ms": 0
}

Atomic commit: The manifest is written LAST (atomic rename from tmp/). Interrupted builds leave unreferenced temp objects without corrupting the last published index.

SQLite Catalog Schema (catalog.sqlite)

CREATE TABLE IF NOT EXISTS schema_meta (
    key TEXT PRIMARY KEY, value TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS symbols (
    symbol_id       TEXT PRIMARY KEY,
    name            TEXT NOT NULL,
    qualified_name  TEXT NOT NULL,
    namespace       TEXT,
    language        TEXT NOT NULL,
    file_path       TEXT NOT NULL,
    start_line      INTEGER NOT NULL,
    end_line        INTEGER,
    kind            TEXT NOT NULL,
    signature       TEXT,
    backend         TEXT NOT NULL,
    precision       TEXT NOT NULL CHECK (precision IN ('semantic','typecheck','ast','symbol-only','heuristic','deferred','coverage-only')),
    corpus          TEXT DEFAULT 'default',
    partition_key   TEXT,
    unit_cache_key  TEXT,
    source_layer    TEXT NOT NULL DEFAULT 'baseline'
);

CREATE TABLE IF NOT EXISTS call_edges (
    edge_id     INTEGER PRIMARY KEY AUTOINCREMENT,
    caller_id   TEXT NOT NULL,
    callee_id   TEXT,
    callee_name TEXT NOT NULL,
    file_path   TEXT NOT NULL,
    line        INTEGER NOT NULL,
    edge_kind   TEXT NOT NULL CHECK (edge_kind IN ('direct','indirect','virtual','macro','unresolved')),
    corpus      TEXT DEFAULT 'default',
    partition_key TEXT,
    source_layer TEXT NOT NULL DEFAULT 'baseline',
    FOREIGN KEY (caller_id) REFERENCES symbols(symbol_id),
    FOREIGN KEY (callee_id) REFERENCES symbols(symbol_id)
);

CREATE TABLE IF NOT EXISTS function_boundaries (
    symbol_id   TEXT PRIMARY KEY,
    file_path   TEXT NOT NULL,
    start_line  INTEGER NOT NULL,
    end_line    INTEGER NOT NULL,
    signature   TEXT,
    language    TEXT NOT NULL,
    FOREIGN KEY (symbol_id) REFERENCES symbols(symbol_id)
);

CREATE TABLE IF NOT EXISTS coverage (
    file_path   TEXT NOT NULL,
    indexed     INTEGER NOT NULL DEFAULT 0,
    backend     TEXT,
    precision   TEXT,
    unit_cache_key TEXT,
    status      TEXT NOT NULL DEFAULT 'pending',
    corpus      TEXT DEFAULT 'default',
    partition_key TEXT,
    PRIMARY KEY (file_path, corpus)
);

CREATE TABLE IF NOT EXISTS unit_cache (
    cache_key       TEXT PRIMARY KEY,
    unit_id         TEXT NOT NULL,
    language        TEXT NOT NULL,
    extractor_name  TEXT NOT NULL,
    extractor_version TEXT NOT NULL,
    file_count      INTEGER NOT NULL,
    symbol_count    INTEGER NOT NULL,
    edge_count      INTEGER NOT NULL,
    source_bytes    INTEGER NOT NULL,
    created_at      TEXT NOT NULL,
    snapshot_id     TEXT NOT NULL
);

CREATE TABLE IF NOT EXISTS deferred_units (
    unit_id     TEXT PRIMARY KEY,
    language    TEXT NOT NULL,
    files       TEXT NOT NULL,
    priority    INTEGER NOT NULL,
    reason      TEXT NOT NULL,
    cache_key   TEXT,
    created_at  TEXT NOT NULL
);

-- Indexes: both directions!
CREATE INDEX IF NOT EXISTS idx_symbols_name ON symbols(name);
CREATE INDEX IF NOT EXISTS idx_symbols_qualified ON symbols(qualified_name);
CREATE INDEX IF NOT EXISTS idx_symbols_file ON symbols(file_path, start_line);
CREATE INDEX IF NOT EXISTS idx_symbols_lang ON symbols(language);
CREATE INDEX IF NOT EXISTS idx_edges_caller ON call_edges(caller_id);
CREATE INDEX IF NOT EXISTS idx_edges_callee ON call_edges(callee_id);
CREATE INDEX IF NOT EXISTS idx_edges_callee_name ON call_edges(callee_name);
CREATE INDEX IF NOT EXISTS idx_edges_file ON call_edges(file_path, line);
CREATE INDEX IF NOT EXISTS idx_boundaries_file ON function_boundaries(file_path, start_line, end_line);
CREATE INDEX IF NOT EXISTS idx_coverage_status ON coverage(status);
CREATE INDEX IF NOT EXISTS idx_deferred_priority ON deferred_units(priority);

Partitioning: When symbols exceed 500K or edges exceed 2M, partition into shard SQLite files under shards/ by lang:{language}:bucket:{NN} (stable hash). Each shard has the same schema. The query helper routes queries to the correct shard(s) based on the manifest.

Content-Addressed Cache Key

cache_key = sha256(
    schema_version +
    extractor_name@version +
    language +
    compile_context_digest +
    ordered_input_content_digests +
    dependency_or_interface_digest
)

Critical: snapshot_id is NOT in the cache key. It goes in provenance only. This permits reuse across commits, branches, and snapshots with identical units.

Canonical Symbol IDs

Native IDs (when semantic backend is available):

• SCIP: scip:{symbol}
• Kythe: kythe:{uri}
• clangd: clangd:{usr}

Fallback IDs (when no semantic backend):

fallback:{language}:{file_path}:{sha256(qualified_name|start_line|signature)[:16]}

This distinguishes namespaces, overloads, methods, local functions, and duplicate names across languages or repositories.

Edge Kinds

Edge kinds are kept explicitly separate — never silently merged:

• direct — statically resolved call to a known symbol
• indirect — function pointer, closure, callback
• virtual — virtual method dispatch (runtime-resolved)
• macro — call introduced by macro expansion
• unresolved — callee name found but no symbol_id resolved

Per-Language Semantic Units

Fallback Granularity Rule: For the fallback tier (regex, AST parser, or lightweight crawler without a language build system), each source file MUST be its own independent semantic unit. Never bundle multiple source files into a single fallback unit. Per-file unit isolation is essential for content-addressed cache efficiency — when one file changes in a future snapshot, only that single file's unit invalidates while all unchanged files hit the cache (reused = N-1).

Baseline + Delta Overlay

• Static baseline: Built once in CI for the complete snapshot.
• Incremental cache: Immutable unit outputs reused across snapshots.
• Delta overlay: Changed/target files indexed locally, layered over baseline.
• Remote serving: Optional when global index is too costly locally.

At query time, the query helper merges baseline + overlay results. Overlay symbols supersede baseline symbols for the same file. The manifest records provider.kind = "baseline+overlay" with baseline manifest reference and overlay unit count.

Compatibility JSONL Export

The old structural_index.jsonl format is retained as a compatibility export only — it is NOT the query contract. Below the threshold (default 10K records), a complete JSONL export is emitted. Above the threshold, only a provenance pointer is written. Large consumers MUST use the query interface.

Line 1 — provenance header:

{"_provenance": true, "snapshot_id": "abc123", "tool": "tree-sitter"}

Lines 2+ — structural records (one per line, _type discriminator):

{"_type": "function", "key": "src/parser.c:parse_input", "start_line": 45, "end_line": 120, "signature": "int parse_input(char *buf, size_t len)", "calls": ["malloc", "validate_input", "memcpy"]}
{"_type": "call_edge", "caller": "parse_input", "callee": "malloc", "file": "src/parser.c", "line": 78}
{"_type": "call_edge", "caller": "main", "callee": "parse_input", "file": "src/main.c", "line": 203}

Provenance header fields:

Record types:

Snapshot Safety

1. Build from CODE_ROOT, not live tree (when pinned). When the snapshot is pinned, the structural index is built from the pinned CODE_ROOT, ensuring it reflects the exact bytes the pipeline is analyzing.
2. Reuse-on-match. If manifest.json already carries the current SNAPSHOT_ID (and SNAPSHOT_ID is NOT "unknown" — in MODE-OFF, always rebuild), reuse it — do not rebuild. Rebuild when SNAPSHOT_ID differs or is "unknown". Individual units may still be reused from the content-addressed cache. This bounds cost across retries/crash-resume.
3. Manifest atomicity. The manifest is written LAST via atomic rename from tmp/. Interrupted builds leave unreferenced temp objects without corrupting the last published index. On resume, check manifest.json status and snapshot_id.
4. STALE flag in HALT mode. When snapshot_pinned is false (HALT), the index may be built from the unpinned CODE_ROOT but is marked as potentially stale. Consumers treat structural hints as advisory.
5. MODE-OFF: build, do not skip. When active_snapshot is absent (MODE-OFF), build against the current directory (cwd) with provenance snapshot_id set to "unknown". This is the standalone-efficiency case — the index is still useful for the researcher/planner even without snapshot pinning. Do NOT return an empty index merely because the snapshot is absent.

Consumption Contract

These are runtime instructions for callers — they define how consumers use the structural index. The structural index is a HINT-only enhancement; skills that do not use it behave exactly as they do today.

Query Interface

The query helper IS the contract. All consumers use query_structural_index.py (or a compatible remote endpoint). The JSONL file is NOT the query contract — it is a compatibility export only.

Operations:

1. resolve_symbol(name, language?, file?, namespace?) → {results, total, ambiguous, coverage}

   • Returns ALL matches — no silent selection among ambiguous symbols.
   • Caller MUST disambiguate before calling find_callers / find_callees.

2. find_callers(symbol_id, limit=100, offset=0) → {results, total, has_more, coverage}

   • Bounded, paginated caller lookup.
   • Each result carries precision, backend, edge_kind.
   • Empty results carry coverage.partition_status.

3. find_callees(symbol_id, limit=100, offset=0) → {results, total, has_more, coverage}

   • Same shape as find_callers.

4. get_function_boundary(file, line) → {symbol_id, start_line, end_line, signature, precision, backend}

5. get_coverage(file?) → {total_files, indexed_files, failed, deferred, partition_status, backends_used}

Key properties:

• Bounded results and pagination (prevents loading entire index into memory).
• Explicit name resolution before graph traversal (no silent selection among ambiguous symbols).
• Precision and backend attached to every result.
• Coverage attached to empty results.
• Identical operations for local and remote providers (remote via MANTIS_STRUCTURAL_INDEX_URL env var or manifest provider.kind = "remote").

Coverage on empty results:

mantis-plan

• Use the structural index query helper for function-level dependency fan-out. When planning investigations, call resolve_symbol() then find_callers() to identify all functions that call into a target — this broadens the audit set beyond single-file analysis.
• The structural index decides ORDER of investigations (which functions to audit first based on call-graph centrality), never MEMBERSHIP. It may broaden the audit set (safe over-reporting), but must never REMOVE or drop a file; the planner's existing logic remains the membership floor.

mantis-researcher

• Wave 1 (Rapid Triage): Run a repo-wide grep for the function name to build the exhaustive set of candidate call-sites — this is the mandatory floor. Then use the structural index query helper (resolve_symbol then find_callers) to RANK and prioritize which call-sites to audit first (the index distinguishes actual calls from comments/strings/variable names). Audit the union of both result sets — the structural index may miss macro-based calls, function pointers, and dynamic dispatch, so grep remains the floor.
• Wave 2 (Deep Audit): Use get_function_boundary(file, line) to start with the enclosing function, expanding to callers/callees/file as needed for cross-function context — this saves context while preserving coverage.
• If the structural index is absent or empty, fall back to grep-based discovery (today's behavior). The structural index is a coverage HINT only — it improves audit quality but is never required.

Safety

Non-negotiable invariants:

1. Agnostic. Nothing is ever required; no-tool / parse-fail / not-invoked → empty index → grep fallback = today's behavior byte-for-byte. Optional in the Pass Lifecycle Contract; a non-conformant harness simply skips it.
2. HINT-only / union / never MEMBERSHIP. Consumers audit the union with grep; a symbol the index misses must still be reachable by the exhaustive sweep. The structural index decides ORDER, never MEMBERSHIP. The query contract enforces this: find_callers returns HINTs, never authoritative membership — consumers MUST union with grep.
3. No verdicts, touches no findings. Cannot violate INV-1 and cannot itself drop a finding.
4. Narrow scope: source cross-reference only. Binary/build-derived reachability ("is it compiled into production") is explicitly out of scope — a dev customization, not part of this skill, because absence-from-a-build can hide a real finding (INV-2). Do not fold build-derived reachability into this skill.

A reference blueprint is available at mantis-pipeline-adapter/references/mantis-structural-index.md. It is a stub that points to this SKILL.md as the single source of truth — do not duplicate spec content there.