Index Management

Index Management

Intro

index-management is the read-side foundation. It walks the project's context/ directory, parses every entity file, and writes a SQLite database that other tools query. Its peer at Layer 0 is id-management (the write side, which allocates new IDs). Both are foundational to every entity-creating skill.

MCP server. This skill ships a self-contained MCP server at mcp/server.py (PEP 723 script — requires uv and Python ≥ 3.10 on PATH). Agent harnesses may reach it directly from this skill's mcp/mcp-config.json or through the processkit gateway. Gateway deployments must surface only the processkit servers present in the installed/merged MCP configuration; this skill does not imply that unrelated processkit tools are available.

Overview

What it provides

• reindex() — rebuild the SQLite index from scratch
• query_entities(kind?, state?, limit?) — list entities matching filters
• get_entity(id) — fetch a single entity by ID
• search_entities(text, limit?) — FTS5 search across titles, bodies, specs
• semantic_status() — report semantic chunk/vector availability
• semantic_search_entities(text, limit?) — sqlite-vec semantic search when available
• hybrid_search_entities(text, limit?) — RRF over FTS5 + semantic results, with FTS-only fallback
• query_events(event_type?, subject?, actor?, limit?) — query the event log
• list_errors() — files that failed to parse during the last reindex
• stats() — counts of entities/events/errors in the index

When to use

Agents call the MCP server whenever they need fast lookup. Other MCP servers (workitem-management, decision-record, etc.) refresh the index after writing a new entity to keep reads current. In gateway deployments, the gateway only routes to index-management if this server is present in the installed MCP configuration.

Where the database lives

<project-root>/context/.cache/processkit/index.sqlite. Gitignored. Rebuildable from source files at any time via reindex().

Gotchas

Agent-specific failure modes — provider-neutral pause-and-self-check items:

• Grepping the filesystem instead of calling query_entities. The index is faster, context-cheaper, and reflects parsed semantics (state, kind, links) that grep can't see. Reach for query_entities first; fall back to grep only if the query you need isn't supported.
• Trusting the index immediately after a hand-edit. When you edit an entity file directly (bypassing the MCP write tools), the index is stale until the next reindex() call. Either go through the write tools, or call reindex() yourself before querying.
• Forgetting to call reindex after writing a new entity. Every entity-creating MCP server should call index_management.reindex (or the lighter upsert_entity) after a write. Not doing so means the next query won't see the just-created entity.
• Treating index queries as transactional. The index is eventually consistent. A query immediately after a write may or may not return the new row depending on whether reindex completed. If you need strict consistency, reindex explicitly first.
• Trusting list_errors() as proof of correctness. A clean errors table means parsing succeeded, not that the entities are semantically valid. Schema validation is a separate concern from parse success.
• Using search_entities for exact-ID lookup. search_entities is ranked FTS5 search over parsed content, with a fallback substring search for invalid FTS syntax. Use get_entity(id) for exact lookup.
• Assuming semantic search is always active. The semantic chunk table is always rebuilt, but vector KNN requires the optional sqlite-vec extension to load. Check semantic_status(); use hybrid_search_entities() when you want an FTS-backed fallback.
• Re-indexing inside a hot loop. A full reindex walks the whole context/ tree and is expensive. Batch your writes and reindex once at the end, not after every entity.

Full reference

Database schema

Three tables — see src/lib/processkit/index.py for the DDL.

entities(id, kind, api_version, path, storage_location, created, updated, title, state, labels_json, spec_json, body)
entities_fts(id, kind, state, title, body, spec_json)
semantic_chunks(rowid, chunk_id, entity_id, kind, state, path, ordinal, text)
entity_vec(embedding)
events(id, timestamp, event_type, actor, subject, subject_kind, summary, details_json, correlation_id, path)
errors(path, message)

spec_json holds the full spec block as JSON for queries we did not anticipate. storage_location is NULL for hot-tree files and points at cold archive payloads for archived entities. entities and events overlap for LogEntry rows: a LogEntry appears in both, with the events table denormalizing the event-specific fields for fast filtering.

entities_fts is a rebuildable SQLite FTS5 virtual table. It mirrors the searchable fields from entities and is cleared/rebuilt during reindex(). If a stripped-down SQLite build lacks FTS5, the index still opens and search_entities falls back to the previous LIKE %text% behaviour.

semantic_chunks is a rebuildable chunk table. When the optional sqlite-vec package can be imported and loaded, entity_vec is created as a vec0 virtual table containing deterministic local embeddings for those chunks. The initial embedding strategy is provider-neutral local-hashed-bag-of-words; it gives a stable offline baseline and a storage contract for later provider/local embedding model upgrades.

Reindex strategy

reindex() is destructive and atomic — it deletes all rows and re-inserts. For typical projects this is fast (sub-second). For very large projects, an incremental update mode lands later (Phase 4+).

Errors table

Files that fail to parse get a row in the errors table instead of crashing the reindex. list_errors() returns all such rows so the agent can fix them. The errors table is cleared at the start of each reindex.

Tools that other servers call

workitem-management, decision-record, binding-management, and event-log all call the shared index library after writing a new file. This keeps the index in sync without a full reindex on every mutation. From an MCP-protocol perspective each configured server has its own process, or is reached through the gateway, and they communicate by sharing the same SQLite database file.

Configuration

The index database path is configurable via:

• processkit.toml [index] path = "..." (relative to project root)
• The PROCESSKIT_INDEX_DB environment variable
• Default: context/.cache/processkit/index.sqlite (gitignored cache)

Limitations at v0.3.0

• WAL mode enabled (v0.4.0+). Multiple readers and a single writer are safe; concurrent writes still serialize via WAL. Typical AI-assisted sessions are single-writer anyway.
• FTS5 search. Search uses SQLite FTS5 ranking when available and falls back to LIKE %text% for invalid FTS syntax or unsupported SQLite builds.
• Optional sqlite-vec. Semantic search uses sqlite-vec only when the extension is installed and loadable; otherwise semantic search returns no vector results and hybrid search falls back to FTS5.
• No incremental indexing. Every reindex is a full sweep.