Symbol Declaration Index

Symbol Declaration Indexing Protocol (Ctags & AST Tagging)

Overview

When an agent is tasked with modifying a function (e.g., calculateMonthlyUsage), default agents execute multiple broad grep_search calls across dozens of files, reading 500 lines of irrelevant call sites before finally discovering where the function is defined.

Grepping for symbols without an index causes:

1. Call-Site Token Flooding: Searching for user or get_session returns 200 call-site matches across the codebase.
2. Multiple Exploratory Turns: Takes 3 to 5 roundtrips just to identify the target source file and line range.
3. High Latency: Repetitive file reads stall agent momentum.

The Symbol Declaration Indexing Protocol builds a lightweight, 1-line-per-symbol flat index (symbols.tsv) mapping every class, interface, and function directly to its definition file:line and type signature.

Unindexed Call-Site Search vs. 1-Line Symbol Index Lookup

┌─────────────────────────────────────────────────────────────┐
│                 Symbol Localization Dynamics                │
│                                                             │
│  Unindexed Grep Search (3 Turns / 4,200 Tokens):            │
│  • Search: `validateJwt` ──► Returns 45 call-site matches   │
│  • Agent inspects `routes.ts`, `middleware.ts`, `tests/`    │
│  • Turn 3: Finally finds definition in `src/auth/jwt.ts:42` │
│  ↳ 3 Turns, 4,200 tokens billed, 12 seconds elapsed         │
│                                                             │
│  1-Line Symbol Index Lookup (1 Turn / 32 Tokens - 99.2% Cut):│
│  • Query: `rg "^validateJwt\t" symbols.tsv`                 │
│  ↳ `validateJwt  fn  src/auth/jwt.ts:42  (token: str): Decoded`│
│  ↳ 1 Turn, 32 tokens billed, direct line jump in 1ms!       │
└─────────────────────────────────────────────────────────────┘

The Standardized Symbol Index Format (symbols.tsv)

Every entry in symbols.tsv is a clean, tab-delimited 4-column record:

<symbol_name>\t<kind>\t<file_path>:<line_number>\t<signature>

Example TSV Index:

AuthMiddleware	class	src/middleware/auth.ts:14	class AuthMiddleware implements IMiddleware
validateJwt	fn	src/auth/jwt.ts:42	(token: string): DecodedSession
revokeSession	fn	src/auth/session.ts:88	async (sessionId: string): Promise<void>
UserRole	enum	src/types/user.ts:5	enum UserRole { ADMIN, USER }

Production Python Symbol Index Generator

Generate a repository-wide symbol index using regex/AST in 20 milliseconds:

import re
from pathlib import Path

SYMBOL_PATTERNS = [
    # TypeScript / JavaScript: Functions, Classes, Interfaces
    (re.compile(r'^(?:export\s+)?(?:async\s+)?function\s+([a-zA-Z0-9_]+)\s*(\([^\)]*\))'), 'fn'),
    (re.compile(r'^(?:export\s+)?class\s+([a-zA-Z0-9_]+)'), 'class'),
    (re.compile(r'^(?:export\s+)?interface\s+([a-zA-Z0-9_]+)'), 'interface'),
    (re.compile(r'^(?:export\s+)?const\s+([a-zA-Z0-9_]+)\s*=\s*(?:async\s*)?\([^\)]*\)\s*=>'), 'fn'),
    # Python: def and class
    (re.compile(r'^\s*def\s+([a-zA-Z0-9_]+)\s*(\([^\)]*\))'), 'fn'),
    (re.compile(r'^\s*class\s+([a-zA-Z0-9_]+)'), 'class')
]

def generate_symbol_index(root_dir: Path, output_file: Path = Path("scratch/symbols.tsv")) -> Path:
    """Scans repository and compiles 1-line-per-symbol TSV index."""
    output_file.parent.mkdir(parents=True, exist_ok=True)
    records = []

    for file_path in root_dir.rglob("*"):
        if file_path.suffix in [".ts", ".tsx", ".js", ".py"] and not any(p in file_path.parts for p in ["node_modules", "venv", "dist"]):
            rel_path = file_path.relative_to(root_dir)
            try:
                for line_no, line in enumerate(file_path.read_text(encoding="utf-8").splitlines(), start=1):
                    for pattern, kind in SYMBOL_PATTERNS:
                        match = pattern.search(line)
                        if match:
                            sym_name = match.group(1)
                            sig = match.group(2) if len(match.groups()) > 1 else ""
                            records.append(f"{sym_name}\t{kind}\t{rel_path}:{line_no}\t{sig.strip()}")
            except Exception:
                continue

    output_file.write_text("\n".join(records), encoding="utf-8")
    return output_file

Benchmark Comparison

Locating 20 target function definitions in a 3,000-file repository:

Agent Operational Directive

MANDATORY: For large multi-file codebases, generate a 1-line-per-symbol index in scratch/symbols.tsv. Query the TSV index to jump directly to exact definition file and line numbers rather than running repetitive exploratory grep searches.