Rbtr Languages

rbtr language plugins

How rbtr turns source into chunks, and the conventions for changing it. Pairs with the rbtr-testing skill (the sample/snapshot harness) and the rbtr-languages tags reference in references/tags-scm-reference.md.

Comprehensiveness principle

rbtr aims to index everything an engineer might search for or navigate to. If a developer could plausibly grep a name, or want "jump to definition" on it, it should be a chunk. Bias toward over-capture, not under-capture.

• "Don't capture X" must be justified by "X is not a definition / has no searchable identity" — never by "X is rare/niche". Rarity is not a reason to skip.
• Definition-shaped constructs are in scope even when they need bespoke work: C function prototypes (the API surface in headers), namespace/mod as their own symbols, Bash aliases (a written command definition, like source), SCSS @mixin/@function/$variables. Capture them.
• Written constructs only — the written code is the unit, not its runtime effects. rbtr extracts what is in the source, not what a macro/ metaprogramming call generates. Ruby attr_accessor :name is written and findable as content (it sits in the class chunk), so full-text search already locates it — but do not synthesise name/name= method chunks for the accessors it generates at runtime; those are not written code. (Capturing the written :name token as a field is a possible future nicety, not a default.)
• Comments are content, not decoration. Top-level comments — banners, licence headers, section notes — and module docstrings are captured as COMMENT chunks so their text is searchable, not dropped; a comment above a definition folds into it as documentation. See Comment handling.
• Legitimate skips: truly anonymous nodes with no name a human would search (a tree_sitter_query pattern with no outer label — captured only as an anonymous section), pure metadata (COMMENT ON), usage/references (which belong to edges, not symbol chunks), and runtime-generated symbols (per above).
• When unsure, capture it. A spurious chunk is cheaper than an invisible symbol.

Contained definitions & the data-scope exception

A definition that lives inside a larger definition we already chunk is represented by that parent chunk — it is full-text-findable there and does not get its own chunk. Statements inside a function, and data members of a class (class attributes, enum members, dataclass fields), are represented by the enclosing function/class chunk. Methods are the exception: substantial, separately-navigable units, so they do get their own chunk.

The data-scope exception (language philosophy). Variable capture targets the language's idiomatic top-level data scope, not nested members — and what counts as "top-level" differs by language:

• Languages with a module / file / package data scope (Python module, Go/Rust/C file or package, JS/TS module, Bash script) capture variables there; class/struct data members stay contained in their parent chunk.
• Class-only languages with no top-level data scope (Java): the class is the only namespace, so fields are the top-level data definitions and are captured (scoped to their class). Not capturing them would leave Java with zero variable chunks.

This is not an inconsistency between languages — it is the same rule (capture top-level data at the language's idiomatic scope) applied to differing language philosophies. It keeps capture comprehensive without exploding every class into per-field chunks, while never leaving a language's data definitions invisible.

Architecture

A plugin is a package (rbtr-lang-<lang>) exposing one or more module-level LanguageRegistration values — each named by its language id — through the rbtr.languages entry-point group; the LanguageManager discovers them via importlib.metadata (no pluggy — languages are single-dispatch by id). Core's bundled languages register the same way, from core's own pyproject. build_index routes extraction down one of three paths:

1. Chunker — a chunker attached via @reg.chunker: prose (markdown, rst) and SFCs' markup (svelte, vue). The chunker owns extraction. It takes an optional ranges and must set parser.included_ranges when given it, so it can also serve as an injection target for an embedded block (see below).
2. Query — reg.grammar_module + a QueryExtraction: code, plus config/data whose scope a query can express (python, rust, …, json, css, html, toml, yaml, hcl, tree_sitter_query). Goes through extract_symbols. HTML captures its semantic elements (head, body, sectioning content, landmarks) as doc sections, named by id else tag via a name_extractor. The tree_sitter_query plugin indexes .scm files themselves: each top-level pattern is a @doc_section, named by its own outer capture else anonymous.
3. Plaintext fallback — no grammar/detection: fixed-size raw chunks.

extract_symbols is the query engine: parse → run query → captures → Chunks. It takes the LanguageRegistration and delegates naming, scoping, and imports to it (reg.resolve_name / resolve_scope / resolve_import), each calling the language's resolver — the built-in, or an override composed over it.

Injection (embedded languages) is an orthogonal capability that runs in addition to the primary path. To extract code embedded in a host file (an SFC's <script>/<style>, a Markdown fenced block, an HTML inline <script>/<style>), set reg.injection_query: a tree-sitter query over the host grammar that captures each embedded block as @injection.content and names its target language one of two ways:

• Static — (#set! injection.language "<id>"), for a closed set (SFC/HTML <script>→js/ts, <style>→css), with an optional (#set! injection.priority "<n>") so a lang-tagged rule beats a bare one.
• Dynamic — capture the language name as @injection.language (a Markdown fence's info string). The engine resolves the captured text via the registry's own id and extension maps (python/py both reach python); an unknown hint is left unparsed. No per-language mapping table.

The engine delegates each block's range to the target's full primary extraction (extract_primary — chunker or query, so a chunker target like yaml/toml works, not just query targets) and recurses into the target's own injection (an HTML block containing an inline <script> yields its js), all at absolute line numbers. Every file also gets a host-language chunk (a content-less presence chunk if it would produce none), so dedup works. See ARCHITECTURE “Dispatch chain” for the mechanism and rationale.

Where queries live

Every query — reg.extraction.query, reg.injection_query, and any query a chunker compiles — is a .scm file co-located in the plugin package (rbtr_lang_<lang>/<name>.scm), loaded at import via load_query (import it: from rbtr.languages.registration import load_query; call it: load_query(__package__, "<name>")). Call it directly in the extraction or injection_query field — never hoist it into a module-level _QUERY constant; load_query is cached on (package, name), so a query shared by two registrations (svelte and vue's SFC injection query) is read once. Never inline a query as a Python string literal (the house rule against embedding a foreign language). The uv build backend ships .scm as package data with no extra config.

Compose in Python when a query is built from parts — the query language has no #include: js/ts concatenate shared fragment files with + (load_query(pkg, "javascript") + load_query(pkg, "shared") + …); SQL groups its DDL verbs into [...] alternations within one sql.scm. Prefer these to generating query text from Python data. Editing a .scm is an extraction change — the extraction_serial bump rule (below) applies.

Capture conventions

The query's capture names drive the chunk kind (see _CAPTURE_KINDS in languages/treesitter.py):

• @function / @_fn_name — functions
• @class / @_cls_name — classes, structs, enums, traits, types, and named collections of declarations (CSS rule sets, @media, @keyframes)
• @method / @_method_name — methods (a @function whose nearest scope is class-like is also promoted to a method)
• @variable / @_var_name — module/top-level variables, constants, fields
• @import — import statements (metadata via import_extractor)
• @_scope — optional: a node whose text becomes the symbol's innermost scope segment, for scopes lexical nesting can't reach (e.g. a Go method's receiver type). Strictly additive: absent → no effect.
• @_docstring — interior first-statement docstring (Python)
• @doc_section / @_section_name — chunker/data section units
• @config_key / @_section_name — config/data keys (JSON object keys, TOML tables, YAML mapping keys, HCL blocks, CSS @charset), reusing the @_section_name name capture
• @comment — top-level comments and module docstrings; grouped into blocks and either folded into the definition below or emitted as standalone COMMENT chunks (see Comment handling)

Capture names starting with _ are read but never become chunks.

The display name comes from the paired @_*_name capture via the built-in name resolver. When a query cannot express the name, attach a name_extractor — @reg.name_extractor for a single-use local override, or reg.name_extractor(fn) for a shared/imported one — as a last resort. It is wrap-style (pydantic WrapValidator shape): signature (resolver, capture_name, node, captures) -> str, where resolver is the built-in resolver handed in; call it to delegate the cases you don't special-case, exactly as an import_extractor receives the built-in import resolver. Bash strips the = the grammar fuses onto an alias; HTML names an element by its id, else its tag.

The scope address comes from tree ancestry (scope_types, below) plus the scope_extractor — the scope twin of name_extractor, whose built-in resolver contributes the @_scope capture. Attach a custom one the same way (wrap-style signature (resolver, capture_name, node, captures) -> list[str], outermost-first) as a last resort, for a hierarchy neither ancestry nor @_scope can reach; its segments are appended to the ancestry scope. Two real cases:

• CSS/SCSS/Less nested rules — walk the ancestor rule_set selectors so .card { .title { … } } scopes .title under .card:

  def css_nesting_scope(_resolver, capture_name, node, captures):
      segments: list[str] = []
      for rule_set in enclosing_nodes_of_type(node, frozenset({"rule_set"})):
          for child in rule_set.children:
              if child.type == "selectors" and child.text:
                  segments.append(child.text.decode().strip())
                  break
      return segments
  

• TOML dotted tables — the hierarchy is a dotted-key string, not tree ancestry, so a name_extractor returns the last segment and a scope_extractor the preceding ones ([tool.ruff] → name ruff, scope tool).

Comment handling

Comments are captured, not configured. Give your grammar's comment node(s) a root-scoped @comment capture so only top-level comments match — a comment inside a function body stays part of that body's chunk:

• most grammars: (translation_unit (comment) @comment), (program (comment) @comment), (source_file (comment) @comment), (stylesheet (comment) @comment) — use the grammar's actual root and comment node type;
• multiple comment node types go in an alternation: Rust/Java [(line_comment) (block_comment)], SQL [(comment) (marginalia)], SCSS/Less [(comment) (js_comment)];
• Python also captures its module docstring: (module (expression_statement (string) @comment)).

The engine does the rest, identically for every language: it groups top-level comment runs into blank-line-delimited blocks, folds a block flush above a definition into that definition, leaves interior comments in their body, and emits everything else as a standalone COMMENT chunk. You only declare the node types. A comment trailing code on its line documents that statement and never folds forward. The routing rules and their rationale live in ARCHITECTURE.

Scope & promotion (engine layers — already generic)

Set on the language's QueryExtraction (the extraction field); extract_symbols applies them to every captured node:

• scope_types — node types that open a naming scope; composed into the :: address. Include nesting containers (classes, namespaces, modules, functions where nested defs matter).
• class_scope_types — the subset that is class-like; a function directly inside one is promoted to a method. Defaults to scope_types.
• Non-lexical scope comes from @_scope (above) or, when even that can't reach it, a scope_extractor (above), not these.

Authoring or extending a plugin

1. Read the grammar: its queries/tags.scm (the authors' definition list — see the tags reference) and the real node structure (parse a snippet, print the tree). Never guess node types.
2. Edit the language's .scm query file (or chunker) — queries live in the plugin package (rbtr_lang_<lang>/*.scm), loaded via load_query, never inline (see Where queries live). Verify against a parsed snippet.
3. Add the construct to that language's sample in the package's tests/samples/ and regenerate the snapshot with just snapshots; review the diff.
4. Bump extraction_serial — any extraction change triggers re-extraction of stored blobs. It is independent of the package version: bump it during development (before any release) when output changes, and never bump it for a package-only release or pure move.
5. just check. (Samples are exempt from lint/type/format — see below.)

Gotchas (all learned the hard way; verified)

• Require a body on type captures. (struct_specifier name: …) matches references too (e.g. inside typedef struct G G;, or a parameter type), producing spurious class chunks. Require body: so only definitions match (C/C++ struct/enum/class).
• Take a span's last line from last_line. Tree-sitter rows are 0-based, so end_point[0] + 1 is right for a node ending mid-line and one too many for a node that consumes its trailing newline and ends at column 0 of the next row. Ten sites once computed this three ways: a toml table claimed the next table's header line, a rust comment block lost its last line, and a heading-less rst paragraph ran backwards (line 7 to line 6). Import last_line from rbtr.languages.chunks and delete any local trailing-newline correction — a compensation left beside it decrements twice.
• Non-lexical scope → @_scope. Scope is otherwise lexical-ancestry only (_enclosing_scopes walks parents). A Go method's receiver is a child, so capture it as @_scope.
• Determinism in test helpers. next(iter(reg.extensions)) over a frozenset varies with PYTHONHASHSEED; the chunk id hashes file_path, so derive paths deterministically (sorted(...), or pass an explicit path).
• Imports are always bespoke. tags.scm has no imports; the edge system depends on them. Keep per-language import_extractors.
• SQL / multi-dialect: don't gate on parse-clean. One generic SQL grammar serves all .sql. Tree-sitter error recovery is local — a dialect construct it can't parse breaks only its own subtree; surrounding statements still extract. Treat has_error as informational, not a gate.
• Known-unsupported constructs → strict xfail. Record a construct that should extract but can't (grammar/plugin limit) as an xfail(strict=True) case, so closing the gap flips the test and prompts an update. Reserve for symbol-shaped gaps (kind, name, scope); import-identity gaps don't fit.

tags.scm: a reference, not a runtime source

Every code grammar ships queries/tags.scm — the authors' standard definition/reference query (@definition.* / @name / @reference.*). It is itself a tree-sitter query, so it is inspiration for ours, not a drop-in:

• Mine it: take good patterns, modify weak ones, ignore wrong ones. Verify every pattern against a real parse — quality is uneven (e.g. C's union pattern misses standalone union U {}).
• It captures no imports and few variables, omits some constructs (TS enum/type alias, Java enum/record), some grammars inherit others (ts ← js), and only the 9 code grammars ship one.
• Running it live would couple extraction to upstream drift. We don't.

Distinct from rbtr's own .scm files (see Where queries live): we load those at runtime (reg.extraction.query / reg.injection_query) as the source of truth, whereas tags.scm we only mine for ideas. And the tree_sitter_query plugin now indexes .scm files found in a repo — including third-party tags.scm / highlights.scm / injections.scm — as content, orthogonal to whether we run them.

Curated per-language verdicts (take / modify / ignore) and the @definition.* → ChunkKind mapping live in references/tags-scm-reference.md.

Testing

Each language has a sample mini-project under its package's tests/samples/ (one or more files), golden-snapshotted (full model_dump_json), coverage-checked, and parse-clean-checked. See rbtr-testing. Samples are exempt from the repo's linters/type-checker (they're fixtures) — validated only by their own tests.