tomevault-io Tree-Sitter Parse Testing
Overview
Use tree-sitter parse -d to debug parsing issues. The debug output shows the state machine transitions, helping identify where parsing fails.
Parsing a Code Snippet
Run from the parser directory (fsharp/ or fsharp_signature/):
echo 'let x = 1' | npx tree-sitter parse -d
Interpreting Output
Successful Parse
(file [0, 0] - [1, 0]
(declaration_expression [0, 0] - [0, 9]
...))
- Node format:
(nodename [start_row, start_col] - [end_row, end_col]) - Children indented inside parent nodes
Parse Errors
(ERROR [0, 0] - [0, 7]
...)
- ERROR nodes indicate parsing failure at that location
Missing Tokens
(MISSING identifier [0, 8] - [0, 8])
- MISSING indicates expected token that wasn't found
Debug Output Analysis
Key patterns to look for:
| Pattern | Meaning |
|---|---|
detect_error |
Parser encountered unexpected token |
recover_with_missing |
Parser inserted placeholder to recover |
recover_with_error |
Parser skipped tokens to recover |
shift state:N |
Parser shifted to state N |
reduce sym:X |
Parser reduced using rule X |
accept |
Parse completed successfully |
Debugging Workflow
- Run parse without
-dto see the tree - If tree looks wrong (ERROR nodes, unexpected structure):
- Run with
-dto see the debug output - Look for
detect_errornear the problem location - Check which state the parser was in when error occurred
- Compare expected tokens at that state against actual input
Infinite Loops
If the debug output repeatedly emits the same token or state indefinitely, the parser has entered an infinite loop. This typically happens when:
- A grammar rule injects tokens without consuming any input
- The scanner (
src/scanner.c) is returning tokens that don't advance the parse position
** Symptoms:**
- Debug output shows the same token being processed repeatedly
- The parser never reaches
acceptor an error state - Command appears to hang
What to do:
- Interrupt the command (Ctrl+C)
- Review the grammar rules and scanner logic
- Look for rules that could cause the parser to re-enter the same state without consuming input
Example Debug Session
# 1. See the parse tree
echo 'type X = ' | npx tree-sitter parse
# 2. Debug output shows recovery
echo 'type X = ' | npx tree-sitter parse -d | grep -A5 detect_error
The debug output shows the parser detecting an error at column 8 and recovering by inserting a MISSING identifier.
Common Issues
- Unexpected ERROR node: Run with
-d, finddetect_error, check what tokens were expected - MISSING token: Parser expected something that wasn't there
- Extra nodes: Check if grammar rule is too greedy (look at reduce steps)
Source: ionide/tree-sitter-fsharp — distributed by TomeVault.