YAML Semi Indexing

YAML Semi-Indexing Skill

Implementation details for YAML semi-indexing using succinct data structures.

Comprehensive documentation: See docs/parsing/yaml.md for full parsing architecture.

Semi-Index Structure

YAML uses more components than JSON due to its richer structure:

• Interest Bits (IB): Marks structural positions (keys, values, items)
• Balanced Parentheses (BP): Encodes tree structure for navigation
• Type Bits (TY): Distinguishes mappings (0) from sequences (1)
• Sequence Item Bits (seq_items): Marks BP positions that are sequence item wrappers

Key Insight: Sequence Items vs Containers

Sequence items have BP open/close pairs but NO TY entry. This affects TY index calculations:

// WRONG: Direct rank gives incorrect TY index when seq_items exist
let ty_idx = bp.rank1(bp_pos);

// CORRECT: Subtract sequence items to get real container index
let bp_opens_before = bp.rank1(bp_pos);
let seq_items_before = count_seq_items_before(bp_pos);
let ty_idx = bp_opens_before.saturating_sub(seq_items_before);

Block Sequence Parsing

Multi-line Sequence Items

For YAML like:

-
  name: Mark
  hr: 65

The sequence item must remain open so content on subsequent lines becomes the item's value:

1. Parse - at indent 0 → open sequence, open item, push (indent+1, SequenceItem) to stacks
2. at_line_end() is true → return (item stays open)
3. Next line name: Mark at indent 2 → parse_mapping_entry(2)
4. Mapping opened inside the still-open item
5. Subsequent lines at same indent add to mapping
6. When indent returns to 0, close_deeper_indents closes mapping, item, etc.

Inline Compact Mappings

For - name: Mark\n hr: 65:

1. Parse - → open sequence, open item
2. looks_like_mapping_entry() is true → call parse_compact_mapping_entry(indent+2)
3. Mapping opened but NOT closed after first entry
4. Item also NOT closed
5. Next line hr: 65 at indent 2 adds to the same mapping

Critical: Don't close compact mappings eagerly. Let close_deeper_indents handle it.

Nested Sequences

For - - item:

Check for nested sequence BEFORE checking for mapping:

if self.peek() == Some(b'-') && matches!(self.peek_at(1), Some(b' ') | ...) {
    // Nested sequence - recurse with indent+2
    self.parse_sequence_item(indent + 2)?;
} else if self.looks_like_mapping_entry() {
    // Compact mapping
}

Common Debugging Patterns

Tracing BP Structure

for bp_pos in 0..30 {
    let is_open = index.bp().is_open(bp_pos);
    if is_open {
        if let Some(text_pos) = index.bp_to_text_pos(bp_pos) {
            let is_seq_item = index.is_seq_item(bp_pos);
            println!("BP[{}] = OPEN at text[{}] seq_item={}", bp_pos, text_pos, is_seq_item);
        }
    } else {
        println!("BP[{}] = CLOSE", bp_pos);
    }
}

Checking Stack State

When debugging incorrect structure, trace indent_stack and type_stack:

• After each parse_* call, verify stacks have expected entries
• close_deeper_indents should be called BEFORE checking need_new_sequence/mapping

Order of Operations Bug

Wrong:

let need_new = type_stack.last() != expected;  // Check first
if need_new {
    close_deeper_indents(indent);  // Then close
    // But need_new was computed with OLD stack state!
}

Correct:

close_deeper_indents(indent);  // Close first
let need_new = type_stack.last() != expected;  // Then check

Test Suite Notes

The YAML test suite (tests/yaml_test_suite.rs) is generated from the official YAML test suite.

Important: Tests must compare JSON output, not just parse success. A test that only checks result.is_ok() doesn't verify correctness.

When regenerating tests, ensure the generator properly escapes JSON for Rust string literals:

json_escaped = json_normalized.replace('', '').replace('"', '\\"')

See Also

• docs/parsing/yaml.md - Full YAML parsing documentation
• src/yaml/parser.rs - Parser implementation
• src/yaml/light.rs - Cursor and value extraction
• src/yaml/index.rs - Index structure and TY calculations

Converted and distributed by TomeVault — claim your Tome and manage your conversions.