Context Extraction

Context Extraction

OpenViking uses a three-layer async architecture for document parsing and context extraction.

Overview

Input File → Parser → TreeBuilder → SemanticQueue → Vector Index
              ↓           ↓              ↓
          Parse &     Move Files     L0/L1 Generation
          Convert     Queue Semantic  (LLM Async)
          (No LLM)

Design Principle: Parsing and semantics are separated. Parser doesn't call LLM; semantic generation is async.

Parser

Parser handles document format conversion and structuring, creating file structure in temp directory.

Supported Formats

Core Flow (Document Example)

# 1. Parse file
parse_result = registry.parse("/path/to/doc.md")

# 2. Returns temp directory URI
parse_result.temp_dir_path  # viking://temp/abc123

Smart Splitting

If document_tokens <= 1024:
    → Save as single file
Else:
    → Split by headers
    → Section < 512 tokens → Merge
    → Section > 1024 tokens → Create subdirectory

Return Result

ParseResult(
    temp_dir_path: str,    # Temp directory URI
    source_format: str,    # pdf/markdown/html
    parser_name: str,      # Parser name
    parse_time: float,     # Duration (seconds)
    meta: Dict,            # Metadata
)

TreeBuilder

TreeBuilder moves temp directory to AGFS and queues semantic processing.

Core Flow

building_tree = tree_builder.finalize_from_temp(
    temp_dir_path="viking://temp/abc123",
    scope="resources",  # resources/user/agent
)

5-Phase Processing

1. Find document root: Ensure exactly 1 subdirectory in temp
2. Determine target URI: Map base URI by scope
3. Recursively move directory tree: Copy all files to AGFS
4. Clean up temp directory: Delete temp files
5. Queue semantic generation: Submit SemanticMsg to queue

URI Mapping

SemanticQueue

SemanticQueue handles async L0/L1 generation and vectorization.

Message Structure

SemanticMsg(
    id: str,           # UUID
    uri: str,          # Directory URI
    context_type: str, # resource/memory/skill
    status: str,       # pending/processing/completed
)

Processing Flow (Bottom-up)

Leaf directories → Parent directories → Root

Single Directory Processing Steps

1. Concurrent file summary generation: Limited to 10 concurrent
2. Collect child directory abstracts: Read generated .abstract.md
3. Generate .overview.md: LLM generates L1 overview
4. Extract .abstract.md: Extract L0 from overview
5. Write files: Save to AGFS
6. Vectorize: Create Context and queue to EmbeddingQueue

Configuration Parameters

Code Skeleton Extraction (AST Mode)

For code files, OpenViking supports AST-based skeleton extraction via tree-sitter as a lightweight alternative to LLM summarization, significantly reducing processing cost.

Modes

Controlled by code_summary_mode in ov.conf (see Configuration):

What AST Extracts

The skeleton includes:

• Module-level docstring (first line)
• Import statement list
• Class names, base classes, and method signatures (ast mode: first-line docstrings only; ast_llm mode: full docstrings)
• Top-level function signatures

Supported Languages

The following languages have dedicated extractors built on tree-sitter:

Other languages automatically fall back to LLM.

Fallback Behavior

The following conditions trigger automatic fallback to LLM, with the reason logged. The overall pipeline is unaffected:

• Language not in the supported list
• File has fewer than 100 lines
• AST parse error
• Extraction produces an empty skeleton

File Structure

openviking/parse/parsers/code/ast/
├── extractor.py      # Language detection and dispatch
├── skeleton.py       # CodeSkeleton / FunctionSig / ClassSkeleton data structures
└── languages/        # Per-language extractors

Three Context Types Extraction

Flow Comparison

Resource Extraction

# Add resource
await client.add_resource(
    "/path/to/doc.pdf",
    reason="API documentation"
)

# Flow: Parser → TreeBuilder(scope=resources) → SemanticQueue

Skill Extraction

# Add skill
await client.add_skill({
    "name": "search-web",
    "content": "# search-web\\n..."
})

# Flow: Direct write to viking://agent/skills/{name}/ → SemanticQueue

Memory Extraction

# Memory auto-extracted from session
await session.commit()

# Flow: MemoryExtractor → TreeBuilder(scope=user) → SemanticQueue

Related Documents

• Architecture Overview - System architecture
• Context Layers - L0/L1/L2 model
• Storage Architecture - AGFS and vector index
• Session Management - Memory extraction details