castorini Ragnarok Quickstart
Reference for the ragnarok CLI — the tool for RAG answer generation with citation support, TREC evaluation, and multi-stage retrieval pipelines.
CLI Entry Point
ragnarok <command> [options]
Primary Commands
| Command | Purpose |
|---|---|
generate |
Generate cited answers from query + candidate passages |
validate |
Validate request payloads or TREC RAG output files |
convert |
Convert output to TREC submission formats |
Introspection Commands
| Command | Purpose |
|---|---|
doctor |
Check Python version, API keys, backend readiness |
describe <cmd> |
Machine-readable command contract |
schema <name> |
Print JSON Schema for inputs/outputs |
prompt list|show|render |
Inspect and render prompt templates |
view <path> |
Inspect existing artifact files |
Two Generation Modes
File-backed (batch JSONL)
ragnarok generate --input-file requests.jsonl --output-file answers.jsonl \
--model gpt-4o --prompt-mode ragnarok_v4
Dataset-backed (retrieval + generation)
ragnarok generate --dataset rag24.raggy-dev \
--retrieval-method bm25,rank_zephyr_rho --topk 100,20 \
--model gpt-4o --prompt-mode ragnarok_v4 --output-file answers.jsonl
Reference Files
Read these on demand for details:
references/cli-examples.md— Common invocations for each commandreferences/input-output-examples.md— JSONL format examplesreferences/prompt-modes.md— Prompt mode descriptions and selection guidereferences/dataset-workflow.md— Dataset-backed generation details
Key Concepts
- Prompt modes: Template families controlling answer format and citation style (e.g.,
ragnarok_v4,chatqa) - Cited answers: Each sentence has a
citationsarray referencing candidate indices - topk: Comma-separated pipeline depths (e.g.,
100,20= retrieve 100, rerank to 20) - Backends: OpenAI/compatible (default), Cohere (
command-r*models), open-source (llama,mistral,qwen)
Gotchas
--modeland--prompt-modeare both required forgenerate.--datasetmode does not support--execution-mode asyncyet.--topkis comma-separated: first value is retrieval depth, last value is generation depth. For file-backed mode, only the last value matters.- Model name detection is automatic:
command-r*→ Cohere backend,llama/mistral/qwen→ open-source backend, everything else → OpenAI-compatible. --retrieval-methodis comma-separated and must align with--topkfor multi-stage pipelines.- TREC RAG 2025 validation (
validate rag25-output) has--fix-length,--fix-citations, and--apply-fixesflags for repairing submissions. - Config file (
.ragnarok.tomlor~/.config/ragnarok/config.toml) can set defaults; CLI flags override.