Nuggetizer Quickstart

Nuggetizer Quickstart

Reference for the nuggetizer CLI — the tool for creating, scoring, and assigning factual nuggets for RAG evaluation using LLM backends.

CLI Entry Point

nuggetizer <command> [options]

Primary Commands

Introspection Commands

Quick Workflow

# 1. Check environment
nuggetizer doctor

# 2. Create nuggets from a pool file
nuggetizer create --input-file pool.jsonl --output-file nuggets.jsonl --model gpt-4o

# 3. Assign nuggets to answers
nuggetizer assign --contexts answers.jsonl --nuggets nuggets.jsonl \
  --input-kind answers --output-file assignments.jsonl --model gpt-4o

# 4. Calculate metrics
nuggetizer metrics --input-file assignments.jsonl --output-file metrics.jsonl

Reference Files

Read these on demand for details:

• references/cli-examples.md — Common invocations for each command
• references/input-output-examples.md — JSONL format examples (create input, create output, assign input/output, metrics output)
• references/prompt-templates.md — How the YAML prompt template system works

Key Concepts

• Nuggets: Atomic facts extracted from candidate passages, each scored as vital or okay
• Assignment modes: 3-grade (support, partial_support, not_support) or 2-grade (support, not_support)
• Backends: OpenAI (default), Azure OpenAI (--use-azure-openai), OpenRouter (--use-openrouter), vLLM (local)
• Write policies: --resume (append, skip processed), --overwrite (truncate), --fail-if-exists

Gotchas

• create has separate --creator-model and --scorer-model overrides. If only --model is set, both stages use it.
• assign --input-kind answers expects --contexts to be a JSONL with topic_id and answer fields. assign --input-kind retrieval expects the same format as create input.
• --nuggets is required for batch assign but not for direct input (nuggets are in the JSON payload).
• metrics operates on assign output only — it reads assignment labels, not raw nuggets.
• --execution-mode async is available but the default is sync.
• Legacy scripts in scripts/ translate to CLI calls with --resume by default.