rank_llm Quickstart
Reference for the packaged rank-llm CLI.
CLI Entry Point
rank-llm <command> [options]
Primary Commands
| Command |
Purpose |
rerank |
Run reranking from dataset retrieval, request files, or direct JSON input |
evaluate |
Aggregate trec_eval metrics across stored rerank outputs |
analyze |
Analyze stored responses and error counts |
retrieve-cache |
Build cached retrieval JSON from an existing TREC run |
serve http |
Start the HTTP server |
serve mcp |
Start the MCP server |
Introspection Commands
| Command |
Purpose |
doctor |
Check Python version and optional dependency readiness |
describe <cmd> |
Return structured command metadata |
schema <name> |
Print JSON Schema for supported inputs and outputs |
validate rerank |
Validate rerank inputs without executing a model |
prompt list|show|render |
Inspect bundled prompt templates |
view <path> |
Inspect rerank JSONL, request JSONL, TREC runs, and invocation histories |
Quick Workflow
# 1. Check environment
rank-llm doctor
# 2. Run retrieval + reranking
rank-llm rerank --model-path castorini/rank_zephyr_7b_v1_full --dataset dl20 \
--retrieval-method bm25 --top-k-candidates 100 \
--output-jsonl-file rerank_results.jsonl --output-trec-file rerank_results.trec
# 3. Inspect the artifact
rank-llm view rerank_results.jsonl
# 4. Aggregate trec_eval metrics from stored outputs
rank-llm evaluate --model-name castorini/rank_zephyr_7b_v1_full
# 5. Analyze invocation histories or stored response files
rank-llm analyze --files demo_outputs/inference_invocations_history.json --verbose
Reference Files
Read these on demand for details:
references/cli-examples.md - Common invocations for each commandreferences/input-output-examples.md - JSONL, TREC, and invocation-history artifact shapesreferences/workflows.md - Backend and workflow selection guide
Key Concepts
- Input modes:
rerank accepts dataset retrieval, request files, direct JSON payloads, or stdin.
- Artifact families: the CLI works with request JSONL, rerank JSONL, TREC runs, invocation histories, and aggregated evaluation JSONL.
- Hosted vs local backends: hosted provider paths usually need
cloud; local model paths usually need local or a batched backend extra.
- Prompt templates: rerank behavior is template-driven. Inspect bundled templates with
rank-llm prompt.
Gotchas
rerank requires one input source: --dataset, --requests-file, --input-json, or --stdin.- Dataset-backed
rerank also requires --retrieval-method.
rank-llm view detects .trec, .jsonl, and invocation-history .json artifacts, but it does not inspect arbitrary JSON files.evaluate operates on stored rerank outputs in a directory and writes trec_eval_aggregated_results_<model>.jsonl.analyze can return partial_success when a file mixes valid and malformed model outputs.serve http needs the api extra. serve mcp needs the mcp stack.
1---2name: rank-llm-quickstart3description: rank_llm Quickstart4---56# rank_llm Quickstart78Reference for the packaged `rank-llm` CLI.910## CLI Entry Point1112```bash13rank-llm <command> [options]14```1516## Primary Commands1718| Command | Purpose |19| --- | --- |20| `rerank` | Run reranking from dataset retrieval, request files, or direct JSON input |21| `evaluate` | Aggregate `trec_eval` metrics across stored rerank outputs |22| `analyze` | Analyze stored responses and error counts |23| `retrieve-cache` | Build cached retrieval JSON from an existing TREC run |24| `serve http` | Start the HTTP server |25| `serve mcp` | Start the MCP server |2627## Introspection Commands2829| Command | Purpose |30| --- | --- |31| `doctor` | Check Python version and optional dependency readiness |32| `describe <cmd>` | Return structured command metadata |33| `schema <name>` | Print JSON Schema for supported inputs and outputs |34| `validate rerank` | Validate rerank inputs without executing a model |35| `prompt list\|show\|render` | Inspect bundled prompt templates |36| `view <path>` | Inspect rerank JSONL, request JSONL, TREC runs, and invocation histories |3738## Quick Workflow3940```bash41# 1. Check environment42rank-llm doctor4344# 2. Run retrieval + reranking45rank-llm rerank --model-path castorini/rank_zephyr_7b_v1_full --dataset dl20 \46 --retrieval-method bm25 --top-k-candidates 100 \47 --output-jsonl-file rerank_results.jsonl --output-trec-file rerank_results.trec4849# 3. Inspect the artifact50rank-llm view rerank_results.jsonl5152# 4. Aggregate trec_eval metrics from stored outputs53rank-llm evaluate --model-name castorini/rank_zephyr_7b_v1_full5455# 5. Analyze invocation histories or stored response files56rank-llm analyze --files demo_outputs/inference_invocations_history.json --verbose57```5859## Reference Files6061Read these on demand for details:6263- `references/cli-examples.md` - Common invocations for each command64- `references/input-output-examples.md` - JSONL, TREC, and invocation-history artifact shapes65- `references/workflows.md` - Backend and workflow selection guide6667## Key Concepts6869- **Input modes**: `rerank` accepts dataset retrieval, request files, direct JSON payloads, or stdin.70- **Artifact families**: the CLI works with request JSONL, rerank JSONL, TREC runs, invocation histories, and aggregated evaluation JSONL.71- **Hosted vs local backends**: hosted provider paths usually need `cloud`; local model paths usually need `local` or a batched backend extra.72- **Prompt templates**: rerank behavior is template-driven. Inspect bundled templates with `rank-llm prompt`.7374## Gotchas7576- `rerank` requires one input source: `--dataset`, `--requests-file`, `--input-json`, or `--stdin`.77- Dataset-backed `rerank` also requires `--retrieval-method`.78- `rank-llm view` detects `.trec`, `.jsonl`, and invocation-history `.json` artifacts, but it does not inspect arbitrary JSON files.79- `evaluate` operates on stored rerank outputs in a directory and writes `trec_eval_aggregated_results_<model>.jsonl`.80- `analyze` can return `partial_success` when a file mixes valid and malformed model outputs.81- `serve http` needs the `api` extra. `serve mcp` needs the `mcp` stack.
castorini