# Rank LLM Quickstart

> rank_llm Quickstart

- Skill: `castorini/rank-llm-quickstart` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add castorini/rank-llm-quickstart`
- Raw SKILL.md: https://api.skillmd.com/api/skills/castorini/rank-llm-quickstart/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: castorini (https://skillmd.com/u/castorini)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/castorini/rank-llm-quickstart

---


# rank_llm Quickstart

Reference for the packaged `rank-llm` CLI.

## CLI Entry Point

```bash
rank-llm <command> [options]
```

## Primary Commands

| Command | Purpose |
| --- | --- |
| `rerank` | Run reranking from dataset retrieval, request files, or direct JSON input |
| `evaluate` | Aggregate `trec_eval` metrics across stored rerank outputs |
| `analyze` | Analyze stored responses and error counts |
| `retrieve-cache` | Build cached retrieval JSON from an existing TREC run |
| `serve http` | Start the HTTP server |
| `serve mcp` | Start the MCP server |

## Introspection Commands

| Command | Purpose |
| --- | --- |
| `doctor` | Check Python version and optional dependency readiness |
| `describe <cmd>` | Return structured command metadata |
| `schema <name>` | Print JSON Schema for supported inputs and outputs |
| `validate rerank` | Validate rerank inputs without executing a model |
| `prompt list\|show\|render` | Inspect bundled prompt templates |
| `view <path>` | Inspect rerank JSONL, request JSONL, TREC runs, and invocation histories |

## Quick Workflow

```bash
# 1. Check environment
rank-llm doctor

# 2. Run retrieval + reranking
rank-llm rerank --model-path castorini/rank_zephyr_7b_v1_full --dataset dl20 \
  --retrieval-method bm25 --top-k-candidates 100 \
  --output-jsonl-file rerank_results.jsonl --output-trec-file rerank_results.trec

# 3. Inspect the artifact
rank-llm view rerank_results.jsonl

# 4. Aggregate trec_eval metrics from stored outputs
rank-llm evaluate --model-name castorini/rank_zephyr_7b_v1_full

# 5. Analyze invocation histories or stored response files
rank-llm analyze --files demo_outputs/inference_invocations_history.json --verbose
```

## Reference Files

Read these on demand for details:

- `references/cli-examples.md` - Common invocations for each command
- `references/input-output-examples.md` - JSONL, TREC, and invocation-history artifact shapes
- `references/workflows.md` - Backend and workflow selection guide

## Key Concepts

- **Input modes**: `rerank` accepts dataset retrieval, request files, direct JSON payloads, or stdin.
- **Artifact families**: the CLI works with request JSONL, rerank JSONL, TREC runs, invocation histories, and aggregated evaluation JSONL.
- **Hosted vs local backends**: hosted provider paths usually need `cloud`; local model paths usually need `local` or a batched backend extra.
- **Prompt templates**: rerank behavior is template-driven. Inspect bundled templates with `rank-llm prompt`.

## Gotchas

- `rerank` requires one input source: `--dataset`, `--requests-file`, `--input-json`, or `--stdin`.
- Dataset-backed `rerank` also requires `--retrieval-method`.
- `rank-llm view` detects `.trec`, `.jsonl`, and invocation-history `.json` artifacts, but it does not inspect arbitrary JSON files.
- `evaluate` operates on stored rerank outputs in a directory and writes `trec_eval_aggregated_results_<model>.jsonl`.
- `analyze` can return `partial_success` when a file mixes valid and malformed model outputs.
- `serve http` needs the `api` extra. `serve mcp` needs the `mcp` stack.