# Castorini Rank LLM Rank LLM

> rank_llm Quickstart

- Skill: `tomevault-io/castorini-rank-llm-rank-llm` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/castorini-rank-llm-rank-llm`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/castorini-rank-llm-rank-llm/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/castorini-rank-llm-rank-llm

---


# rank_llm Quickstart

Reference for the packaged `rank-llm` CLI.

## CLI Entry Point

```bash
rank-llm <command> [options]
```

## Primary Commands

| Command | Purpose |
| --- | --- |
| `rerank` | Run reranking from dataset retrieval, request files, or direct JSON input |
| `evaluate` | Aggregate `trec_eval` metrics across stored rerank outputs |
| `analyze` | Analyze stored responses and error counts |
| `retrieve-cache` | Build cached retrieval JSON from an existing TREC run |
| `serve http` | Start the HTTP server |
| `serve mcp` | Start the MCP server |

## Introspection Commands

| Command | Purpose |
| --- | --- |
| `doctor` | Check Python version and optional dependency readiness |
| `describe <cmd>` | Return structured command metadata |
| `schema <name>` | Print JSON Schema for supported inputs and outputs |
| `validate rerank` | Validate rerank inputs without executing a model |
| `prompt list\|show\|render` | Inspect bundled prompt templates |
| `view <path>` | Inspect rerank JSONL, request JSONL, TREC runs, and invocation histories |

## Quick Workflow

```bash
# 1. Check environment
rank-llm doctor

# 2. Run retrieval + reranking
rank-llm rerank --model-path castorini/rank_zephyr_7b_v1_full --dataset dl20 \
  --retrieval-method bm25 --top-k-candidates 100 \
  --output-jsonl-file rerank_results.jsonl --output-trec-file rerank_results.trec

# 3. Inspect the artifact
rank-llm view rerank_results.jsonl

# 4. Aggregate trec_eval metrics from stored outputs
rank-llm evaluate --model-name castorini/rank_zephyr_7b_v1_full

# 5. Analyze invocation histories or stored response files
rank-llm analyze --files demo_outputs/inference_invocations_history.json --verbose
```

## Reference Files

Read these on demand for details:

- `references/cli-examples.md` - Common invocations for each command
- `references/input-output-examples.md` - JSONL, TREC, and invocation-history artifact shapes
- `references/workflows.md` - Backend and workflow selection guide

## Key Concepts

- **Input modes**: `rerank` accepts dataset retrieval, request files, direct JSON payloads, or stdin.
- **Artifact families**: the CLI works with request JSONL, rerank JSONL, TREC runs, invocation histories, and aggregated evaluation JSONL.
- **Hosted vs local backends**: hosted provider paths usually need `cloud`; local model paths usually need `local` or a batched backend extra.
- **Prompt templates**: rerank behavior is template-driven. Inspect bundled templates with `rank-llm prompt`.

## Gotchas

- `rerank` requires one input source: `--dataset`, `--requests-file`, `--input-json`, or `--stdin`.
- Dataset-backed `rerank` also requires `--retrieval-method`.
- `rank-llm view` detects `.trec`, `.jsonl`, and invocation-history `.json` artifacts, but it does not inspect arbitrary JSON files.
- `evaluate` operates on stored rerank outputs in a directory and writes `trec_eval_aggregated_results_<model>.jsonl`.
- `analyze` can return `partial_success` when a file mixes valid and malformed model outputs.
- `serve http` needs the `api` extra. `serve mcp` needs the `mcp` stack.

---
> Source: [castorini/rank_llm](https://github.com/castorini/rank_llm) — distributed by [TomeVault](https://tomevault.io).
<!-- tomevault:4.0:skill_md:2026-06-29 -->