How do I install the Castorini Rank LLM Rank LLM skill?

Run npx skillmds@latest add tomevault-io/castorini-rank-llm-rank-llm in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the Castorini Rank LLM Rank LLM skill do?

rank_llm Quickstart It is listed under AI & ML on SkillMD.

Is Castorini Rank LLM Rank LLM safe to use?

This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Castorini Rank LLM Rank LLM?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Castorini Rank LLM Rank LLM free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Castorini Rank LLM Rank LLM?

tomevault-io (@tomevault-io) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Castorini Rank LLM Rank LLM by tomevault-io

Name: Castorini Rank LLM Rank LLM
Author: tomevault-io

File contents

rank_llm Quickstart

Reference for the packaged rank-llm CLI.

CLI Entry Point

rank-llm <command> [options]

Primary Commands

Command	Purpose
`rerank`	Run reranking from dataset retrieval, request files, or direct JSON input
`evaluate`	Aggregate `trec_eval` metrics across stored rerank outputs
`analyze`	Analyze stored responses and error counts
`retrieve-cache`	Build cached retrieval JSON from an existing TREC run
`serve http`	Start the HTTP server
`serve mcp`	Start the MCP server

Introspection Commands

Command	Purpose
`doctor`	Check Python version and optional dependency readiness
`describe <cmd>`	Return structured command metadata
`schema <name>`	Print JSON Schema for supported inputs and outputs
`validate rerank`	Validate rerank inputs without executing a model
`prompt list\|show\|render`	Inspect bundled prompt templates
`view <path>`	Inspect rerank JSONL, request JSONL, TREC runs, and invocation histories

Quick Workflow

# 1. Check environment
rank-llm doctor

# 2. Run retrieval + reranking
rank-llm rerank --model-path castorini/rank_zephyr_7b_v1_full --dataset dl20 \
  --retrieval-method bm25 --top-k-candidates 100 \
  --output-jsonl-file rerank_results.jsonl --output-trec-file rerank_results.trec

# 3. Inspect the artifact
rank-llm view rerank_results.jsonl

# 4. Aggregate trec_eval metrics from stored outputs
rank-llm evaluate --model-name castorini/rank_zephyr_7b_v1_full

# 5. Analyze invocation histories or stored response files
rank-llm analyze --files demo_outputs/inference_invocations_history.json --verbose

Reference Files

Read these on demand for details:

references/cli-examples.md - Common invocations for each command
references/input-output-examples.md - JSONL, TREC, and invocation-history artifact shapes
references/workflows.md - Backend and workflow selection guide

Key Concepts

Input modes: rerank accepts dataset retrieval, request files, direct JSON payloads, or stdin.
Artifact families: the CLI works with request JSONL, rerank JSONL, TREC runs, invocation histories, and aggregated evaluation JSONL.
Hosted vs local backends: hosted provider paths usually need cloud; local model paths usually need local or a batched backend extra.
Prompt templates: rerank behavior is template-driven. Inspect bundled templates with rank-llm prompt.

Gotchas

rerank requires one input source: --dataset, --requests-file, --input-json, or --stdin.
Dataset-backed rerank also requires --retrieval-method.
rank-llm view detects .trec, .jsonl, and invocation-history .json artifacts, but it does not inspect arbitrary JSON files.
evaluate operates on stored rerank outputs in a directory and writes trec_eval_aggregated_results_<model>.jsonl.
analyze can return partial_success when a file mixes valid and malformed model outputs.
serve http needs the api extra. serve mcp needs the mcp stack.

Source: castorini/rank_llm — distributed by TomeVault.

---name: castorini-rank-llm-rank-llmdescription: rank_llm Quickstart---# rank_llm QuickstartReference for the packaged `rank-llm` CLI.## CLI Entry Point```bashrank-llm <command> [options]```## Primary Commands| Command | Purpose || --- | --- || `rerank` | Run reranking from dataset retrieval, request files, or direct JSON input || `evaluate` | Aggregate `trec_eval` metrics across stored rerank outputs || `analyze` | Analyze stored responses and error counts || `retrieve-cache` | Build cached retrieval JSON from an existing TREC run || `serve http` | Start the HTTP server || `serve mcp` | Start the MCP server |## Introspection Commands| Command | Purpose || --- | --- || `doctor` | Check Python version and optional dependency readiness || `describe <cmd>` | Return structured command metadata || `schema <name>` | Print JSON Schema for supported inputs and outputs || `validate rerank` | Validate rerank inputs without executing a model || `prompt list\|show\|render` | Inspect bundled prompt templates || `view <path>` | Inspect rerank JSONL, request JSONL, TREC runs, and invocation histories |## Quick Workflow```bash# 1. Check environmentrank-llm doctor# 2. Run retrieval + rerankingrank-llm rerank --model-path castorini/rank_zephyr_7b_v1_full --dataset dl20 \ --retrieval-method bm25 --top-k-candidates 100 \ --output-jsonl-file rerank_results.jsonl --output-trec-file rerank_results.trec# 3. Inspect the artifactrank-llm view rerank_results.jsonl# 4. Aggregate trec_eval metrics from stored outputsrank-llm evaluate --model-name castorini/rank_zephyr_7b_v1_full# 5. Analyze invocation histories or stored response filesrank-llm analyze --files demo_outputs/inference_invocations_history.json --verbose```## Reference FilesRead these on demand for details:- `references/cli-examples.md` - Common invocations for each command- `references/input-output-examples.md` - JSONL, TREC, and invocation-history artifact shapes- `references/workflows.md` - Backend and workflow selection guide## Key Concepts- **Input modes**: `rerank` accepts dataset retrieval, request files, direct JSON payloads, or stdin.- **Artifact families**: the CLI works with request JSONL, rerank JSONL, TREC runs, invocation histories, and aggregated evaluation JSONL.- **Hosted vs local backends**: hosted provider paths usually need `cloud`; local model paths usually need `local` or a batched backend extra.- **Prompt templates**: rerank behavior is template-driven. Inspect bundled templates with `rank-llm prompt`.## Gotchas- `rerank` requires one input source: `--dataset`, `--requests-file`, `--input-json`, or `--stdin`.- Dataset-backed `rerank` also requires `--retrieval-method`.- `rank-llm view` detects `.trec`, `.jsonl`, and invocation-history `.json` artifacts, but it does not inspect arbitrary JSON files.- `evaluate` operates on stored rerank outputs in a directory and writes `trec_eval_aggregated_results_<model>.jsonl`.- `analyze` can return `partial_success` when a file mixes valid and malformed model outputs.- `serve http` needs the `api` extra. `serve mcp` needs the `mcp` stack.---> Source: [castorini/rank_llm](https://github.com/castorini/rank_llm) — distributed by [TomeVault](https://tomevault.io).

Frequently asked questions

Castorini Rank LLM Rank LLM