castorini
- 33 skills
- 0 followers
- 9 hours ago last updated
- ▌ Corpus Analysis · castorini bundleMeasure a parquet document corpus - token-length distribution, exact and near-duplicate detection, and how much survives deduplication. Use when asked to analyse corpus statistics, find duplicates, count tokens, compute what dedup would remove, or produce corpus figures for a paper. Triggers include "corpus stats", "duplicate detection", "near duplicates", "MinHash", "dedup", "how many tokens", "document length distribution", "ClimbMix analysis".
- ▌ Qrels Construction · castorini bundleBuild and audit per-question relevance judgments for a projected benchmark - join a reviewer workbook with verified questions, judge candidate documents against hop standards, expand the pool over exact and near duplicates, and produce the distribution figures. Use when asked to build qrels, finalize relevance judgments, expand judgments over duplicates, check how redundant a judged pool is, or make qrels figures. Triggers include "qrels", "relevance judgments", "judge documents", "duplicate expansion", "near duplicate qrels", "how many relevant documents per question".
- ▌ Use Piika · castorini bundleConfigure, run, evaluate, compare, summarize, and report Piika research and retrieval benchmarks. Use for Piika package setup; built-in, prebuilt, or custom benchmark manifests; local Anserini or remote Pyserini backends; single, shared, or sharded runs; answer, ranked-list, or combined outputs; supplied-document bundles; and reproducible evaluation workflows.
- ▌ Run Piika Nanoknow · castorini bundleInstall, run, evaluate, and report NanoKnow NQ-Open or NanoKnow SQuAD through Piika using the ClimbMix-supported query sets and climbmix-400b backend. Use for NanoKnow runs with normal retrieval or supplied-document bundles, multi-answer ground-truth preparation, smoke tests, full runs, and side-by-side experimental comparisons.
- ▌ Install Pyserini Uv · castorini bundleUse this skill when the user wants to install Pyserini with uv, create a uv-managed environment for Pyserini, add Pyserini to an existing uv project, install optional Pyserini extras, or debug uv/Pyserini installation issues involving Python, Java, PyPI, or dependency resolution.
- ▌ Install Pyserini Dev · castorini bundleUse this skill when the user wants to clone Pyserini from GitHub for local development, install Pyserini from source with pip editable mode using `pip install -e .`, create or reuse a development environment, install optional development extras, or debug source-checkout installation issues involving Python, Java, Git, pip, PyPI, or editable installs.
- ▌
- ▌ Anserini CLI · castorini bundleRun Anserini command-line and REST workflows from either a built fatjar or an Anserini source checkout. Use for PrebuiltIndexRegistry, TopicsRegistry, ad hoc search, interactive search, output formats, and RestServer examples.
- ▌ Anserini Reproduction · castorini bundleReproduce experimental results with Anserini. Use to run or explain Anserini reproduction workflows for published or reported results, including reproductions with prebuilt indexes, reproductions from raw document collections, reproduction YAMLs, run generation, evaluation, and metric verification.
- ▌ Install Anserini Fatjar · castorini bundleInstall and verify Anserini quickly by downloading the published fatjar from Maven Central instead of cloning or building the source repository. Use when users want fast setup, smoke tests, or CLI examples from a released Anserini jar.
- ▌ Install Anserini Dev Env · castorini bundleSet up and verify Anserini source-development environments. Use for JDK 21, Maven 3.9+, Anserini build scripts, smoke tests, and Java/Maven troubleshooting in castorini/anserini.
- ▌ Nuggetizer Eval · castorini bundleUse when analyzing nuggetizer metrics outputs — comparing runs, computing agreement between models, reporting token usage and latency from trace fields, and building side-by-side evaluation tables. Use after running metrics to interpret and compare results.
- ▌ Nuggetizer Verify · castorini bundleUse when validating nuggetizer batch outputs — checks JSONL integrity, duplicate qids, missing nugget fields, assignment label validity, and metrics consistency. Wraps `nuggetizer validate` plus custom assertions. Use after running create, assign, or metrics to verify output correctness.
- ▌ Nuggetizer Install · castorini bundleSet up a nuggetizer development environment — checks Python 3.11+, installs via uv or pip, and verifies with doctor. Use when someone is onboarding, setting up a fresh clone, or troubleshooting their environment.
- ▌ Nuggetizer Quickstart · castorini bundleUse when working with nuggetizer CLI commands (create, assign, metrics), prompt templates, JSONL formats, or introspection (doctor, describe, schema, validate). Covers all entry points, flags, and the YAML prompt system.
- ▌ Ragnarok Verify · castorini bundleUse when validating ragnarok generate outputs — checks TREC format compliance, citation integrity (valid indices, non-empty references), response length bounds, and JSONL structure. Wraps `ragnarok validate` plus custom assertions. Use after running generate or convert to verify output correctness.
- ▌ Ragnarok Dataset · castorini bundleUse when working with ragnarok dataset-backed generation — retrieval configuration, pyserini dependency setup, topk multi-stage pipelines, dataset naming conventions, and retrieval method selection. Use when setting up or debugging dataset-backed workflows.
- ▌ Ragnarok Install · castorini bundleSet up a ragnarok development environment — checks Python 3.11+, installs via uv or pip with cloud extras, and verifies with doctor. Use when someone is onboarding, setting up a fresh clone, or troubleshooting their environment.
- ▌ Ragnarok Quickstart · castorini bundleUse when working with ragnarok CLI commands (generate, validate, convert), prompt modes (chatqa, ragnarok_v4, etc.), dataset-backed or file-backed generation, TREC RAG output formats, or introspection (doctor, describe, schema). Covers all entry points, flags, and generation workflows.
- ▌ Rank LLM Eval · castorini bundleUse when analyzing rank_llm evaluation outputs across runs or models. Covers aggregated trec_eval JSONL files, response-analysis metrics, retrieval-cache handoff files, and side-by-side comparison of stored evaluation artifacts.
- ▌ Rank LLM Verify · castorini bundleUse when validating rank_llm artifacts after rerank, retrieve-cache, or related CLI workflows. Checks JSONL integrity, TREC formatting, candidate shape, invocation-history structure, and duplicate query identifiers. Wraps rank-llm view plus custom assertions.
- ▌ Rank LLM Install · castorini bundleSet up a rank_llm development environment. Use when someone is onboarding, setting up a fresh clone, choosing extras such as cloud, api, local, or pyserini, or troubleshooting whether the packaged rank-llm CLI is ready.
- ▌
- ▌ Umbrela Eval · castorini bundleUse when analyzing umbrela evaluation results — comparing nDCG@10 scores across backends, interpreting confusion matrices, computing kappa agreement, and comparing modified qrels against human judgments. Use after running evaluate to interpret results.
- ▌ Umbrela Verify · castorini bundleUse when validating umbrela judge outputs — checks label range (0–3), qid/docid completeness, result_status consistency, backend metadata, and JSONL integrity. Wraps `umbrela validate` plus custom assertions. Use after running judge or evaluate to verify output correctness.
- ▌ Umbrela Install · castorini bundleSet up an umbrela development environment — checks Python 3.11+, installs via uv or pip with cloud extras, and verifies with doctor. Use when someone is onboarding, setting up a fresh clone, or troubleshooting their environment.
- ▌ Umbrela Quickstart · castorini bundleUse when working with umbrela CLI commands (judge, evaluate), backend selection (gpt, gemini, hf, os, ensemble), qrel handling (dl19-passage, dl20-passage, etc.), relevance labels (0–3), or introspection (doctor, describe, schema, validate). Covers all entry points, flags, and evaluation workflows.
- ▌ Anserini Fatjar · castorini bundleInstall and run Anserini quickly by downloading the latest fatjar from the official Maven Central repo instead of building from source. Use when users want fast setup, smoke tests, or command execution without Maven project compilation.
- ▌ Castorini Serve · castorini bundleUse when serving Anserini retrieval together with any subset of rank_llm, ragnarok, nuggetizer, or umbrela over HTTP, especially for local port planning, direct request payload compatibility, curl or jq pipelines, or sequencing retrieval, reranking, generation, nugget creation, nugget assignment, and passage judging from an Anserini fatjar RestServer.
- ▌ Castorini Onboard · castorini bundleUse when onboarding to nuggetizer, ragnarok, rank_llm, or umbrela and you need development environment setup for one repo or several repos at once, including clone-if-needed, uv or pip installation paths, shared virtualenv reuse, and smoke tests.
- ▌ Castorini Release · castorini bundleUse when publishing nuggetizer, ragnarok, rank_llm, or umbrela to PyPI or TestPyPI and you need the release sequence for version bumps, build checks, twine validation, TestPyPI dry-runs, or final production publishing.
- ▌ Castorini Pipeline · castorini bundleUse when coordinating an end-to-end Castorini pipeline across rank_llm, ragnarok, nuggetizer, and umbrela, especially for stage handoffs, JSONL compatibility, retrieval-to-answer evaluation flow, or reproducing a multi-stage experiment.
- ▌ Castorini CLI Reference · castorini bundleUse when building, debugging, or reviewing CLI commands across nuggetizer, ragnarok, rank_llm, or umbrela and you need the shared castorini.cli.v1 envelope, common introspection commands, artifact shapes, or cross-repo CLI consistency rules.