Corpus Analysis

Measure a parquet document corpus - token-length distribution, exact and near-duplicate detection, and how much survives deduplication. Use when asked to analyse corpus statistics, find duplicates, count tokens, compute what dedup would remove, or produce corpus figures for a paper. Triggers include "corpus stats", "duplicate detection", "near duplicates", "MinHash", "dedup", "how many tokens", "document length distribution", "ClimbMix analysis".

castorini Updated
Projecting QA benchmarks onto ClimbMix

File contents

castorini/cmass commit d240b3eee2

Frequently asked questions

npx get-skill add castorini/corpus-analysis