# Common Crawl URL Index Miner

> Queries the Common Crawl Index API and CC-MAIN collections to surface historical URL coverage, MIME types, and crawl snapshots at scale. Handy for research workflows that need broad web recall without building a full crawler from scratch.

- Skill: `agentskillexchange/common-crawl-url-index-miner` (Agent Skill)
- Install (CLI): `npx skillmds@latest add agentskillexchange/common-crawl-url-index-miner`
- Raw SKILL.md: https://api.skillmd.com/api/skills/agentskillexchange/common-crawl-url-index-miner/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: agentskillexchange (https://skillmd.com/u/agentskillexchange)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/agentskillexchange/common-crawl-url-index-miner

---


# Common Crawl URL Index Miner

Queries the Common Crawl Index API and CC-MAIN collections to surface historical URL coverage, MIME types, and crawl snapshots at scale. Handy for research workflows that need broad web recall without building a full crawler from scratch.

## Installation

Use the upstream install or setup path that matches your environment:
- docker build . -t cc-index-table
- docker run --rm -ti cc-index-table --help
- docker run --rm --entrypoint=/opt/spark/bin/spark-submit cc-index-table
- docker run --mount=type=bind,source=/tmp/data,destination=/data --rm cc-index-table /data/in /data/out

Requirements and caveats from upstream:
- ## Building and running using Docker
- A [Dockerfile](./Dockerfile) is provided to compile the project and run the Spark job in a Docker container.
- build the Docker image:

Basic usage or getting-started notes:
- This projects provides a comprehensive set of example queries (SQL) and also Java code to fetch and process the WARC records matched by a SQL query.
- Run mvn spotless:check and mvn spotless:apply, see the [Spotless Maven guide](https://github.com/diffplug/spotless/blob/main/plugin-maven/README.md). Java formatting rules are defined in [eclipse-formatter.xml](eclips...
- run the table converter tool, here showing the command-line help (--help):

- Source: https://github.com/commoncrawl/cc-index-table
- Extracted from upstream docs: https://raw.githubusercontent.com/commoncrawl/cc-index-table/HEAD/README.md

## Source

- [Agent Skill Exchange](https://agentskillexchange.com/skills/common-crawl-url-index-miner/)