API Database Scout

Discover every relevant API, database, dataset, and documentation resource for a given domain. Spawns parallel research agents across subcategories and delivers a comprehensive markdown catalog with doc links per source.

Usage: /api-database-scout <domain description>

Architecture — Why It Works This Way

This skill uses a file-based coordination pattern to stay within token limits. Research agents write their findings directly to files in /tmp/scout/{domain-slug}/. The orchestrator (you) never accumulates raw research results in your own context — you only track progress and delegate. A final compiler agent reads all files and produces the report. This keeps every agent lean and focused.

Instructions

You are the orchestrator. Your job is to break the domain into subcategories, spawn research agents, and delegate report compilation. You do NOT compile the report yourself.

The user's domain: $ARGUMENTS

Phase 1: Setup and Subcategory Expansion

First, create a domain-specific working directory. Derive a short kebab-case slug from the user's domain (e.g., "health and fitness tracking app" → "health-fitness"). Use this {domain-slug} consistently everywhere.

mkdir -p /tmp/scout/{domain-slug}

Then break the user's domain into the most granular subcategories you can. Think from multiple angles:

• Core functions — what does this domain actually do?
• Data types — what kinds of data flow through it?
• Adjacent domains — what fields overlap or feed into this one?
• Industry verticals — are there sector-specific data needs?
• Infrastructure — what supporting services does it need?

Aim for 12-20 subcategories. Present them to the user:

I've broken down "{domain}" into {N} subcategories for research:

1. {subcategory} — {one-line description}
2. {subcategory} — {one-line description}
...

Spawning research agents now. This will take a few minutes.

Phase 2: Parallel Deep Research

Spawn one research agent per subcategory — all in a single message so they run concurrently. Each agent is fully autonomous: it searches, iterates on gaps within its own subcategory, and writes results to a file.

Each research agent gets this prompt:

You are a research agent discovering APIs, databases, datasets, and documentation for one subcategory. Your job is to be THOROUGH — a developer is depending on this to know what exists. If you return only the 3 obvious results, you've failed.

DOMAIN: {user's domain}
SUBCATEGORY: {subcategory name} — {subcategory description}

## Search Strategy — Do ALL of These

1. **Direct search**: "{subcategory} API", "{subcategory} dataset", "{subcategory} open data"
2. **Directory search**: Search RapidAPI, public-apis GitHub repo, API aggregator sites
3. **Government portals**: Search data.gov, EU Open Data Portal, national statistics offices
4. **Academic/research**: Search Kaggle, Zenodo, UCI ML Repository, Papers With Code
5. **GitHub awesome-lists**: Search for "awesome-{subcategory}" repos
6. **Industry-specific**: Aggregators, consortiums, standards bodies for this subcategory
7. **Regional/international**: EU, UK, Asian equivalents — don't only find US sources
8. **Documentation**: For each API/dataset you find, locate the best documentation link (API reference, getting started guide, developer docs). Also search for standalone domain documentation — standards specs, whitepapers, regulatory guides, architecture references, and tutorials that aren't tied to a specific API but are essential for building in this subcategory.

## CRITICAL: URL Verification

You MUST verify every URL you include. For each source:
1. **WebFetch the URL** to confirm it resolves (HTTP 200) and the page content matches what you expect
2. If a URL returns 404, 403, or doesn't match expectations, **search again** to find the correct current URL
3. If you cannot verify a URL, either find the correct one or **drop the source entirely**
4. NEVER include a URL from memory alone — always verify by fetching it
5. Mark each source with a Verified field: "yes" (fetched and confirmed), "partial" (domain exists but exact page couldn't be confirmed), or "no" (could not verify — include only if the source is important enough to flag)

This is the single most important quality rule. A report full of dead links is worse than a shorter report with all working links.

## Iterate Within Your Subcategory

After your first round of searches:
- Check if you missed any of the 7 source types (Public API, Commercial API, Open Dataset, Government DB, Research Repo, SDK, Documentation)
- If a source type is missing, do a targeted search for it
- Split your subcategory into 2-3 sub-subcategories and search those with domain-specific terminology
- Search for "alternatives to {most popular source you found}" to find lesser-known options

Aim for **8-15 sources**. If you're under 5, search harder with different terms.

## Output Format

For each API, dataset, or SDK source, format as:

### {Source Name}
| Field | Value |
|-------|-------|
| URL | {direct link to docs or homepage} |
| Type | Public API / Commercial API / Open Dataset / Government DB / Research Repo / SDK |
| Access | free / freemium / paid / API key / OAuth / open data |
| Auth | {specific auth mechanism or "none"} |
| Format | {REST JSON, GraphQL, CSV, etc.} |
| Rate limits | {if known, otherwise "unknown"} |
| Data | {what it provides — be specific} |
| Docs | {direct link to API reference, getting started guide, or best developer documentation — if different from URL above. "see URL" if same} |
| Verified | yes / partial / no — did you WebFetch this URL and confirm it resolves? |
| Fit | {one sentence on relevance} |

For standalone documentation (specs, guides, whitepapers, tutorials not tied to a specific API), format as:

### {Document Title}
| Field | Value |
|-------|-------|
| URL | {direct link} |
| Type | Documentation |
| Doc type | Specification / Whitepaper / Regulatory guide / Tutorial / Architecture reference / Best practices guide |
| Access | free / paid / registration required |
| Relevance | {one sentence on why a developer in this subcategory needs this} |

## Save Your Results

Write your complete findings to: /tmp/scout/{domain-slug}/{subcategory_slug}.md

The file should start with a heading: `## {Subcategory Name}`
Then list all API/dataset/SDK sources first, followed by a `### Domain Documentation` sub-section with standalone docs.
At the end, add these lines:
- `<!-- sources: {count} -->` with the total number of sources you found (APIs + datasets + docs combined).
- `<!-- verified: {count} -->` with the number of sources where Verified = "yes".
- `<!-- unverified: {count} -->` with the number of sources where Verified = "no" or "partial".

Do NOT return your findings as text. Write them to the file. Just confirm: "Wrote {N} sources to /tmp/scout/{domain-slug}/{subcategory_slug}.md"

IMPORTANT: Replace {subcategory_slug} with a kebab-case version of the subcategory name (e.g., "Clinical Data" → "clinical-data"). This keeps filenames clean.

After all agents complete, do a quick tally:

grep -r '<!-- sources:' /tmp/scout/{domain-slug}/ | awk -F': ' '{sum += $2} END {print "Total sources:", sum}'

Tell the user:

Research complete: {N} sources found across {M} subcategories.
Compiling the final report now.

Phase 2.5: Validation Sweep

After all research agents complete, spawn a validation agent to spot-check URL quality and deduplicate cross-subcategory entries. This agent reads all research files and produces a cleaned version.

Validation agent prompt:

You are a validation agent. Your job is to improve the quality of a research report by checking URLs and removing duplicates.

DOMAIN: {user's domain}
RESEARCH DIR: /tmp/scout/{domain-slug}/

## Step 1: Read All Research Files

Read every .md file in /tmp/scout/{domain-slug}/ (except _all_sources.md if it exists).

## Step 2: Deduplicate

Many sources appear in multiple subcategories (e.g., a central government portal). For each duplicate:
- Keep the MOST COMPLETE entry (the one with the best description and most fields filled)
- In the other subcategory files, replace the full entry with a cross-reference line:
  `> **{Source Name}** — see [{original subcategory}](#{anchor}) for full details.`
- Track the total unique sources vs total mentions

## Step 3: Spot-Check Unverified URLs

For any source marked `Verified: no` or `Verified: partial`:
- WebFetch the URL to check if it resolves
- If it resolves, update to `Verified: yes`
- If it doesn't resolve, search for the correct URL. If found, update the entry. If not found, add a warning: `⚠️ URL could not be verified`
- Check at least 20 URLs total, prioritizing sources marked "no" or "partial"

## Step 4: Write Summary

Write a validation summary to /tmp/scout/{domain-slug}/_validation_summary.md:
- Total sources (with duplicates): {N}
- Unique sources (deduplicated): {N}
- Verified URLs: {N}
- Unverified/dead URLs: {N}
- Sources removed (dead links, no replacement found): {N}

Also update the individual subcategory files in place with your fixes.

Confirm: "Validation complete: {unique} unique sources, {verified} verified, {dead} removed."

After the validation agent completes, report the validation summary to the user before compiling.

Phase 3: Compile the Report

The research files are already well-formatted. Instead of having an agent re-read and rewrite everything (which can hit token limits for large reports), use a two-step compilation:

Step 3a: Concatenate all research files into the report body

Run this bash command to combine all subcategory files into one document:

cat /tmp/scout/{domain-slug}/*.md > /tmp/scout/{domain-slug}/_all_sources.md

Step 3b: Spawn a compiler agent to write the summary header

This agent only needs to scan the combined file for counts and write a short header — it does NOT rewrite the full report. This keeps it well within token limits.

Compiler agent prompt:

You are a report compiler. Your job is to create the FINAL report by writing a summary header and prepending it to the already-compiled source data.

DOMAIN: {user's domain}
REPORT PATH: {cwd}/{domain-slug}-api-scout-report.md

## Steps

1. Read /tmp/scout/{domain-slug}/_all_sources.md to understand what was found
2. Read /tmp/scout/{domain-slug}/_validation_summary.md for deduplication and verification stats
3. Count sources by type — search for "| Type |" lines and tally: Public API, Commercial API, Open Dataset, Government DB, Research Repo, SDK, Documentation
4. Count verified vs unverified — search for "| Verified |" lines and tally yes/partial/no
5. Build "Quick picks" lists by scanning "| Access |" lines — only include sources where Verified = "yes"
6. Count subcategories (## headings)
7. Write the FINAL report file to {cwd}/{domain-slug}-api-scout-report.md

The final report structure:
- First: write the summary header (title, date, coverage summary table, verification stats, quick picks)
- Include a "Data Quality" section showing: unique sources, verified count, unverified count
- Then: append the full contents of _all_sources.md after the header

Use this approach to write the file:
1. First, compose the header as a string
2. Then read _all_sources.md
3. Write a single file that combines: header + "\n---\n" + all_sources content

This way you never need to rewrite or reformat the individual source entries — they're already in the correct table format.

After writing the report, confirm: "Report saved to {path} — {N} sources across {M} subcategories."

After the compiler finishes, tell the user where the report is and give them the coverage summary.

Principles

• File-based coordination. Research agents write to /tmp/scout/{domain-slug}/. The compiler reads from there. No agent accumulates the full dataset in its context — this is how the skill scales to 100+ sources without hitting token limits.
• Exhaustiveness over speed. The whole point is to not miss things. Each research agent iterates within its own subcategory to fill gaps, so thoroughness is built into every agent, not just a top-level review.
• Verified URLs only. Every source must have a URL the user can visit. Research agents MUST WebFetch URLs to verify them. The validation phase catches anything that slips through. A shorter report with 100% working links beats a longer report with dead links.
• Deduplicate across subcategories. Sources that appear in multiple subcategories get one canonical entry and cross-references elsewhere. Report unique counts, not inflated totals.
• Parallel everything. Spawn all research agents in a single message. The user's time is valuable.
• Structure enables scanning. Consistent table format per entry lets users and Claude quickly filter and act on the results.
• Acknowledge gaps. If a subcategory has few results, say so explicitly — "No public APIs found for X" is useful information.
• Optimized for follow-up work. The report will be fed back to Claude for integration tasks. Auth, Format, and Rate limits fields let Claude write integration code immediately.