API Data Sources

Research and document all authoritative APIs and data sources for: $ARGUMENTS

$ARGUMENTS should include:

• A description of the app, project, or feature to research
• The target domain / industry (e.g. real estate, fitness, publishing)
• Optionally: budget constraints or pricing tier preferences (free-only, paid OK, enterprise)
• Optionally: data freshness requirements (real-time, daily, weekly, static)
• Optionally: a project slug for the output directory name
• Optionally: a custom output directory (default: _docs/api-sources/)
• Empty — ask the user to describe their project and use case

Purpose

This skill produces a complete API and data-source research package — a set of markdown files that fully document every authoritative data source available for a given project, how to access each one, what data it returns, what it costs, and exactly how to integrate it all into a React + Supabase + Vercel + AI Agents application.

What it produces:

• A project directory under _docs/api-sources/[project-slug]/
• _OVERVIEW.md — executive summary of all discovered sources
• apis/[api-name].md — one detailed file per API
• data-sources/[source-name].md — one file per non-API data source
• _DATA_CATALOG.md — human-readable inventory of all available data types
• _INTEGRATION_BLUEPRINT.md — step-by-step guide for React + Supabase + Vercel + AI integration

Phase 1 — Elicitation & Scoping

Before researching anything, understand the project deeply enough to filter results.

If the user's description is brief or ambiguous, ask up to 4 targeted questions. Skip any that are already answered by the description.

1. What does the app do? — What is the core user-facing functionality? What problem does it solve?
2. What data does the app need? — What types of information must the app display, process, or act on? (e.g. property listings, workout plans, book metadata, weather data)
3. What's the budget posture? — Free-tier only? Willing to pay per-request? Enterprise contracts OK? This filters which APIs to recommend.
4. What's the data freshness requirement? — Does data need to be real-time (websockets, streaming), near-real-time (polling every few minutes), daily batch, or static/occasional?

After scoping, write a Scoping Summary (internal, not output) that captures:

• The core domain(s) to search
• The data types needed (entities, fields, relationships)
• Budget constraints
• Freshness requirements
• Any explicit user preferences for specific providers

Phase 2 — API & Data Source Discovery

2.1 — Search Strategy

For each data type identified in the scoping summary, search systematically:

1. Search for official/authoritative APIs — Use web search with queries like:
   • "[domain] API" site:rapidapi.com OR site:programmableweb.com
   • "[data type] API" developer documentation
   • "[domain] open data" API
   • "[domain] REST API" OR "[domain] GraphQL API"
2. Check major API aggregators — RapidAPI, ProgrammableWeb, API List, Public APIs (github.com/public-apis/public-apis)
3. Check government/institutional open data — data.gov, EU Open Data Portal, World Bank, census.gov, etc.
4. Check domain-specific authoritative sources — Industry databases, professional associations, research institutions
5. Identify non-API data sources — RSS/Atom feeds, CSV/JSON open datasets, web scraping targets (with legality notes), file downloads, existing databases

2.2 — Source Classification

Categorize every discovered source into one of these types:

2.3 — Evaluation Criteria

For each source, assess:

• Authority — Is this the canonical/official source for this data? (prefer first-party APIs)
• Reliability — How stable is the API? Is there an SLA? When was it last updated?
• Coverage — Does it provide all the fields/entities needed, or only a subset?
• Cost — Free tier limits? Per-request pricing? Monthly caps?
• Developer Experience — Quality of docs, SDKs, community support
• Terms of Service — Any restrictions on storage, display, commercial use, or redistribution?

Rank sources as: Primary (recommended), Alternative (viable backup), or Supplementary (fills a gap not covered by primary).

Phase 3 — Per-Source Documentation

3.1 — API Documentation Template

For each API, create a file at apis/[api-name].md with this structure:

# [API Name]

> [One-line description of what this API provides]

| Field | Value |
|-------|-------|
| **Provider** | [Company/org name] |
| **Type** | REST / GraphQL / WebSocket / etc. |
| **Base URL** | `https://api.example.com/v2` |
| **Auth Method** | API Key / OAuth 2.0 / Bearer Token / None |
| **Official Docs** | [link to official documentation] |
| **Status Page** | [link if available] |
| **Role** | Primary / Alternative / Supplementary |

## Pricing & Quotas

| Tier | Price | Rate Limit | Monthly Quota | Notes |
|------|-------|------------|---------------|-------|
| Free | $0 | X req/sec | Y req/month | [any restrictions] |
| Pro | $X/mo | ... | ... | ... |
| Enterprise | Contact | ... | ... | ... |

> **Cost estimate for this project:** Based on [estimated usage pattern], expect ~[X] requests/month → [tier] tier → $[Y]/month.

## Getting Started

### Step 1 — Create an Account
[Exact steps: go to [URL], click sign up, verify email]

### Step 2 — Get API Credentials
[Exact steps: navigate to developer dashboard, create app, copy API key]

### Step 3 — Make Your First Request

```bash
curl -X GET "https://api.example.com/v2/endpoint" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json"

// TypeScript / fetch
const response = await fetch('https://api.example.com/v2/endpoint', {
  headers: {
    'Authorization': `Bearer ${process.env.API_KEY}`,
    'Content-Type': 'application/json',
  },
});
const data = await response.json();

Step 4 — Verify the Response

[Expected response shape — actual JSON example if available]

Key Endpoints

[Endpoint Name]

• Method: GET / POST / etc.
• Path: /v2/resource
• Description: [what it returns]
• Parameters: [key params with types and descriptions]
• Response Shape:

{
  "field": "type — description"
}

[Repeat for each relevant endpoint]

Data Fields Available

Rate Limits & Error Handling

• Rate limit: [X requests per second/minute]
• Rate limit header: X-RateLimit-Remaining
• Retry strategy: [exponential backoff recommended]
• Common errors:

  Status	Meaning	Action
  401	Invalid/expired API key	Refresh credentials
  429	Rate limited	Back off, retry after Retry-After header
  500	Server error	Retry with backoff

Terms of Service Notes

• [Any restrictions on caching, storage, redistribution, attribution requirements]
• [Link to full ToS]

SDK / Libraries

• Official SDK: [npm package name if available]
• Community libraries: [notable wrappers]

### 3.2 — Non-API Data Source Template

For each non-API source, create a file at `data-sources/[source-name].md`:

```markdown
# [Source Name]

> [One-line description]

| Field | Value |
|-------|-------|
| **Provider** | [Organization] |
| **Type** | Open Dataset / RSS Feed / Scraping Target / File |
| **URL** | [access URL] |
| **Format** | CSV / JSON / XML / HTML / PDF |
| **Update Frequency** | Real-time / Daily / Weekly / Monthly / Static |
| **License** | [license type — CC-BY, MIT, proprietary, etc.] |
| **Role** | Primary / Alternative / Supplementary |

## How to Access

[Step-by-step instructions — download, subscribe, or fetch]

## Data Fields Available

| Field | Type | Description |
|-------|------|-------------|
| ... | ... | ... |

## Ingestion Strategy

[How to get this data into Supabase — manual upload, scheduled cron, Edge Function, etc.]

## Legal & Attribution

[Any attribution requirements, usage restrictions, or licensing terms]

Phase 4 — Data Catalog

Create _DATA_CATALOG.md — a human-readable inventory of all data types available across all sources, filtered to the user's use case.

Structure:

# Data Catalog — [Project Name]

> All data types available for [use case description], organized by category.

## [Category 1 — e.g. "Properties" or "User Profiles"]

| Data Point | Type | Source(s) | Freshness | Notes |
|-----------|------|-----------|-----------|-------|
| [field name] | string/number/etc. | [API name(s)] | real-time/daily/etc. | [any notes] |
| ... | ... | ... | ... | ... |

## [Category 2 — e.g. "Market Data"]

| Data Point | Type | Source(s) | Freshness | Notes |
|-----------|------|-----------|-----------|-------|
| ... | ... | ... | ... | ... |

[Repeat for each logical category of data]

## Coverage Matrix

Shows which sources provide which data categories:

| Source | [Cat 1] | [Cat 2] | [Cat 3] | ... |
|--------|---------|---------|---------|-----|
| [API 1] | ✅ | ✅ | ❌ | ... |
| [API 2] | ❌ | ✅ | ✅ | ... |
| [Dataset 1] | ✅ | ❌ | ❌ | ... |

## Gaps & Recommendations

[Any data the user needs that isn't available from discovered sources, with suggestions for alternatives — AI generation, user-submitted data, manual entry, etc.]

Rules for the data catalog:

• Only include data types that are relevant to the stated use case — don't list every field an API returns
• Group by user-facing category, not by source
• Use plain language — a non-technical stakeholder should be able to read this and understand what data is available
• Call out gaps explicitly — if a needed data type isn't available from any source, say so

Phase 5 — Integration Blueprint

Create _INTEGRATION_BLUEPRINT.md — a step-by-step integration guide for wiring all sources into a React + Supabase + Vercel + AI Agents application.

Structure:

# Integration Blueprint — [Project Name]

> Step-by-step guide to integrate [N] APIs and [M] data sources into a React + Supabase + Vercel + AI Agents stack.

## Architecture Overview

[High-level description of how data flows through the stack]

[ASCII or Mermaid diagram showing]: External APIs → Supabase Edge Functions → Supabase DB → React Client ↕ AI Agents (OpenAI)

## Step 1 — Supabase Setup

### 1.1 — Database Schema

```sql
-- Tables derived from the data catalog
CREATE TABLE [entity] (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  -- fields mapped from API responses
  created_at TIMESTAMPTZ DEFAULT now(),
  updated_at TIMESTAMPTZ DEFAULT now()
);

[One CREATE TABLE per major entity, with comments explaining field origins]

1.2 — Row Level Security

-- RLS policies for each table
ALTER TABLE [entity] ENABLE ROW LEVEL SECURITY;

CREATE POLICY "..." ON [entity]
  FOR SELECT USING (auth.uid() = user_id);

1.3 — Environment Variables

# .env.local
SUPABASE_URL=your-project-url
SUPABASE_ANON_KEY=your-anon-key
SUPABASE_SERVICE_ROLE_KEY=your-service-key

# API keys for each external source
[API_NAME]_API_KEY=your-key
OPENAI_API_KEY=your-key

Step 2 — API Integration Layer

2.1 — Supabase Edge Functions

For each external API, create a Supabase Edge Function that:

1. Accepts a request from the client
2. Adds the API key server-side (never expose keys to the client)
3. Calls the external API
4. Transforms the response to match the DB schema
5. Optionally caches in Supabase

// supabase/functions/[api-name]/index.ts
import { serve } from 'https://deno.land/std@0.168.0/http/server.ts'
import { createClient } from 'https://esm.sh/@supabase/supabase-js@2'

serve(async (req) => {
  // [Implementation specific to each API]
})

[One code block per Edge Function with full implementation]

2.2 — Data Transformation

// lib/transformers/[api-name].ts
// Maps external API response shapes to internal DB schema
export function transform[Entity](apiResponse: ExternalType): InternalType {
  return {
    // field mappings
  }
}

2.3 — Caching Strategy

Step 3 — React Client Integration

3.1 — Supabase Client Setup

// lib/supabase.ts
import { createClient } from '@supabase/supabase-js'

export const supabase = createClient(
  process.env.NEXT_PUBLIC_SUPABASE_URL!,
  process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY!
)

3.2 — Data Hooks

// hooks/use[Entity].ts
import { useQuery } from '@tanstack/react-query' // or SWR, or Supabase realtime

export function use[Entity](params: Params) {
  return useQuery({
    queryKey: ['entity', params],
    queryFn: async () => {
      const { data, error } = await supabase
        .from('entity')
        .select('*')
        .match(params)
      if (error) throw error
      return data
    },
  })
}

[One hook per major data entity]

3.3 — Edge Function Calls (for fresh external data)

// lib/api/[api-name].ts
export async function fetch[Data](params: Params) {
  const { data, error } = await supabase.functions.invoke('[api-name]', {
    body: params,
  })
  if (error) throw error
  return data
}

Step 4 — AI Agent Integration

4.1 — Agent Architecture

[Describe how AI agents fit into the data flow]:

• Which data gets sent to the AI agent as context
• What the agent produces (summaries, recommendations, transformations, etc.)
• Whether the agent runs on-demand (user-triggered) or in the background (automated)

4.2 — OpenAI Integration

// lib/ai/agent.ts
import OpenAI from 'openai'

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
})

export async function run[Agent](context: ContextType) {
  const completion = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [
      { role: 'system', content: `[System prompt specific to this use case]` },
      { role: 'user', content: JSON.stringify(context) },
    ],
  })
  return completion.choices[0].message.content
}

4.3 — Agent as Supabase Edge Function

// supabase/functions/ai-[task]/index.ts
// Wraps the AI call in an Edge Function so the API key stays server-side

4.4 — Cost Estimation

Recommendation: [Which model to use and why, based on the use case complexity vs. cost]

Step 5 — Vercel Deployment

5.1 — Environment Variables

Set these in the Vercel dashboard under Project → Settings → Environment Variables:

5.2 — API Routes (if using Next.js API routes instead of / in addition to Edge Functions)

// app/api/[endpoint]/route.ts

5.3 — Cron Jobs (if needed)

// vercel.json
{
  "crons": [
    {
      "path": "/api/cron/refresh-[data]",
      "schedule": "0 */6 * * *"
    }
  ]
}

5.4 — Deployment Checklist

• All environment variables set in Vercel
• Supabase Edge Functions deployed (supabase functions deploy)
• Database migrations applied (supabase db push)
• RLS policies verified
• API rate limits tested under expected load
• AI agent costs estimated and budget alerts set
• Error monitoring configured (Sentry / Vercel Analytics)

Cost Summary

---

## Execution Steps

### Step 1 — Scope the Project
Read `$ARGUMENTS`. If insufficient, ask the 4 elicitation questions. Produce an internal scoping summary.

### Step 2 — Research APIs & Data Sources
Use web search extensively. For each domain/data-type identified in scoping, search for authoritative APIs. Visit official documentation pages to verify endpoints, pricing, and auth methods. Check API aggregators for alternatives.

### Step 3 — Create the Output Directory
Create `_docs/api-sources/[project-slug]/`, `apis/`, and `data-sources/` subdirectories.

### Step 4 — Write _OVERVIEW.md
Summarize all discovered sources, their roles (primary/alternative/supplementary), and the overall data strategy.

### Step 5 — Write Individual API Files
One file per API in `apis/`. Follow the API documentation template exactly. **Include working code examples** — don't leave placeholders. If you cannot verify an endpoint, note it as "unverified."

### Step 6 — Write Individual Data Source Files
One file per non-API source in `data-sources/`. Follow the data source template.

### Step 7 — Write _DATA_CATALOG.md
Compile the full data inventory. Organize by user-facing category. Include the coverage matrix. Call out gaps.

### Step 8 — Write _INTEGRATION_BLUEPRINT.md
Produce the full integration guide. Every code block should be as close to production-ready as possible. Include actual SQL schemas, actual TypeScript types, actual Edge Function boilerplate.

### Step 9 — Report
Summarize what was created using the output format below.

---

## Output Format

After generating all files, output:

API & Data Source Research: [Project Name]

Sources Discovered

• APIs: [count] ([count] primary, [count] alternative, [count] supplementary)
• Data Sources: [count]
• Data Types Cataloged: [count] across [count] categories

Files Created

• _docs/api-sources/[slug]/_OVERVIEW.md — Executive summary
• _docs/api-sources/[slug]/apis/[name].md — [one-line desc]
• ...
• _docs/api-sources/[slug]/data-sources/[name].md — [one-line desc]
• ...
• _docs/api-sources/[slug]/_DATA_CATALOG.md — Full data inventory
• _docs/api-sources/[slug]/_INTEGRATION_BLUEPRINT.md — React + Supabase + Vercel + AI guide

Estimated Monthly Cost

Coverage Gaps

• [Any data the user needs that no source provides]

Next Steps

1. Review _OVERVIEW.md for the recommended source selection
2. Sign up for API keys (links in each API file)
3. Follow _INTEGRATION_BLUEPRINT.md step by step
4. Start with the primary APIs; add alternatives if you hit rate limits or need more coverage

---

## Anti-Patterns

- **Don't recommend APIs you haven't verified.** If you can't confirm an API exists and has current documentation, don't include it. Note any uncertainty explicitly.
- **Don't skip pricing.** Every API file must include pricing information. If pricing isn't publicly available, say "Contact sales — pricing not public" rather than guessing.
- **Don't recommend scraping when an API exists.** Always prefer official APIs over scraping. Only suggest scraping as a last resort, and always note ToS implications.
- **Don't write placeholder code.** Code examples should be as close to copy-paste-ready as possible. Use realistic variable names, proper error handling, and TypeScript types.
- **Don't conflate data types with API endpoints.** The data catalog should be organized by what the user *sees* (entities, fields), not by which API returns them.
- **Don't ignore free tiers.** Many projects can run entirely on free tiers. Always document the free tier first, then paid upgrades.
- **Don't ignore rate limits.** Rate limits are a production concern. Always document them and include retry/backoff strategies.
- **Don't be vague about auth.** "Uses API key" is not enough. Document exactly where to put the key (header, query param, bearer token) with a working example.
- **Don't forget AI cost estimation.** AI agents can be the most expensive part of the stack. Always include token/cost estimates for the specific use case.