Public API Integration

Public API Integration

The source catalog is github.com/public-apis/public-apis — a Markdown table at README.md, fetchable raw from raw.githubusercontent.com/public-apis/public-apis/master/README.md. 1,683 entries, 52 categories, five columns: API, Description, Auth, HTTPS, CORS.

There is no rate-limit column, no uptime column, no licence column, and no last-verified column. Reading a row and writing a fetch call against it assumes all four. Treat the table as a shortlist generator, never as an integration spec.

The old api.publicapis.org JSON service is unreliable and should not be depended on. Parse the Markdown; it is the artefact the project actually maintains.

1. What the columns actually say

Only 297 of 1,683 entries are simultaneously no-auth, HTTPS, and CORS Yes — under one in five. That single number decides the architecture below.

Parsing note: rows are not perfectly uniform. Some carry trailing pipes, a few wrap the Auth value in backticks or embed a Postman button in the cell. Match on ^\|\s*\[(name)\]\((url)\)\s*\| and split the remainder on |, then normalise — strip backticks, trim, and treat anything unrecognised as Unknown.

2. The pipeline

Frame → Shortlist → Probe → Architect → Adapt → Degrade → Record.

Frame

Write one sentence: what field does the interface render, at what freshness, for how many users? "A weather API" is not a requirement. "Current temperature and condition icon for one city, refreshed every 10 minutes, on a page with maybe 200 daily views" is — and it rules out three quarters of the candidates immediately.

Shortlist

Fetch the raw README, filter to the category, then rank: HTTPS Yes is a hard filter; Auth No beats apiKey beats OAuth; CORS Yes beats Unknown beats No. Keep three candidates, not one. The top choice fails the probe often enough that having a second costs nothing now and saves a round trip later.

Probe — the step that is always skipped

Before writing a line of integration code, call the real endpoint:

curl -s -o /tmp/probe.json -w '%{http_code} %{time_total}s %{size_download}B\n' \
  'https://api.example.com/v1/thing?q=test'
curl -sI 'https://api.example.com/v1/thing?q=test' | grep -iE 'access-control|ratelimit|retry-after'

You are answering five questions the table cannot: does it still exist, what is the real response shape, how slow is it, what rate limit headers come back, and does Access-Control-Allow-Origin actually appear. A CORS header that is absent here overrides a Yes in the table. The table drifts; the response does not.

Architect

• Access-Control-Allow-Origin present and auth is No → browser-direct is allowed.
• Anything else → server-side route. This is the default, and it is the right default for two thirds of the catalog. A proxy route also gives you the only place to put caching, the key, and a timeout.

Adapt

One module per provider, exporting a function that returns your domain type, not theirs:

export interface Weather { tempC: number; condition: string; observedAt: Date }
export async function getWeather(city: string): Promise<Weather> { /* map here */ }

Provider fields never leak past this file. Swapping APIs then edits one module instead of every component that touched data.main.temp.

Degrade

Every call gets a timeout, a bounded retry with jitter on 429/5xx only, and a defined state for failed as well as loading and empty. A free API you do not pay for owes you nothing; treat unavailability as a normal state, not an exception.

Record

Note which API was chosen, which two were rejected and why, and what the probe measured. The next person to hit a failure needs to know whether the alternatives were already tried.

3. Running it

The three steps above ship as scripts, so this is a pipeline rather than a reading exercise. Run them; do not reimplement them.

The scripts ship beside this file, and the working directory is the project root rather than the skill directory — so resolve the skill root once and use it. A project install and a user install are both possible, hence the two candidates:

API_SKILL="$(ls -d .claude/skills/public-api-integration \
  ~/.claude/skills/public-api-integration 2>/dev/null | head -1)"

python3 "$API_SKILL/scripts/find_api.py" --need "weather forecast" --no-auth
python3 "$API_SKILL/scripts/probe_api.py" --url '<endpoint>' --name weather \
  --out probe-report.json
python3 "$API_SKILL/scripts/scaffold_client.py" --report probe-report.json --out ./src \
  --env-key WEATHER_API_KEY

find_api.py shortlists candidates and prints a docs_url for each. That is a documentation link, not a callable endpoint: open it, find the request URL its docs specify, and probe that. Probing the docs_url directly returns an HTML page, which probe_api.py reports as documentation-page.

find_api.py returns candidates, never a decision — it filters HTTPS hard, ranks relevance from term hits alone, and refuses to let a key-free API look like a match just because it is convenient. probe_api.py answers what the table cannot and writes the report. scaffold_client.py builds the client from the payload that was actually observed, so the types describe the response rather than the documentation.

Scaffolding refuses to overwrite without --force, and --dry-run prints the plan first. When the probe verdict is server-side-only the generated client points at a route handler rather than the third party, because that is the only arrangement in which the credential stays off the client.

4. Secrets

An apiKey entry means a secret exists from that moment. It goes in .env.local, is read only in server code, and .env* is in .gitignore before the key is pasted anywhere. NEXT_PUBLIC_, VITE_, and every equivalent prefix ship the value to the browser — a key behind one of those is public, and rotating it is the only fix.

Rules

MUST NOT — Integrate an API whose HTTPS column is No, or any endpoint that answers only over http://.

Why: A plaintext dependency inside an HTTPS page is blocked as mixed content by every current browser, so it fails outright in production even when it works in local development. Where it is not blocked — a server-side call — the response is modifiable in transit by anything on the path, which means the interface renders attacker-controlled data.

Incorrect:

Row: | [SomeAPI](http://api.example.com) | ... | No | No | Unknown |  → integrated anyway

Correct:

HTTPS column is `No` → discard the candidate and take the next one on the shortlist.

MUST NOT — Place an API key in client-side code, in a client-exposed environment variable (NEXT_PUBLIC_, VITE_, REACT_APP_), or in a committed file.

Why: Build-time inlining means a prefixed variable is a literal string in the shipped bundle, readable by anyone who opens the network tab — the prefix is an explicit declaration that the value is public, not a naming convention. A committed key stays in git history after deletion, so the only real remediation is rotation, and that is a task nobody schedules until the quota is already exhausted by someone else.

Incorrect:

const key = process.env.NEXT_PUBLIC_WEATHER_KEY  // shipped to the browser

Correct:

// app/api/weather/route.ts — server only
const key = process.env.WEATHER_API_KEY

MUST NOT — Integrate several APIs for the same field "for redundancy" before one has been shown to be insufficient.

Why: Each additional provider multiplies the failure surface, the response shapes to reconcile, and the secrets to manage, in exchange for redundancy the interface has not yet been shown to need. Fallback chains are also the code least likely to be exercised, so the second provider is usually broken by the time the first one fails.

Exceptions:

• A measured availability requirement exists and the primary has been observed to miss it.

MUST — Call the live endpoint and inspect status, body shape, latency, and response headers before writing any integration code against it.

Why: The catalog carries no last-verified field, so a row is evidence that an API existed when someone added it and nothing more. The probe is also the only source for the four facts the table omits entirely — real response shape, rate limit headers, latency, and whether CORS headers are actually sent — and each of those changes the code you would write.

Incorrect:

// table says CORS: Yes
const r = await fetch(url)  // written before anyone called the endpoint

Correct:

curl -s -o /tmp/p.json -w '%{http_code} %{time_total}s\n' 'https://api.example.com/v1/thing'
curl -sI 'https://api.example.com/v1/thing' | grep -i 'access-control\|ratelimit'

MUST — Route third-party calls through a server-side handler unless the probe showed an Access-Control-Allow-Origin header and the API needs no key.

Why: CORS is Unknown for 977 of the 1,683 catalogued entries and No for 150, so browser-direct is unsupported or unverified for roughly two thirds of the catalog and a Yes in the table is a claim rather than a measurement. The server route is also the only place that can hold the key, the cache, and the timeout, so choosing it by default collapses four decisions into one.

Exceptions:

• A probe confirmed the CORS header and the API requires no credential — then browser-direct removes a hop and a server dependency.

MUST — Wrap every third-party call in one module that returns a domain type you define, so no provider-shaped field reaches a component.

Why: Free APIs are the dependencies most likely to disappear, rate-limit, or change shape without notice, and the cost of that event is set entirely by how many files mention their field names. An adapter makes the swap a single-file edit and makes the response shape mockable in tests without a network.

Incorrect:

<span>{data.main.temp}</span>  // OpenWeather's shape, in a component

Correct:

const weather = await getWeather(city)   // Weather { tempC, condition, observedAt }
<span>{weather.tempC}</span>

MUST — Give every third-party call an explicit timeout, a bounded retry with jitter on 429 and 5xx only, and a designed failed state alongside loading and empty.

Why: An unpaid third-party API owes the project no availability, so failure is a normal operating state rather than an exception, and an interface with no failed state renders a spinner forever when it occurs. Retrying on 4xx other than 429 cannot succeed — the request is wrong, not unlucky — and retrying without jitter synchronises every client into a thundering herd against a service that is already struggling.

Incorrect:

const r = await fetch(url)          // no timeout, no retry policy, no failed state

Correct:

const r = await fetch(url, { signal: AbortSignal.timeout(5000) })
// retry 429/5xx only, 2 attempts, backoff * (1 + Math.random() * 0.3)

MUST — Read the provider's own terms, quota, and attribution requirements before shipping, and record what they say.

Why: The catalog has no licence or rate-limit column, so a row cannot tell you whether commercial use is permitted, whether attribution is required, or how many requests a day are free. Discovering any of those after launch means either a takedown or an unplanned migration, and both cost more than the ten minutes the check takes.

SHOULD NOT — Treat the Auth, HTTPS, or CORS columns as authoritative once a live response contradicts them.

Why: The table is community-maintained with no automated re-verification, so its columns record what was true when a contributor last looked. Where the observed response and the row disagree, the response is the current fact and the row is history.

SHOULD — Carry three candidates through the probe rather than committing to the first matching row.

Why: Probes fail often enough on a directory with no liveness checking that a single candidate turns a routine step into a restart of the whole selection. The marginal cost of two extra probes is seconds; the cost of re-deriving the shortlist later is the whole task.

Before reporting completion

Run these checks against your own output. Answer each question explicitly rather than assuming the answer, because the point of the exercise is to notice what you did not notice while building.

Probe the endpoint and fail if it is unreachable, non-2xx, or not JSON. Writes the report the scaffolder needs. (blocking)

python3 "$(ls -d .claude/skills/public-api-integration ~/.claude/skills/public-api-integration 2>/dev/null | head -1)"/scripts/probe_api.py --url '<endpoint>' --name '<name>' --out probe-report.json

Show every file the scaffolder would write, before it writes any of them.

python3 "$(ls -d .claude/skills/public-api-integration ~/.claude/skills/public-api-integration 2>/dev/null | head -1)"/scripts/scaffold_client.py --report probe-report.json --out ./src --dry-run

Confirm the API was chosen from evidence rather than from the first matching row. (blocking)

• Was the live endpoint actually called before integration code was written, and what were the status, latency, and response shape?
• Did the probe show an Access-Control-Allow-Origin header, and does that agree with the CORS column? If they disagree, which one did the architecture follow?
• Is the HTTPS column Yes for the chosen API, and does the endpoint refuse plaintext?
• Were three candidates shortlisted, and is it written down which two were rejected and why?
• Have the provider’s terms, quota, and attribution requirements been read, and is what they say recorded somewhere the next person will find it?

Confirm no credential reaches the client and the call path is deliberate. (blocking)

• Grep the built client bundle for the key value itself, not the variable name. Does it appear?
• Is every environment variable holding a secret free of a client-exposed prefix (NEXT_PUBLIC_, VITE_, REACT_APP_, PUBLIC_)?
• Was .env* in .gitignore before any real key was written to disk, and does git log -S "<key prefix>" come back empty?
• If the call is browser-direct, was that chosen because a probe confirmed CORS and no credential is needed — or because it was simply easier?
• Does the server route validate and constrain its own input, so it cannot be used as an open proxy to arbitrary upstream URLs?

Confirm the integration behaves when the third party is slow, rate-limited, or gone. (blocking)

• Block the endpoint in DevTools and reload. Does the interface reach a designed failed state, or does it spin forever?
• Does every call carry an explicit timeout, and is the retry policy limited to 429 and 5xx with jitter on the backoff?
• Does any provider-shaped field name appear outside the adapter module?
• What happens on a 429 specifically — is Retry-After read, and is the user told something more useful than "error"?
• Can the whole integration be tested without a network, by mocking the adapter rather than the HTTP layer?

Further reference

These are not loaded by default. Read one only when its question is the question you currently have.

• references/catalog-parsing.md — How do I turn the README table into a filtered, ranked candidate list, and what does the messy real-world markup look like?