API Data Fetcher

API Data Fetcher

Generate rigorous, reproducible Python pipelines for fetching economic data from public APIs. Apply DIME Analytics' reproducibility principles and data-cleaning discipline to Python: an immutable raw layer, a master orchestration script, dynamic absolute paths, secrets in env vars only, codebook-style source documentation, and asserted output schemas.

Operating Principles

1. Raw is immutable, like in DIME. Every API response is written to data/raw/ with a vintage suffix (_v2026-05-05) and never edited. Cleaning happens in data/intermediate/; analysis-ready outputs in data/processed/. This mirrors DIME's DataWork folder.
2. Secrets are PII. API keys are treated like personally identifiable information (DIME PII page): never hardcode, never commit, never log. Read from environment variables or .env files that are .gitignored.
3. Master script orchestrates. A pipelines/run_all.py file is the Python equivalent of DIME's master do-file: installs packages, defines paths via pathlib, sets seeds, runs each fetcher in order, and is the single command that reproduces the dataset.
4. Document provenance. Every fetched series carries a source codebook entry: source name, series ID, units, frequency, transformation, vintage, access date, license. This is the DIME data documentation principle applied to API data.
5. Cache deterministically. Cache files are content-addressed by request parameters; identical calls return identical bytes; cache invalidation is explicit, not silent. requests-cache or a hand-rolled JSON cache work fine.
6. Validate the schema. After every fetch, assert: expected columns, dtypes, frequency, date range, no extra series IDs returned, no silent NaN explosions. Use pandera or hand-rolled assertions.

Decision Policy

This skill follows the repo-wide Agent Policy. For cross-cutting data discipline (unit of analysis, ID variables, merging, master dataset, PII), this skill defers to working-with-data.

ASK before proceeding (blocking):

1. Indicators (which exact series IDs / indicator codes), geography, frequency, and date range.
2. Data vintage policy: real-time first releases vs current revisions (matters for forecasting / policy work).
3. Update cadence: one-shot vs scheduled refresh (changes how vintage tagging and caching are set up).
4. License attribution requirements (especially for redistribution).

DEFAULT + flag (use this default; tell the user how to override):

• One immutable parquet per series in data/raw/<source>/<series>_<vintage>.parquet.
• Long-format country/entity-date-indicator file in data/processed/ as the canonical analysis input.
• requests-cache SQLite cache or content-addressed JSON cache in data/cache/; explicit invalidation only.
• Hand-rolled assert schema validation; upgrade to pandera when the schema gets non-trivial.
• Source codebook CSV at docs/source_codebook.csv, one row per series.

DOCUMENT and proceed (write into the decisions log + docs/source_codebook.csv):

• Vintage and access timestamp for every fetched series.
• Source-side units, transformations applied locally, and the resulting unit.
• Any silent re-mapping (e.g. World Bank country code aliases).

PROCEED items: API keys via env vars only (.env + .gitignore), polite throttling (≥ 0.2s between calls), exponential backoff on 429/5xx, dynamic absolute paths via pathlib, mkdir(parents=True, exist_ok=True) for output folders.

Pre-flight Checklist

Before writing code, confirm — and write the answers in the script docstring:

• Indicators. Which series IDs (FRED) / indicators (World Bank) / topic codes (OECD)?
• Geography. Single country, custom country list, or all-available?
• Frequency. Annual, quarterly, monthly, daily? Are mixed frequencies acceptable?
• Date range. Fixed or rolling window?
• Vintage. Real-time data (fred.get_series_first_release) or current revisions?
• Update cadence. One-shot, daily, monthly?
• Output format. Long-format country-year-indicator parquet, or wide-format CSVs per indicator?
• License and citation. Is there an attribution requirement?

Project Layout (DIME-style, in Python)

ProjectABC/
├── pyproject.toml                       # pinned deps via uv / pip-tools
├── .env                                 # API keys; .gitignore'd
├── .gitignore
├── data/
│   ├── raw/                             # IMMUTABLE; vintage-suffixed files
│   │   ├── fred/
│   │   │   └── GDP_2026-05-05.csv
│   │   └── worldbank/
│   ├── intermediate/                    # cleaned per-source files
│   ├── processed/                       # analysis-ready, multi-source
│   └── cache/                           # request cache (content-addressed)
├── docs/
│   └── source_codebook.csv              # one row per fetched series
├── pipelines/
│   ├── run_all.py                       # master orchestration script
│   ├── fetch_fred.py
│   ├── fetch_worldbank.py
│   ├── fetch_bls.py
│   ├── fetch_oecd.py
│   ├── fetch_imf.py
│   └── build_panel.py                   # combine sources
└── README.md

Source Selection Decision Tree

US macro time series (GDP, CPI, unemployment, FFR, yields, M2):
└── FRED via `fredapi`. Set FRED_API_KEY.
    For real-time vintages: fred.get_series_first_release(...)
    or use ALFRED via the same package.

Cross-country development (GDP per capita, poverty, education, health):
└── World Bank Indicators via `wbdata` (no key needed) or `pandas-datareader`.

US labor (CES, CPS, JOLTS, OES):
└── BLS Public Data API via `bls` package. Register for a key for >25 series/day.

OECD panels (Main Economic Indicators, OECD.Stat):
└── `pandasdmx` against the OECD SDMX endpoint. No key.

IMF:
└── IFS, WEO, BOP via `imf-reader` (formerly `imfpy`) or direct SDMX with pandasdmx.

Eurostat:
└── `eurostat` package or `pandasdmx`.

Markets / financial:
└── `yfinance` for equities; for fixed-income use FRED.
    For commercial-grade data, route to Bloomberg / Refinitiv (paid, not in scope).

Geospatial / nightlights / weather:
└── See climate / GIS skills; this skill is for tabular API data.

Output Skeleton (run_all.py)

"""
Project   : ProjectABC
Pipeline  : Fetch, validate, and combine FRED + World Bank data.
Author    : First Last
Created   : 2026-05-05
Inputs    : public APIs (FRED, World Bank)
Outputs   : data/processed/macro_panel.parquet
            docs/source_codebook.csv
"""
from __future__ import annotations

import os
from datetime import date
from pathlib import Path

from pipelines.fetch_fred       import fetch_fred
from pipelines.fetch_worldbank  import fetch_worldbank
from pipelines.build_panel      import build_macro_panel

# 1. Paths (DIME-style: dynamic absolute paths)
PROJECT     = Path(__file__).resolve().parents[1]
DATA_RAW    = PROJECT / "data" / "raw"
DATA_INT    = PROJECT / "data" / "intermediate"
DATA_FINAL  = PROJECT / "data" / "processed"
DOCS        = PROJECT / "docs"
for p in (DATA_RAW, DATA_INT, DATA_FINAL, DOCS):
    p.mkdir(parents = True, exist_ok = True)

VINTAGE = date.today().isoformat()

# 2. Secrets (never hardcoded; read from env)
assert os.getenv("FRED_API_KEY"), "Set FRED_API_KEY in .env"

# 3. Run each fetcher (each writes to RAW with vintage suffix)
fetch_fred(out_dir = DATA_RAW / "fred",        vintage = VINTAGE)
fetch_worldbank(out_dir = DATA_RAW / "worldbank", vintage = VINTAGE)

# 4. Build the analysis-ready panel
build_macro_panel(
    raw_dir   = DATA_RAW,
    int_dir   = DATA_INT,
    final_dir = DATA_FINAL,
    docs_dir  = DOCS,
    vintage   = VINTAGE,
)

print(f"Pipeline complete. Vintage = {VINTAGE}")

Common Pitfalls

• Hardcoding API keys in scripts or notebooks. Use os.environ + .env.
• Overwriting raw files when re-running. Always include a vintage suffix.
• Silent frequency mixing (annual GDP joined to monthly CPI without resampling).
• Trusting series titles instead of series IDs. FRED renames titles; IDs are stable.
• Ignoring data revisions. For policy-relevant analysis use real-time vintages, not the latest revisions.
• Caching by URL only and missing changes in headers/parameters. Hash the full request.
• Pulling enormous datasets into memory. Stream to parquet and load lazily with pyarrow.
• Not recording units. World Bank / OECD often switch units between releases.
• Treating missing values as zero. World Bank coverage is uneven; drop or impute deliberately.

Additional Resources

• reference.md — extended patterns: source codebook schema, caching, validation, real-time vintages, rate limiting.
• examples/ — runnable scripts:
  • examples/economic_data_fetcher.py — minimal FRED + World Bank fetch (legacy reference)
  • examples/run_all.py — master orchestration script
  • examples/fetch_fred.py — vintage-aware FRED fetcher
  • examples/fetch_worldbank.py — World Bank country-year panel
  • examples/fetch_oecd_sdmx.py — OECD via SDMX/pandasdmx
  • examples/fetch_bls.py — BLS Public Data API
  • examples/build_panel.py — combine sources, validate schema, write parquet
  • examples/source_codebook.csv — example provenance log

Requirements

• Python >= 3.10.
• Core: pandas, pyarrow, requests, python-dotenv.
• Source clients: fredapi, wbdata, pandasdmx, bls, eurostat, yfinance.
• Validation: pandera (optional but recommended).

pip install pandas pyarrow requests python-dotenv pandera \
            fredapi wbdata pandasdmx bls eurostat yfinance

.env template:

FRED_API_KEY=your_fred_key_here
BLS_API_KEY=your_bls_key_here

.gitignore:

.env
data/raw/
data/intermediate/
data/cache/

References

DIME Conceptual Inputs

• DIME Analytics, Reproducible Research
• DIME Analytics, DataWork Folder
• DIME Analytics, Master Do-files
• DIME Analytics, Data Cleaning
• DIME Analytics, Personally Identifiable Information (PII)
• DIME Analytics, Data Documentation

API Documentation

• FRED API
• World Bank Indicators API
• BLS Public Data API
• OECD SDMX
• IMF Data Services
• Eurostat REST API

Style

• QuantEcon, Python Programming for Economics and Finance.
• Wickham (2014), Tidy Data.