Fundamental And Macro Data

Fundamentals and macro data

Both halves of this domain share one failure: the obvious join is a look-ahead bug. Fundamentals are published months after the period they describe and are then rewritten; macro series are revised for years. Sections 2 and 5 are the ones that change your results.

1. SEC EDGAR — the two hard rules

HDR = {"User-Agent": "Your Company you@example.com"}   # REQUIRED — no UA → HTTP 403

✅ Empirically confirmed: no User-Agent → 403; any UA → 200. The header is enforced for presence, not content — but sending a fake browser UA is a policy violation and gets IPs blocked. ✅ Rate limit: 10 requests/second, quoted from the SEC webmaster FAQ.

🚨 Frames period syntax: duration facts use CY2023/CY2023Q1, but instantaneous facts (Assets, StockholdersEquity) need the I suffix — CY2023Q4I. ✅ Verified: Assets/USD/CY2023Q1 → 404, CY2023Q4I → 200. The #1 frames beginner error.

Library choice: edgartools (MIT, 2,648★, released 2026-09-02 — typed objects for 20+ form types, XBRL-standardized statements, built-in MCP server) is the default. sec-edgar-downloader downloads raw files only. Arelle is the only full XBRL validator — use it when correctness beats convenience. 🚨 python-xbrl is dead (2016). 🚨 edgar-crawler is GPL-3.0.

2. 🚨 Point-in-time fundamentals

2.1 Three dates, only one is availability

• period / reportDate — fiscal period end. The data does not exist yet. Using it is a 30–90 day look-ahead. The most common fundamental-data bug there is.
• filed / filingDate — the date EDGAR assigns after its cutoff (17:30 ET periodic reports; 22:00 ET for Forms 3/4/5). Safe-ish, but still admits a 90-minute post-close leak.
• acceptanceDateTime — the true wall-clock moment. Use this.

🚨 Verified, undocumented timezone inconsistency: the Submissions API's acceptanceDateTime is genuine UTC; the Financial Statement Data Sets' sub.txt.accepted is Eastern Time. The test: for filings stamped 18:00–21:59, Submissions shows 454 same-day / 4 next-day filingDate (consistent with UTC) while FSDS shows 0 / 370 (consistent with ET). Comparing them without converting introduces a silent 4–5 hour error.

Worked example: Apple's 8-K accepted 2026-07-30T20:30:28Z = 16:30 ET — 30 minutes after the close, but stamped filingDate 2026-07-30. Apple's earnings 8-Ks cluster at 20:30Z/21:30Z, i.e. systematically post-close. Rule: convert to exchange local; if ≥16:00, the earliest tradeable bar is the next session's open.

2.2 ✅ companyfacts IS point-in-time reconstructable — the bug is yours

It returns every vintage, one row per accession that ever reported the period, each with filed:

# WRONG — silently selects the restated value
df.drop_duplicates(subset=['start', 'end'], keep='last')

# RIGHT — latest vintage KNOWN AS OF the decision date
df[df.filed <= as_of].sort_values('filed').groupby(['start', 'end']).last()

Scale, measured on Apple alone: 408 (start, end, form) groups have differing values across vintages. AccountsPayableCurrent FY2017: 49,049M → 44,242M. AntidilutiveSecurities FY2019: 15.5M → 62.0M — that one is the 2020 4-for-1 split retroactively rewriting share counts. Every per-share metric in XBRL is silently split-adjusted backwards, so a PIT EPS study compares split-adjusted denominators against unadjusted prices unless you handle it.

2.3 🚨 The frames API is not point-in-time and cannot be made so

Its records carry no filed and no form — you cannot filter by vintage. Measured on the CY2023 annual-revenue frame, by extracting the filing year from each accn:

Only 18 of 3,141 values come from a filing actually made in 2023; over half come from filings made three years later. Never use frames for backtesting. It is a current-state cross-section only.

2.4 ✅ The PIT-safe bulk path: Financial Statement Data Sets

70 quarterly ZIPs, ~85 MB each, four TSVs (sub, num, pre, tag). PIT-safe by construction: each ZIP contains only filings made during that quarter (2026Q1's filed range is 20260102–20260331). Stack ZIPs in order and you get true point-in-time with no filtering logic. sub.txt carries prevrpt (superseded by a later amendment) and detail; num.txt has ddate, qtrs (0 = instantaneous, 1 = quarterly, 4 = annual), segments, coreg.

2.5 Identity, coverage and calendar traps

• ✅ EDGAR itself is survivorship-bias-free: Lehman (6,341 filings), Bear Stearns (3,344), Enron (351) are all intact. 🚨 But company_tickers.json is not — all three return tickers: []. Building a universe from it re-introduces the exact bias EDGAR would have avoided.
• 18% of CIKs map to >1 ticker. GOOGL → [GOOGL, GOOG, GOOGM, GOOGN]; BAC has 17; JPM's 9 include preferred shares (JPM-PC) and ETNs (VYLD, AMJB). A naive ticker→CIK join attaches JPMorgan's income statement to a note. Filter via company_tickers_exchange.json and drop -P* suffixes.
• CIKs are not stable entities — see research-integrity-guards §1 for the verified Washington Mutual → Mr. Cooper example.
• XBRL coverage starts ~2009. Lehman, Bear Stearns and Enron return 404 on companyfacts — they died before the mandate. Any XBRL study is structurally post-2009 and cannot include the GFC's casualties.
• Amendments and late filers (2025): 10-K/A 3,073 · 10-Q/A 1,238 · NT 10-K 746 · NT 10-Q 1,171. An NT 10-K (Form 12b-25) says the filing will be late — itself a tradeable negative signal, and it means the row simply won't exist on schedule. Handle absence explicitly; do not forward-fill.
• Fiscal-year misalignment: 95.8% of 10-Ks use fye=1231, 4.2% don't, and fy is a label: Target's period=20260131 carries fy=2025. Join on period, bucket to calendar quarters yourself. fye is sometimes internally inconsistent — don't trust it blindly.

3. PIT reference snippet

import requests, pandas as pd
HDR = {"User-Agent": "Your Company you@example.com"}

def pit_facts(cik: int, tag: str, as_of: str, taxonomy="us-gaap"):
    """Only data actually filed on or before `as_of`."""
    url = f"https://data.sec.gov/api/xbrl/companyconcept/CIK{cik:010d}/{taxonomy}/{tag}.json"
    facts = requests.get(url, headers=HDR, timeout=30).json()["units"]["USD"]
    df = pd.DataFrame(facts)
    df["filed"] = pd.to_datetime(df["filed"])
    df = df[df["filed"] <= pd.Timestamp(as_of)].sort_values("filed")
    return df.groupby(["start", "end"], as_index=False).last()   # latest vintage KNOWN THEN
# <=10 req/s. Never use /api/xbrl/frames/ for this — it has no `filed`.

4. Macro sources

Dead or moved: 🚨 pandasdmx (dead 2023 → sdmx1) · 🚨 the old IMF endpoint dataservices.imf.org no longer resolves at all (✅ verified; new: api.imf.org/external/sdmx/3.0/) · OECD moved to sdmx.oecd.org — the old host 301-redirects but dataset IDs changed shape, so hardcoded queries break even though the host resolves.

BLS API v2 works without registration (✅ verified). Registered: 500 queries/day, 50 series/query, 20 years/query. Unregistered v1: 25 / 25 / 10. Registration is free but must be renewed annually.

5. 🚨 Macro revisions — the dominant look-ahead

GDP, payrolls and CPI are revised for years. FRED's own example: 2013Q4 GDP was 17102.5 (2014-01-30) → 17080.7 (2014-02-28) → 17089.6 (2014-03-27).

Use realtime_start as the timestamp, not the observation date. A January figure is published in February; indexing it at January is a 1–3 month look-ahead.

🚨 Three verified bugs in fredapi 0.5.2 (from source):

1. get_series_as_of_date does not do what its docstring says — it returns a DataFrame with duplicate date rows (every revision up to the date), not the latest per date. You must add .groupby('date').last() yourself.
2. realtime_end is silently dropped — the parse line is commented out. You cannot directly tell when a vintage was superseded; infer it from the next realtime_start.
3. Unbounded downloads — get_series_as_of_date and get_series_first_release both call get_series_all_releases() with no realtime bounds (defaults 1776-07-04 → 9999-12-31). Cache it once rather than calling per-date in a loop.

from fredapi import Fred
fred = Fred(api_key='...')
allr = fred.get_series_all_releases('GDP')            # date | realtime_start | value
pit  = (allr[allr.realtime_start <= '2014-06-30']
        .sort_values('realtime_start')
        .groupby('date')['value'].last())             # REQUIRED — as_of_date does not do this

import pandas as pd
url = ("https://www.philadelphiafed.org/-/media/frbp/assets/surveys-and-data/"
       "real-time-data/data-files/xlsx/routputqvqd.xlsx")
rt = pd.read_excel(url, index_col=0)          # rows = obs period, cols = vintage
gdp_as_known_1999q1 = rt['ROUTPUT99Q1'].dropna()

Two more macro traps: seasonal adjustment rewrites already-published history with no new information (prefer NSA, or use the vintage current at each decision date); and when mixing quarterly macro with daily prices, forward-fill from the release date, never the observation date, and never interpolate() — interpolation is bidirectional and pulls future values backward.

❓ Not verified

fredapi claims are from source inspection, not execution (no API key available) · which FRED series actually have ALFRED vintages · whether OECD still publishes its revisions dataset under the new SDMX structure (assume no vintage support) · a general DBnomics vintage-enumeration endpoint (two paths returned 404) · Bank of England / Riksbank SDMX availability · the acceptanceDateTime timezone is an empirical inference, not SEC-documented — which is itself the hazard.

Per-library deep dives

The optional fin-libraries plugin carries a dedicated skill for each library below. Load one only after this skill has told you which library you want:

• lib-edgartools — edgartools
• lib-fredapi — fredapi