Market Data Sourcing

Market data sourcing

Pick the source from the constraint that actually binds — usually survivorship coverage or licence, not price or convenience. Then read that library's reference file before writing code; every one of them has a default that is wrong for research.

1. Pick a source

Do not use: investpy (dead 2022, Investing.com blocks it), investiny (dead the day it was released), yahoo_fin (broken by Yahoo's crumb auth), thepassiveinvestor (archived), the PyPI marketstack package (1★, LGPL, 2022 — call the REST API with requests instead), alpaca-trade-api (→ alpaca-py).

2. The four traps that produce wrong numbers silently

2a. Survivorship

Yahoo-derived sources have no delisted securities. yfinance's own error class is YFTickerMissingError with a "possibly delisted; " prefix — the library guesses, because Yahoo won't say. This applies to yfinance, yahooquery, stockdex, defeatbeta-api, financedatabase, financetoolkit(free) and Alpha Vantage alike. Do not build a backtest universe from any of them. Full coverage table: research-integrity-guards §1.

2b. Adjustment defaults disagree across libraries

🚨 And the adjusted series itself is not stable. Yahoo-style adjusted prices are anchored at the present (Adj Close == Close on the latest date), so every new dividend or split rewrites the whole history. Snapshot and hash what you actually used; see ../research-integrity-guards/references/adjustment-conventions.md.

Reconciling yfinance against yahooquery without setting these explicitly produces silent mismatches around every split. Always pass the adjustment argument explicitly, even when the default is what you want — it documents intent and survives the next default flip.

2c. Timezones and off-by-one

• yf.download(end=...) is EXCLUSIVE. end='2023-01-01' gives you a last bar of 2022-12-31.
• 🚨 yfinance intraday returned UTC before 1.4.0 (2026-05-23) and exchange-local since. A stored intraday dataset built across that boundary has shifted bars. Set ignore_tz explicitly.
• multi_level_index=True is the default, so even a single ticker returns a MultiIndex frame. Pass multi_level_index=False for a flat one.

2d. Licence ≠ data licence

yfinance is Apache-2.0, but its README says Yahoo's API is "intended for personal use only" and that your rights to the data are governed by Yahoo's ToS. The same split applies to every scraper. Tiingo's terms are stricter still: "you may only use the data for your own personal use and you may not display or share the data with another person or organization."

🚨 AGPL-3.0 in this domain: openbb, dbnomics. GPL: edgar-crawler. If the work will be served over a network, these are blockers.

3. Free-tier reality (verified 2026-09-03)

4. Canonical safe snippet

import yfinance as yf

df = yf.download(
    ["AAPL", "MSFT"],
    start="2020-01-01", end="2024-01-01",   # end is EXCLUSIVE
    interval="1d",
    auto_adjust=True,        # explicit: the default flipped in 0.2.51 and again at 1.0
    group_by="column",
    multi_level_index=False, # flat columns even for one ticker
    progress=False,          # never print in a pipeline
    threads=True,
)
# NOTE: with auto_adjust=True there is NO 'Adj Close' column — OHLC are already adjusted.
# NOTE: this universe has NO delisted names. Do not backtest a screen on it.

5. Calendars and identifiers

• Sessions/holidays: exchange_calendars (4.13.2, Apache-2.0, genuinely well maintained) or pandas_market_calendars (5.4.0). Never hand-roll a business-day calendar — half-days and holiday drift will break your alignment. But it has verified defects — see below.

🚨 exchange_calendars' default date bounds are a MOVING TARGET, and this breaks reproducibility. ✅ Verified: GLOBAL_DEFAULT_START / _END are computed from pd.Timestamp.now() at import as today − 20 years and today + 1 year. Running on 2026-09-04, get_calendar("XTKS") returns first_session=2006-09-04, last_session=2027-09-03. The same code returns a different calendar tomorrow. → Always pass explicit start= and end=.

🚨 There is no XNSE. ✅ Verified: "XNSE" in get_calendar_names() → False. India is XBOM (BSE) only — used as a silent proxy for NSE, which carries most Indian volume. (XBSE is Bucharest, not Bombay.) Other verified gaps: Korean CSAT late-open dates stop at 2021-11-18 even on master, so XKRX.session_open("2024-11-14") wrongly returns 09:00; XSES models no lunch break ever despite SGX having one until 2011; and XBOM has zero sessions in 2027 because Indian holidays are hand-maintained annually. Detail: ../../../fin-asia/skills/asia-pacific-markets/SKILL.md.

• Never resample intraday bars over wall-clock time across a session break. Compute rolling windows over bar index.
• Identifier mapping (ticker ↔ CIK ↔ FIGI ↔ PERMNO) is a source of silent joins onto the wrong entity — see research-integrity-guards §1 and fundamental-and-macro-data.

6. Reference files

One file per library in references/, each with: exact pip name, version + release date, licence, maintenance verdict, coverage, free-tier limits, traps, and a minimal snippet. Grep them rather than guessing:

grep -ril "delisted" plugins/fin-core/skills/market-data-sourcing/references/
grep -i -A6 "TRAP" plugins/fin-core/skills/market-data-sourcing/references/yfinance.md

Start from references/_decision-table.md for the full side-by-side comparison.

Per-library deep dives

The optional fin-libraries plugin carries a dedicated skill for each library below. Load one only after this skill has told you which library you want:

• lib-yfinance — yfinance

ETF price series are not index series

Daily-reset leverage, NAV against price, distributions that print as phantom drops, and a holdings file used as a look-ahead universe - ../etf-mechanics/SKILL.md.