Ml4t Canonical Schema

Canonical Schema

When every dataset uses different column names - date vs ts_event vs timestamp, asset vs ticker vs symbol - every downstream notebook needs special-case handling.

The Problem

Provider A delivers date, ticker, close; Provider B uses ts_event, symbol, price; Provider C uses timestamp, asset, adj_close. Without a canonical schema, every downstream notebook needs provider-specific renames.

The Pattern

WRONG

import polars as pl

# Different column names per dataset - downstream code breaks constantly
etfs = pl.read_parquet("etfs.parquet")        # has: date, ticker, close
futures = pl.read_parquet("futures.parquet")   # has: ts_event, product, settle
crypto = pl.read_parquet("crypto.parquet")     # has: timestamp, symbol, close

# Every notebook needs provider-specific column mapping
if "date" in df.columns:
    df = df.rename({"date": "timestamp"})
elif "ts_event" in df.columns:
    df = df.rename({"ts_event": "timestamp"})
# Repeat for every column, every dataset, every notebook

CORRECT

import polars as pl

# Canonical schema: enforced once at load time, trusted everywhere after
CANONICAL_COLUMNS = {
    "time": "timestamp",    # ALL frequencies: daily, hourly, minute
    "entity": "symbol",     # Exception: cme_futures uses "product"
}

def enforce_schema(df: pl.DataFrame, dataset: str) -> pl.DataFrame:
    """Rename provider columns to canonical names at load time."""
    renames = {}
    # Time column: accept common variants, output "timestamp"
    for variant in ["date", "ts_event", "datetime", "time"]:
        if variant in df.columns:
            renames[variant] = "timestamp"
    # Entity column: accept common variants, output "symbol"
    if dataset != "cme_futures":  # futures use "product"
        for variant in ["asset", "ticker", "pair", "instrument"]:
            if variant in df.columns:
                renames[variant] = "symbol"
    return df.rename(renames)

# Load once, use everywhere - no downstream renames needed
etfs = enforce_schema(pl.read_parquet("etfs.parquet"), "etfs")
assert "timestamp" in etfs.columns
assert "symbol" in etfs.columns

The Two Canonical Columns

OHLCV Columns

Lowercase, no prefix: open, high, low, close, volume. If adjustments exist: adj_close.

Enforcement Point

Schema enforcement happens at load time, not downstream. This means:

1. Data loaders validate and rename on return
2. Notebooks never import raw provider data directly
3. If a notebook gets a ColumnNotFoundError for date or asset, the notebook is wrong - fix the notebook to use timestamp or symbol

Guardrails

• Never rename canonical columns back to legacy names in notebooks - fix the notebook
• Never add compatibility shims that accept both old and new names - migrate forward
• If a new provider uses a different name, add the rename in the loader, not in 50 notebooks
• product is only for CME futures - do not generalize to other datasets
• Check for legacy names (asset, date, ticker, pair) during code review

Production Implementation

ml4t-data standardizes generic OHLCV fetches to canonical columns:

from ml4t.data import DataManager

dm = DataManager()
panel = dm.batch_load(
    ["SPY", "QQQ"],
    start="2015-01-01",
    end="2024-12-31",
    provider="yahoo",
)
# Columns: timestamp, symbol, open, high, low, close, volume

Checklist

• All data loaded through loaders that enforce canonical names
• Time column is timestamp for every dataset and frequency
• Entity column is symbol (or product for CME futures only)
• OHLCV columns are lowercase: open, high, low, close, volume
• No legacy names (date, asset, ticker) in notebooks; schema enforced at load time