Fractional Differentiation

Fractional differentiation

Returns are stationary and memoryless; prices remember and fail every stationarity test. d is the dial between them, and almost nobody turns it. Advances in Financial Machine Learning (Lopez de Prado 2018), chapter 5. The default pipeline sets d = 1 by reflex — df.pct_change() — and that is the maximum possible amount of differencing, not the minimum needed.

Every number below is printed by scripts/frac_diff.py (numpy + scipy, seed 0, 3.7 s, statsmodels optional). The test series is a mildly non-stationary log price: an AR(1) with phi = 0.999, T = 3,000, so E[r_{t+1} | p_t] = (phi - 1) p_t exactly — the level is the entire edge and a single return contains almost none of it. That is the DGP on which "differencing destroys memory" is a measurable statement rather than a slogan.

⚠️ Section and page numbers below are as cited in mlfinpy 0.1.2's own docstrings; what is verified here is the code, not the book.

1. The weight recursion, and why it is not a moving average

(1 - B)^d expanded as a binomial series gives weights on the levels, newest first:

w[0] = 1        w[k] = -w[k-1] * (d - k + 1) / k

✅ Measured: the recursion equals the closed form w_k = (-1)^k C(d, k) (scipy.special.binom, an independent route) to 1.1e-16 over the first 500 terms at d = 0.15, 0.40 and 0.75. ✅ d = 1 returns exactly [1, -1, 0, 0, ...] — the first difference — and d = 0 the identity.

The tail decays like k^{-(1+d)}, so it never terminates for non-integer d. Two fixes exist and they are different features:

• expanding window (book section 5.4.1): weights recomputed to the start of the sample, so each output point uses a different number of terms. The book notes this "leads to negative drift caused by an expanding window's added weights".
• fixed-width window / FFD (section 5.4.2, page 83): truncate at the first |w_k| < thresh and use the same weight vector everywhere. This is the one to use — every output point is the same linear filter, so the series is comparable with itself over time.

✅ Source-verified against mlfinpy 0.1.2 (mlfinpy/util/frac_diff.py). Its get_weights runs this recursion and returns np.array(weights[::-1]).reshape(-1, 1) — oldest first, as an (N, 1) column, the opposite orientation to the array above. Its get_weights_ffd(diff_amt, thresh, lim) breaks before appending the first sub-threshold weight, so the window length matches. scripts/frac_diff.py carries that function transcribed verbatim and checks the two against each other: max abs diff 0.0e+00 at d = 0.05, 0.20, 0.35, 0.50, 0.75, 1.00.

🚨 sum(w) is the whole point and it is not zero. Truncating at a threshold leaves a residual weight on the level, and that residual is the memory:

At d = 1 the weights sum to zero exactly: the output is a pure difference and carries no level at all. Everything below follows from that one row.

2. The scan: the smallest d that passes ADF

The book's procedure (section 5.6, page 85; mlfinpy's plot_min_ffd) is: sweep d, run adfuller(..., maxlag=1, autolag=None, regression="c") on the FFD series, and plot the ADF statistic against the correlation with the original level. ✅ The script's own ADF t-statistic matches statsmodels 0.15.0's to +3.3e-15 on this series (own -1.6860, statsmodels -1.6860, its 5 % value -2.863).

✅ Measured, seed 0, thresh = 0.01, T = 3,000:

✅ Bisected to tolerance 0.005, the minimum d that passes is d* = 0.219 (ADF t -2.885, window 11 bars), and at d* the correlation with the level is 0.996. The book's claim reproduces: stationarity is bought at d ≈ 0.22 and costs 0.4 % of the correlation with the price level.

3. 🚨 What d = 1 costs, in the units that matter

One path cannot separate an information coefficient of 0.03 from zero (its standard error at T = 3,000 is 0.018), so the script repeats the scan over 50 independent seeded paths:

• ✅ The level's ADF rejects on 6 % of paths - the series genuinely is near-unit-root and the test says so at its nominal size. Nothing is rigged.
• ✅ At d = 0.35 the ADF rejects on 100 % of paths, and the feature keeps 97 % of the level's information coefficient (-0.0347 against -0.0359) and 0.982 correlation with it.
• 🚨 At d = 1 the feature keeps 3 % of it: mean IC -0.0010, which is 0.4 standard errors from zero. The correlation with the level is 0.035.

That is the trap in one line: first differencing passed the stationarity test and deleted the signal, and nothing in the pipeline said so. A model fitted on returns here has no edge to find; the same model on the d = 0.35 feature has essentially the whole edge, from a feature that also passes ADF on every path.

The decay is monotone in d across the whole panel, so d is not a hyperparameter to tune on a validation score — there is no interior optimum to find. Take the smallest value that clears the test.

4. 🚨 The weight threshold is a second, undocumented knob

thresh sets the window width, and the width sets how much level survives. Two people who both "used d = 0.35" have different features. ✅ Measured on seed 0:

🚨 The window moves by 165x across these four, the correlation with the level falls 0.985 -> 0.610 at the same d = 0.35, and the same series reports four different minimum d (0.219, 0.113, 0.113, 0.078). At thresh = 1e-5 only 1,175 of 3,000 bars survive the warm-up.

⚠️ The two thresholds in circulation come from the same file: mlfinpy's frac_diff_ffd defaults to thresh=1e-5, while plot_min_ffd — the routine that finds d — calls it with thresh=0.01. ✅ Source-verified in mlfinpy/util/frac_diff.py 0.1.2. Search for d and build the feature at the same threshold, and record it next to d.

5. Traps

• 🚨 d = 1 by reflex. Sections 2-3. pct_change() is the maximum-differencing corner of a continuum; measure what it costs before accepting it.
• 🚨 The window is warm-up, and it is longer than you think. The first width values are undefined. At thresh=1e-5, d=0.35 that is 1,825 bars of a 3,000-bar sample. Run the feature through ../../../fin-core/skills/signal-construction/scripts/warmup_probe.py; a run that starts at bar 0 with fillna(0) is trading a partial filter.
• 🚨 Fit the weights, not the data. The weights depend only on d and thresh, never on the sample, so the filter itself cannot leak. But choosing d on the full sample does: the ADF statistic that picked d* used the test period. Pick d on a prior window, or record the scan as trials in ../../../fin-core/skills/backtest-validation/scripts/trial_ledger.py — a 21-point d grid crossed with four thresholds is 84 trials.
• 🚨 Difference the log price, not the price. The recursion is linear, so applying it to raw prices makes the feature's scale proportional to the price level; a stock that doubles doubles its feature. Take logs first, as the book's own routine does (np.log(series[["close"]]) in plot_min_ffd).
• 🚨 ADF is not the only stationarity failure. The FFD series passes a unit-root test; it can still have a variance that changes by an order of magnitude across regimes, which breaks a model just as thoroughly. Pair the ADF with a rolling-variance plot, and see ../../../fin-core/skills/regime-detection/SKILL.md.
• ⚠️ This is one series' answer, not a constant. d* = 0.219 here, 0.35 for the book's E-mini example. Re-scan per instrument and per sample; a d copied from a paper is a guess.
• 🚨 The ADF table is the single-series one, and it is only valid because nothing was fitted first. If your feature is a regression residual instead, the table is wrong and over-rejects — ../../../fin-models/skills/stat-arb-cointegration/SKILL.md measures that (16.5 % rejection at a nominal 5 %).
• 🔴 pip install mlfinlab cannot work. ✅ Checked 2026-09-09: https://pypi.org/pypi/mlfinlab/json returns HTTP 404 and https://pypi.org/simple/mlfinlab/ returns a page with zero distribution links. The maintained fork that carries these chapters is mlfinpy 0.1.2 (MIT, released 2024-10-09, baobach/mlfinpy) — ⚠️ and it pins numpy<1.27, so it will not install beside numpy 2.x. That is why this skill transcribes the reference code instead of importing it.

6. Scripts

• scripts/frac_diff.py — the weight recursion and its closed form, the FFD truncation against mlfinpy's transcribed get_weights_ffd, the causal FFD filter, an ADF t-statistic checked against statsmodels, the single-path and 50-path d scans, the bisection for d*, and the threshold table. numpy + scipy; statsmodels is imported inside one function and the demo prints its own MacKinnon value without it. Seed 0, 3.7 s.

7. Where this sits

../triple-barrier-labeling/SKILL.md is the label side of the same chapter's pipeline; this skill is the feature side. ../structural-breaks/SKILL.md uses the same ADF machinery for explosiveness rather than stationarity. ../../../fin-models/skills/time-series-forecasting-models/SKILL.md owns unit-root testing and ARIMA order selection in general. ../../../fin-models/skills/stat-arb-cointegration/SKILL.md owns the ADF table for fitted residuals. ../../../fin-core/skills/signal-construction/SKILL.md owns assert_causal and warmup_probe, both of which apply to any FFD feature. ../../../fin-core/skills/backtest-validation/SKILL.md owns the trial count for the d scan. ../../../fin-libraries/skills/lib-purgedcv/SKILL.md owns the cross-validation once these features meet overlapping labels.