selamy-labs Adaptive Estimation
When you are tracking a quantity you cannot observe directly — a true value that drifts slowly while your measurements are noisy — a fixed-window moving average and a static anomaly threshold are both crude. The window is either too short (jittery) or too long (laggy), and a static threshold cannot tell a real shift from ordinary noise. A Kalman-style recursive estimator solves both: it auto-tunes how much to trust each new measurement, and its prediction error is a built-in surprise detector.
When this applies (scope it tightly)
Use this only when the preconditions hold; otherwise it is over-engineering:
- there is a single latent quantity (or a small state vector) that evolves slowly and roughly linearly between steps
- each measurement is noisy but you can put a number on the uncertainty of both the measurement and the process drift (variances, even rough ones)
- the noise is approximately Gaussian / unimodal — no heavy multi-modal structure
- you want a continuously-updated estimate of the current value plus a measure of how confident you are in it
This is an estimation layer: it smooths and tracks a value and flags surprises. It is not a forecaster of structural regime change, not a way to manufacture predictive signal where none exists, and not a substitute for a real model of the system. It cleans and tracks; it does not divine.
The recursive update (1-D scalar form)
Maintain an estimate x and its variance P. Each step:
Predict (let the model drift; uncertainty grows by process noise Q):
x⁻ = x(orx = F·xif there is known drift dynamics)P⁻ = P + Q
Update with a new measurement z of measurement-noise variance R:
- innovation (residual):
y = z − x⁻ - gain:
K = P⁻ / (P⁻ + R) - new estimate:
x = x⁻ + K·y - new variance:
P = (1 − K)·P⁻
K lives in [0, 1] and auto-tunes trust: when measurements are noisy
relative to the model (R ≫ P⁻), K → 0 and the estimate barely moves; when
the model is uncertain relative to a clean measurement (P⁻ ≫ R), K → 1 and
the estimate snaps to the observation. You set Q and R; the gain adapts on
its own. The vector form generalizes this with matrices F, Q, H, R and the
same predict/update structure.
The innovation is a free anomaly detector
The residual y has expected variance S = P⁻ + R. A normalized innovation
y / √S is, under the model's assumptions, roughly unit-variance. So:
- a normalized innovation beyond a few standard deviations = a statistically surprising measurement — a principled anomaly flag, adaptive to current uncertainty rather than a hand-set fixed threshold
- persistently biased innovations (a run of same-sign residuals) means the model
is wrong — your
Q/Rare mistuned or the dynamics are not what you assumed
You get smoothing and anomaly detection from the same recursion, for free.
When to escalate (and when not to)
- Genuinely non-linear dynamics or measurement function → Extended (EKF) or Unscented (UKF) Kalman filter. Reach for these only when a real non-linearity forces it — they cost complexity and tuning.
- Multi-modal / non-Gaussian state → particle filter. Heavier still.
- Do not jump to EKF/UKF/particle filters by default. The linear scalar filter above handles a surprising share of "track a drifting noisy number" problems; escalate only on evidence (biased innovations, known non-linearity).
Anti-patterns
- a fixed-window moving average that is simultaneously too laggy and too jittery because one window cannot serve both
- a static anomaly threshold that ignores how confident you currently are
- treating this estimation layer as if it predicts future structural change
- inventing
Q/Rand never checking the innovations to see if the model holds - reaching for an EKF/UKF/particle filter before establishing that the simple linear-Gaussian filter is actually insufficient
Done means
A slowly-varying latent value is tracked with a recursive estimator whose gain
auto-tunes trust between model and measurement using stated Q/R; the
normalized innovation is used as an adaptive surprise/anomaly signal; the model's
fit is sanity-checked via the innovation sequence; and any escalation to a
non-linear filter is justified by evidence, not chosen by default.