Batch Vs Realtime Scoring

Batch vs Realtime Scoring (Verbose)

Core Patterns

The Decision Framework

The choice is not about scale or sophistication. It is about whether the model's inputs can change between the moment you would precompute and the moment the prediction is used.

Worked examples: churn propensity — known user set, features move over days, consumed by an email campaign — batch, decisively. Card-present fraud — the transaction does not exist until it happens and the decision blocks a payment — realtime, no alternative. Search ranking — the query is unknown but the corpus is known — hybrid: precomputed item embeddings, realtime query scoring.

A common and expensive error is choosing realtime because the model is "for a user-facing product", when the score is read from a nightly-refreshed profile.

Batch Scoring

def nightly_scoring(snapshot_date: str) -> None:
    ids, features = load_active_entities(snapshot_date)     # point-in-time features
    scores = model.predict_proba(features)[:, 1]

    with redis.pipeline() as p:
        for uid, s in zip(ids, scores):
            p.setex(f"churn:{uid}", 60 * 60 * 36, f"{s:.6f}")   # 36h TTL > 24h cadence
        p.execute()

    warehouse.write(f"scores/churn/dt={snapshot_date}/",       # history, not just cache
                    pd.DataFrame({"entity_id": ids, "score": scores,
                                  "model_version": MODEL_VERSION}))

Three details separate a working batch system from a fragile one:

• TTL longer than the cadence. Without it, one failed run empties the store. With it, consumers read a slightly stale score and nothing breaks.
• Write to a warehouse table as well as the cache. Monitoring, backfills, and "what did we say about this account in March" all need the history.
• Handle the cold entity. A user created after the last run has no score. Use an explicit default — the base rate or a cheap heuristic — rather than a null that a downstream service interprets as zero.

score = redis.get(f"churn:{uid}")
if score is None:
    metrics.increment("score.miss")
    score = POPULATION_BASE_RATE          # explicit default, not None

Cost is the underrated advantage: a nightly spot-instance job scoring ten million entities costs a few dollars, while the equivalent always-on realtime tier costs orders of magnitude more and needs on-call coverage.

Realtime Scoring and the Latency Budget

Write the budget down before designing anything, at p99 — averages hide exactly the tail that times out.

p99 end-to-end budget            50 ms
  ingress + TLS + serialization   8 ms
  feature retrieval              20 ms   <- the design problem
  preprocessing + inference      10 ms
  business logic + response       5 ms
  headroom                        7 ms

Inference is rarely the bottleneck — a gradient-boosted tree scores a row in microseconds, a small neural net in a few milliseconds on CPU. Feature retrieval is where the budget goes.

The consequence: realtime scoring forces feature precomputation. A feature like orders_30d requires scanning a month of events — a multi-second query that no index tuning fits into 20 ms. The aggregation must already exist when the request arrives.

A stream processor (Flink, Spark Structured Streaming) maintains the aggregation in an online store; the request path does a single-digit-millisecond point lookup. Split every feature by volatility:

The request-time tier must be pure arithmetic on the request payload; the moment it needs I/O, it belongs in another tier.

Retrieve in one round trip, and always bound the wait:

# ❌ N round trips: 12 features × 3 ms = 36 ms, blowing the budget
feats = {name: redis.get(f"{name}:{uid}") for name in FEATURE_NAMES}

# ✅ One round trip, with a timeout
try:
    values = await asyncio.wait_for(
        store.mget([f"{n}:{uid}" for n in FEATURE_NAMES]), timeout=0.020)
    feats = dict(zip(FEATURE_NAMES, values))
except asyncio.TimeoutError:
    metrics.increment("feature_store.timeout")
    feats = DEFAULT_FEATURES              # degrade, never hang the request

A model that cannot answer within budget is worse than a heuristic that can. Every realtime scoring path needs a defined fallback — a cached previous score, a simple rule, or a fixed default — chosen deliberately and load-tested.

The Hybrid Pattern

Most large systems precompute the expensive stage offline and run a cheap, context-aware stage inline.

def recommend(user_id: str, session: Session, k: int = 20) -> list[Item]:
    candidates = redis.lrange(f"cands:{user_id}", 0, 499)   # batch, ~2 ms lookup
    ctx = {"items_viewed": session.viewed, "dwell": session.dwell_ms,
           "hour": datetime.now(timezone.utc).hour}
    return reranker.top_k(candidates, ctx, k=k)             # realtime, ~15 ms

The economics work because retrieval over millions of items is expensive and tolerates staleness, while re-ranking a few hundred is cheap and needs freshness. Cost per prediction should be compared across options before committing.

A second hybrid shape is precompute-then-adjust: serve a nightly batch score, then apply realtime overrides for a few fast-moving signals (a fraud flag raised an hour ago, a subscription cancelled this morning).

Consistency Between Paths

The moment a system has both a batch and a realtime path, it has two implementations of the same feature, and they will diverge — the training/serving skew described in the ML deployment skill, arriving through a different door.

Share the definition, but be aware that shared code does not guarantee shared semantics. Three divergences are worth checking explicitly:

• Window boundaries. Batch computes "last 30 days as of midnight UTC"; streaming computes "trailing 30 days from now". At 23:00 those differ by a day.
• Late-arriving events. The batch job sees an event that landed after the streaming watermark passed. The two counts will not match, permanently.
• Null handling. A missing streaming key returns None; the batch job's join produced 0. The model treats those very differently.

Reconcile continuously: sample entity ids, fetch both the online and the offline value, and publish the p99 absolute delta as a gauge. A gap that grows is a bug you would otherwise find during an incident.

Common Anti-Patterns

❌ Choosing realtime because the product is user-facing. ✅ Choose by input freshness. A user-facing page reading a daily-refreshed profile wants batch.

❌ Computing aggregations with a query in the request path. ✅ Precompute into an online store; the request path does point lookups only.

❌ Stating the latency budget as an average. ✅ Budget at p99 — the tail is what times out and what users remember.

❌ Unbounded waits on the feature store, or no fallback when the model is down. ✅ Per-call timeouts plus a load-tested default — cached score, heuristic, or base rate.

❌ Batch writes without a TTL longer than the run interval, or nulls for entities never scored. ✅ TTL exceeding the cadence, plus an explicit cold-start default and a miss-rate metric.

❌ Separate batch and streaming implementations of the same feature. ✅ Shared definitions, plus a running reconciliation job that measures the gap.

Scoring Architecture Checklist

• Required input freshness stated explicitly before choosing an architecture
• Batch chosen wherever the entity set is enumerable and staleness is tolerable
• Latency budget written at p99 and decomposed by stage
• Features tiered by volatility: static, streaming, request-time
• All non-trivial aggregations precomputed into an online store, retrieved in a single batched round trip
• Timeouts on every external call, with a load-tested fallback prediction
• Batch writes carry a TTL longer than the run interval
• Cold-start default defined; store miss rate monitored
• Scores persisted to a warehouse as well as the serving cache
• Batch and streaming feature definitions shared, with running reconciliation of values, window boundaries, and late-event semantics