Trading Automation Runtime

Trading Automation Runtime

1. Overview + routing + ownership carve

This skill owns the process lifecycle and admission around an unattended trading loop — asset-class-agnostic. It is the part of a trading bot that decides whether the strategy is even allowed to run right now, survives kill -9 at any instant, and shuts down without stranding exposure. It is not the strategy, not the order router, and not the risk-threshold owner.

What this skill owns: the boot/lifecycle state machine, halt state that persists across restarts, single-active-instance fencing, the startup admission gate, the asset-neutral reconciliation protocol, the durable order-intent journal, calendar-driven scheduling, and the crash-consistent drain.

What it orchestrates but does NOT own (seam discipline — trip-wired):

• Risk thresholds and automatic-liquidation policy are trading-risk-management verdicts. This runtime persists and reloads halt state and reads the risk mode, but it never defines a threshold and never writes the risk halt cause (see trading-risk-management → KillSwitch state serialization).
• Broker/exchange queries belong to the adapters. This runtime defines the reconciliation protocol (§5) and calls each adapter's implementation — equity-broker-execution for equities, crypto-exchange-integration for crypto — and never implements a broker endpoint itself.
• Feed-health facts are the one measurement schema defined in market-data-engineering (the FeedHealthMeasurement schema, SEAM 1). This runtime aggregates the streams a strategy requires and applies admission policy; it does not re-derive the facts.
• Order placement / pricing / cancel-replace live in the execution adapters.
• Operator UI lives in trading-dashboard-ux; signal mapping / cardinality budgets live in observability (domain events and their meanings are owned here — high-cardinality order/symbol/account IDs go to logs and traces, never metric labels).

2. State model: orthogonal axes, not one FSM

A single boot FSM that mixes lifecycle with risk causes eventually renders a status that is a lie ("ACTIVE" while the account is liquidation-only). Model three independent axes and combine them through a most-restrictive permission lattice:

• lifecycle_phase (runtime-owned): BOOT → QUARANTINED → RECONCILING → ACTIVE, plus DRAINING and STOPPED.
• risk_mode (trading-risk-management-owned): NORMAL | HALTED | LIQUIDATION_ONLY. The runtime never writes this axis; each owner clears only its own halt cause.
• lease_authority (lease-system-owned): CURRENT | STALE | NONE.

Effective permissions are derived, never stored as a fourth truth:

def effective_permissions(lifecycle_phase: str, risk_mode: str, lease_authority: str):
    """Most-restrictive lattice over three orthogonal axes.

    NEW-EXPOSURE requires ACTIVE + NORMAL + CURRENT.
    cancel/reduce survives every restrictive lifecycle/risk state, but requires a
    CURRENT lease: a fenced-out (STALE/NONE) instance has NO mutation authority at all —
    the current leaseholder does the reducing. A state label must never trap a position,
    and a stale twin must never duplicate its exits.
    """
    can_new_exposure = (lifecycle_phase == "ACTIVE"
                        and risk_mode == "NORMAL"
                        and lease_authority == "CURRENT")
    can_cancel_reduce = lease_authority == "CURRENT"   # survives HALTED/DRAINING/LIQUIDATION_ONLY
    return {"increase": can_new_exposure,
            "cancel":   can_cancel_reduce,
            "reduce":   can_cancel_reduce}

The carve-out is binding in both directions: a halt/drain/liquidation state must not gate exits shut for the current leaseholder, and a fenced-out instance must not place "safety" exits (they are mutations — the current epoch owns them).

3. Persistent halt state with a safety contract

Halt state outlives the process. A restart can never clear a halt.

• Versioned, append-only records: (source, epoch, reason, asserted_at, cleared_at, cleared_by). History is never rewritten; an unhalt is a new record, not an edit.
• Compare-and-swap on epoch: a stale writer (an epoch at or below the highest already recorded for that source) cannot erase or supersede a newer halt.
• Single-writer-per-source; explicit, audited unhalt — who may clear which cause is defined, not implicit.
• Storage is a deployment choice, the pattern is the contract: atomic-replace + fsync for a local file; a consistent shared store (etcd/consul/a serializable DB row) for HA.
• Fail closed on absence. Missing, corrupt, conflicting, or UNAVAILABLE halt state means no new exposure while the fenced cancel/reduce path stays open.

def assert_halt(store, record, expected_epoch):
    """Append-only + CAS. A stale writer cannot erase or supersede a newer halt."""
    highest = store.highest_epoch(record["source"])
    if highest is not None and record["epoch"] <= highest:
        raise StaleWriterError(f"epoch {record['epoch']} <= current {highest}")
    store.append_cas(record, expected_epoch)          # atomic; loser of the race retries

def load_halt_state(store):
    """Missing/corrupt/conflicting/UNAVAILABLE halt state FAILS CLOSED for new exposure
    while preserving the fenced cancel/reduce path."""
    try:
        records = store.read_all()
    except StoreUnavailable:
        return {"risk_mode": "HALTED", "reason": "halt store unavailable — fail closed"}
    active = [r for r in records if r["cleared_at"] is None]
    if conflicting(active):
        return {"risk_mode": "HALTED", "reason": "conflicting halt records — fail closed"}
    return derive_mode(active)

trading-risk-management serializes its KillSwitch state into exactly this record shape (thin delta A) — no threshold logic moves; only a serializable projection crosses the seam.

4. Single-active-instance with real fencing

Two live instances sharing one broker account is the worst failure in this skill. Prevent it with fencing enforced at the resource boundary, not just a lease check.

A lease answers "who should be leader now"; a fencing token lets the protected resource reject work from a former leader. Checking the lease immediately before send() only moves the time-of-check/time-of-use window — the holder can pause (GC, VM stall), lose its lease, and resume after a new leader exists. The execution gateway (the component holding broker credentials) is where fencing must live, because brokers do not understand fencing tokens.

The precise idiom — activate before trading, then equality on every mutation. Merely retaining max_seen and rejecting lower tokens has a subtle gap: an old in-flight request can still be accepted before the boundary has seen the newer token. The new leader must persist the activation first:

def activate(gateway, scope: str, epoch: int):
    """New leader takes over: monotonic epoch, persisted BEFORE any trading."""
    if epoch <= gateway.active_epoch(scope):
        raise StaleLeaderError(epoch)
    gateway.persist_active_epoch(scope, epoch)        # durable; survives gateway restart
    gateway.purge_obsolete_command_queue(scope)       # drop lower-epoch queued commands

def mutate(gateway, scope: str, epoch: int, side: str, request):
    """EVERY order/cancel/replace/flatten carries the epoch and fails closed on mismatch."""
    if side.lower() not in ("buy", "sell", "buy_to_cover", "sell_short"):
        raise ValueError(f"invalid side: {side!r}")   # side is a validated input, never fixed
    if epoch != gateway.active_epoch(scope):          # equality, not >= — reject stale AND future
        raise FencedOutError(epoch, gateway.active_epoch(scope))
    return gateway.send(request)

Required properties: every order, cancel, replace, mass-cancel, flatten, and account mutation carries {scope, epoch, request_id} and fails closed on mismatch; epoch activation and sends are serialized; no strategy instance holds credentials or an alternate execution path; the active epoch is durable across gateway restart; idempotency / client-order keys handle commands already accepted before a takeover.

Transition rules:

• Current leader loses its lease → DRAINING for new exposure, keep the fenced cancel/reduce path.
• New leader → full RECONCILING (§5) before ACTIVE.
• Split-brain → highest epoch holds authority; the lower-epoch instance drops to lease_authority = STALE (no mutations, not even "safety" exits) and alerts.

Critical pitfall: if two gateway replicas can both use the broker credentials, the broker cannot atomically compare the epoch and the real resource is still unfenced. You need one serialized credential endpoint (or venue session-takeover / cancel-on-disconnect semantics strong enough to be that endpoint). FIX sequence numbers and cancel-on-disconnect are useful controls but are not general fencing tokens.

Primitives (2025-2026) — pick one that yields a monotonic token: etcd lock create_revision / election LeaderKey.rev (not the lease ID); ZooKeeper czxid / ephemeral-sequential suffix; Consul (Key, LockIndex, Session) sequencer. Redis Redlock and Postgres advisory locks do NOT return a monotonic fencing token — add an explicit generation counter. All such counters are scoped to a coordination-cluster incarnation; a snapshot rollback or cluster replacement can move them backward, so bind a (cluster_generation, counter) compound. (Fencing argument: Martin Kleppmann, "How to do distributed locking", 2016 — martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html.)

5. Startup admission gate + reconciliation protocol (SEAM 2)

Nothing trades until the gate passes. This runtime defines one versioned, asset-neutral protocol; each adapter implements it.

ReconciliationRequest  -> account/venue scope, snapshot watermark, what to fetch
ReconciliationResult   -> open orders, executions/fills (composite keys where needed),
                          positions, balances (where applicable), discrepancy list,
                          pagination completeness, and status: complete | incomplete | unknown

An incomplete or unknown result blocks new exposure — admission fails to QUARANTINED, it is never silently treated as "no open state". Each adapter owns retrieval/normalization: equity-broker-execution names its §4 invariants as the equity implementation (thin delta B); crypto-exchange-integration adds the crypto surface with per-exchange capability checks (thin delta C — CCXT capabilities vary, so incomplete is venue-specific).

The gate runs, in order, before the first strategy callback:

1. Reload halt/risk state (§3) — a persisted halt keeps the bot out of ACTIVE.
2. Reconcile open orders / fills / positions / balances via the protocol above.
3. Resolve the order-intent journal (§6) — no unresolved PREPARED/UNKNOWN intents.
4. Revalidate the compliance snapshot — boot is a session start under equity-trading-compliance §2 revalidation triggers.
5. Evaluate feed health — aggregate the FeedHealthMeasurement facts (market-data-engineering) over the strategy's required streams against fail-closed admission thresholds.

All pass → ACTIVE. Any fail → QUARANTINED + alert. Admission is fail-closed by construction: an unknown answer is a failing answer.

6. Durable order-intent journal (adapter-owned; the runtime consumes it at boot)

This is what makes "compare scheduled intent vs broker reality" possible after kill -9 mid-submit. The submission path writes intent before sending and records the response after:

def record_intent(journal, intent_id, client_order_key, digest, side: str, qty: float):
    """Write PREPARED intent BEFORE the send. `side` is a validated input, never hardcoded."""
    if side.lower() not in ("buy", "sell", "buy_to_cover", "sell_short"):
        raise ValueError(f"invalid side: {side!r}")
    journal.append({"intent_id": intent_id, "client_order_key": client_order_key,
                    "digest": digest, "side": side, "qty": qty, "state": "PREPARED"})

def resolve_intents_at_boot(journal, adapter):
    """Unresolved PREPARED intents become UNKNOWN and are resolved by adapter query —
    NEVER blindly retried (a blind retry doubles exposure)."""
    for intent in journal.where(state="PREPARED"):
        intent["state"] = "UNKNOWN"
        found = (adapter.query_by_client_key(intent["client_order_key"])
                 or adapter.query_by_execution_key(intent["digest"]))
        if found is None:
            journal.mark_ambiguous(intent["intent_id"])   # stay QUARANTINED, manual resolution
        else:
            journal.reconcile(intent["intent_id"], found)

Unresolvable ambiguity keeps the account in QUARANTINED pending manual resolution — the runtime never guesses whether an order landed.

7. Scheduling + session model

Consume versioned venue calendars / capability snapshots — never a hardcoded pre / regular / post triple. Overnight and venue-specific sessions exist and are expanding in 2026, and named sessions differ per venue.

• The snapshot carries: per-venue timezone and trade-date/clearing rules, named sessions (stable IDs, open/close, auction and order-entry windows), maintenance windows, protection-regime coverage per session (LULD, venue collars, halts, clearly-erroneous), holiday/early-close/DST/cross-midnight overrides, and effective dates.
• Misfire policy: skip vs run-late vs quarantine, with idempotent run keys.
• Unknown session / protection facts → fail closed for new exposure.

2026 landscape (verify against the live calendar before shipping): LULD remains regular-hours only (09:30–16:00 ET). Overnight ATS liquidity is live (e.g. Blue Ocean ATS, ~20:00–04:00 ET); national-exchange overnight sessions (Nasdaq Day/Night, NYSE Arca's four-session model, Cboe EDGX) are SEC-approved with launches targeted around 2026-12-06, so "24×5" is in practice 23×5 with a daily maintenance break. An overnight price-band proposal — SEC Release No. 34-105596 (File No. 4-631) — is a Notice of Filing (not an approval) as of 2026-07-14; if adopted, Phase 1 would publish static overnight bands for 21:00–04:00 ET only, not the full daytime sliding-band/limit-state mechanism. Treat overnight band protection as absent until a venue's snapshot says otherwise. The order-time detail lives in equity-broker-execution §7 (including the unattended extended-hours prohibition when band/halt state is unavailable) — reference it, do not restate the rule model here.

8. Graceful shutdown / crash-consistent drain

The drain is a persisted lifecycle phase, not a best-effort sequence. Persist first, then act — so a crash mid-drain is just the boot path.

def drain(runtime, adapter, deadline):
    """Persist DRAINING BEFORE acting. Boot detects an unfinished drain and resumes it
    before normal admission. kill -9 at ANY step is recovered by the boot path."""
    runtime.persist_lifecycle("DRAINING")             # 1. new exposure disabled, persisted first
    adapter.reconcile()                               # 2. broker-authoritative state
    adapter.cancel_exposure_increasing_orders()       # 3. adapter-classified; exits untouched
    adapter.wait_for_terminal_acks(deadline)          # 4. BOUNDED wait for acknowledgements
    adapter.reconcile_fills_during_cancel_race()      # 5. fills that landed during the cancel
    apply_protective_exit_policy(adapter)             # 6. see rule below
    runtime.checkpoint_protective_exits()             # 7. re-adopt at next boot
    runtime.persist_lifecycle("STOPPED")              # 8. final checkpoint

Protective-exit policy (binding): broker-side protective stops stay active and are checkpointed for re-adoption at next boot — never cancel exits and leave a naked position. Client-side-synthesized exits are converted to broker-side orders (or the position is reduced) before the process exits. Define behavior when the supervisor's termination deadline expires mid-drain — a hard SIGKILL at the deadline must still leave the persisted DRAINING state and live protective exits, so boot resumes safely.

9. Test matrix

Each scenario asserts a terminal state and effective permissions, driven by fault injection (the design test is: kill -9 recovery at any point is just the boot path).

Readiness vs liveness: liveness = the process is up; readiness = the admission gate passed. A live-but-not-ready bot must never receive strategy callbacks.

10. Anti-Patterns

Thin deltas that complete this skill's seams: trading-risk-management (KillSwitch → halt-record serialization; risk_mode axis ownership) · equity-broker-execution (§4 invariants as the equity reconciliation implementation; boot call site) · crypto-exchange-integration (per-exchange capability checks, composite execution keys, pagination completeness).

See also: market-data-engineering (the FeedHealthMeasurement schema, SEAM 1) · trading-dashboard-ux (the operator console that displays these axes) · observability (signal mapping / cardinality) · equity-trading-compliance (boot compliance revalidation).