# Subsystem Summary Of Catchup > read this skill for a token-efficient summary of the catchup subsystem - Skill: `stellar/subsystem-summary-of-catchup` (Agent Skill) - Install (CLI): `npx skillmds@latest add stellar/subsystem-summary-of-catchup` - Raw SKILL.md: https://api.skillmd.com/api/skills/stellar/subsystem-summary-of-catchup/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: stellar (https://skillmd.com/u/stellar) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/stellar/subsystem-summary-of-catchup --- # Catchup Subsystem — Technical Summary The catchup subsystem in stellar-core is responsible for synchronizing a node's local ledger state with the rest of the network when it falls behind. It downloads historical data (ledger headers, transactions, and bucket snapshots) from history archives, verifies integrity, and applies the data to bring the node up to date. All files reside in `src/catchup/`. --- ## Key Classes and Data Structures ### CatchupConfiguration **File:** `CatchupConfiguration.h/.cpp` Immutable configuration describing a catchup request. Parameterized by: - `toLedger` — destination ledger number (or `CURRENT = 0` to resolve at runtime from the archive). - `count` — number of ledgers to replay before the destination. `0` = minimal (buckets only), `UINT32_MAX` = complete history. - `Mode` — one of `OFFLINE_BASIC`, `OFFLINE_COMPLETE`, or `ONLINE`. Key methods: - `resolve(uint32_t remoteCheckpoint)` — substitutes `CURRENT` with an actual checkpoint ledger number. - `offline()` / `online()` — predicates for mode. Helper free functions `parseLedger()` and `parseLedgerCount()` parse CLI strings. ### CatchupRange **File:** `CatchupRange.h/.cpp` Computed from `CatchupConfiguration` + the current LCL + `HistoryManager`. Decides **what** the catchup must do: | Field | Meaning | |---|---| | `mApplyBuckets` | Whether a bucket-apply phase is needed. | | `mApplyBucketsAtLedger` | Checkpoint ledger at which to apply buckets (0 if !mApplyBuckets). | | `mReplayRange` (LedgerRange) | Half-open range of ledgers to replay after bucket-apply. | Five logical cases based on LCL position, requested count, and checkpoint boundaries (see comments in header). Invariants enforced by `checkInvariants()`. Key accessors: `applyBuckets()`, `replayLedgers()`, `getBucketApplyLedger()`, `getReplayRange()`, `getFullRangeIncludingBucketApply()`. ### LedgerApplyManager / LedgerApplyManagerImpl **Files:** `LedgerApplyManager.h`, `LedgerApplyManagerImpl.h/.cpp` Abstract interface + concrete implementation. This is the **top-level coordinator** between the consensus layer (Herder) and the catchup/apply machinery. Owned by `Application`. #### Key data members (Impl): - `mCatchupWork` — `shared_ptr`, the running catchup work item (null when not catching up). - `mSyncingLedgers` — `map`, buffer of ledgers received from the network that cannot be applied yet. Has strict invariants: either empty, starts at LCL+1, or contains at most 65 ledgers within a checkpoint boundary. - `mLastQueuedToApply` — tracks the highest ledger sequence queued for application. - `mLargestLedgerSeqHeard` — the highest ledger seq ever received. - `mMetrics` (`CatchupMetrics`) — counters for archive states downloaded, checkpoints, ledgers verified, buckets downloaded/applied, tx sets downloaded/applied. - `mCatchupFatalFailure` — set when catchup fails unrecoverably (e.g., incompatible core version). - `MAX_EXTERNALIZE_LEDGER_APPLY_DRIFT = 12` — maximum ledger drift allowed before entering catchup in parallel-close mode. #### Key methods: - **`processLedger(LedgerCloseData, isLatestSlot)`** — main entry point called by Herder/LedgerManager when a new consensus ledger arrives. Logic: 1. If catchup is done, resets `mCatchupWork`. 2. If ledger is old (≤ mLastQueuedToApply), skip. 3. If ledger is the next sequential one and no catchup running → `tryApplySyncingLedgers()`. 4. Otherwise buffers the ledger, trims the buffer, and decides whether to `startOnlineCatchup()`. - **`startCatchup(CatchupConfiguration, archive)`** — schedules a `CatchupWork` on the `WorkScheduler`. - **`startOnlineCatchup()`** — constructs a `CatchupConfiguration` targeting `firstBuffered - 1` in ONLINE mode. - **`trimSyncingLedgers()`** — garbage-collects old entries from `mSyncingLedgers`, keeping at most one checkpoint's worth plus one. - **`tryApplySyncingLedgers()`** — iterates sequential ledgers in `mSyncingLedgers` and applies them via `LedgerManager::applyLedger()`. In parallel-close mode, posts work to the ledger-close thread. - **`maybeGetNextBufferedLedgerToApply()`** — returns the next buffered ledger (LCL+1) if available; used by `ApplyBufferedLedgersWork`. ### CatchupWork **File:** `CatchupWork.h/.cpp` The central **Work** subclass orchestrating all catchup steps. Extends `Work` (composite work pattern). #### Key data members: - `mLocalState` (HistoryArchiveState) — local BucketList state at catchup start. - `mDownloadDir` (unique_ptr) — temporary directory for downloaded files. - `mLiveBuckets`, `mHotBuckets` — maps from hash → downloaded Bucket objects. - `mCatchupConfiguration` — the resolved configuration. - `mGetHistoryArchiveStateWork`, `mGetBucketStateWork` — work to fetch HAS from archive. - `mDownloadVerifyLedgersSeq` — work sequence for downloading + verifying ledger headers. - `mVerifyLedgers` (VerifyLedgerChainWork) — verifies ledger chain integrity. - `mBucketVerifyApplySeq` — work sequence for downloading, verifying, and applying buckets. - `mTransactionsVerifyApplySeq` (DownloadApplyTxsWork) — work for downloading and applying transactions. - `mApplyBufferedLedgersWork` — applies buffered network ledgers after catchup replay. - `mCatchupSeq` — final composite work sequence. - `mVerifiedLedgerRangeStart` (LedgerHeaderHistoryEntry) — the verified ledger at the start of the catchup range (used for bucket-apply). - `mFatalFailureFuture` — shared_future indicating unrecoverable failure. #### Key control flow (`runCatchupStep()` / `doWork()`): 1. **Get HAS** — `getAndMaybeSetHistoryArchiveState()` fetches the remote history archive state, validates network passphrase, checks that target > LCL. 2. **Resolve CatchupRange** — from config + HAS + LCL. 3. **Get bucket HAS** — `getAndMaybeSetBucketHistoryArchiveState()` if bucket-apply is needed and the bucket HAS differs from the main HAS. 4. **Download & verify ledger chain** — `downloadVerifyLedgerChain()` spawns `BatchDownloadWork` + `VerifyLedgerChainWork` in a `WorkSequence`. 5. **Build catchup sequence** — after ledger verification succeeds: - If `applyBuckets()`: `downloadApplyBuckets()` → `DownloadBucketsWork` + `ApplyBucketsWork`. - If `replayLedgers()`: `downloadApplyTransactions()` → `DownloadApplyTxsWork`. - A Herder consistency check work is prepended. 6. **Bucket-apply completion** — calls `LedgerManager::setLastClosedLedger()` with the verified state, clears rebuild flags. 7. **Apply buffered ledgers** — after the main catchup sequence succeeds, `ApplyBufferedLedgersWork` drains `mSyncingLedgers`. Constants: `PUBLISH_QUEUE_UNBLOCK_APPLICATION = 8`, `PUBLISH_QUEUE_MAX_SIZE = 16` — flow-control the publish queue during catchup. ### VerifyLedgerChainWork **File:** `VerifyLedgerChainWork.h/.cpp` `BasicWork` subclass that verifies a range of downloaded ledger header files. Processes checkpoints from **highest to lowest**, linking each checkpoint's hash chain to the next. #### Key data members: - `mDownloadDir`, `mRange`, `mCurrCheckpoint` — the files to verify and current position. - `mLastClosed` (LedgerNumHashPair) — local LCL for consistency checks. - `mTrustedMaxLedger` (shared_future) — trusted hash from SCP consensus for the range end. - `mVerifiedAhead` (LedgerNumHashPair) — hash-link propagation between checkpoint verifications. - `mVerifiedMinLedgerPrev` (promise) — outgoing: the hash just before the verified range, so bucket-apply can validate. - `mMaxVerifiedLedgerOfMinCheckpoint` — the max ledger of the lowest checkpoint; used by CatchupWork as `mVerifiedLedgerRangeStart`. - `mFatalFailurePromise` — set when a mismatch against trusted hash is detected. - `mChainDisagreesWithLocalState` — records local-state disagreements (e.g., bad LCL hash, incompatible version). #### Key method — `verifyHistoryOfSingleCheckpoint()`: - Opens the checkpoint ledger header file. - Iterates entries, verifying each ledger header hash and link to the previous. - At the range end, verifies against `mTrustedMaxLedger`. - At each checkpoint boundary, checks hash-chain linkage with `mVerifiedAhead`. - On the lowest checkpoint, writes hash-link to `mVerifiedMinLedgerPrev` and records `mMaxVerifiedLedgerOfMinCheckpoint`. - Checks local state (LCL hash, protocol version) and records disagreements. #### `onRun()`: Calls `verifyHistoryOfSingleCheckpoint()` once per crank. On success, decrements `mCurrCheckpoint` and returns `WORK_RUNNING` until all checkpoints are verified. Maps various error statuses to `WORK_FAILURE` with appropriate log messages. ### DownloadApplyTxsWork **File:** `DownloadApplyTxsWork.h/.cpp` `BatchWork` subclass that iterates over checkpoints in a replay range, yielding a work sequence per checkpoint: download → unzip → apply. #### Key data members: - `mRange` (LedgerRange) — the half-open replay range. - `mDownloadDir` — shared temp directory. - `mLastApplied` (LedgerHeaderHistoryEntry&) — reference to the last applied header (updated on success). - `mCheckpointToQueue` — next checkpoint to schedule. - `mLastYieldedWork` — the previous checkpoint's work, used for sequencing. - `mWaitForPublish` — if true, gates application on publish queue size. #### `yieldMoreWork()`: For each checkpoint: 1. Creates `GetAndUnzipRemoteFileWork` for the transaction file. 2. Creates `ApplyCheckpointWork` for the ledger range within that checkpoint. 3. Wraps application in a `ConditionalWork` that: - Waits for the previous checkpoint's work to finish. - Optionally waits for the publish queue to drain below `PUBLISH_QUEUE_MAX_SIZE`. - Optionally waits for BucketList merges. 4. Appends cleanup work to delete temporary files. 5. Returns the whole sequence as a `WorkSequence`. ### ApplyCheckpointWork **File:** `ApplyCheckpointWork.h/.cpp` `BasicWork` subclass that applies transactions from a single checkpoint (at most one checkpoint worth of ledgers). #### Key data members: - `mDownloadDir` — temp dir with ledger + tx files. - `mLedgerRange` — the aligned ledger range to apply. - `mCheckpoint` — the checkpoint number. - `mHdrIn`, `mTxIn` — XDR input streams for ledger headers and transactions. - `mConditionalWork` — wraps `ApplyLedgerWork` in a conditional that waits for BucketList merge futures to resolve. #### Key control flow (`onRun()`): 1. If a conditional work is active, cranks it. On success, verifies the resulting LCL hash matches the expected header hash. 2. Checks if done (all ledgers in range applied). 3. Opens input files if needed. 4. Calls `getNextLedgerCloseData()` which reads the next header from file, performs knitting checks (skip old, verify LCL hash continuity, verify tx set hash), and constructs a `LedgerCloseData`. 5. Creates `ApplyLedgerWork` wrapped in a `ConditionalWork` that waits for BucketList merge futures. ### ApplyLedgerWork **File:** `ApplyLedgerWork.h/.cpp` Minimal `BasicWork` subclass. `onRun()` calls `LedgerManager::applyLedger(lcd, false)` to close a single ledger. No retry. ### ApplyBucketsWork **File:** `ApplyBucketsWork.h/.cpp` `Work` subclass that applies bucket snapshot state to the database. #### Key data members: - `mBuckets` — map of hash → LiveBucket (downloaded buckets). - `mApplyState` (HistoryArchiveState) — the archive state to apply. - `mBucketsToApply` — ordered vector of buckets (L0 curr, L0 snap, L1 curr, ...). - `mBucketApplicator` — the active `BucketApplicator` instance. - `mSeenKeys`, `mSeenKeysBeforeApply` — deduplication sets to ensure only the newest version of each entry is written. - `mIndexBucketsWork` — child work to index bucket files (runs first). - `mAssumeStateWork` — child work to assume BucketList state (runs after all buckets applied). #### Key control flow (`doWork()`): 1. **Index buckets** — spawns `IndexBucketsWork` on first call. 2. **Apply buckets** — iterates through `mBucketsToApply` in order, using `BucketApplicator` to incrementally write entries to the database. Entries already in `mSeenKeys` are skipped (ensures newest-version-wins). After each bucket, runs invariant checks. 3. **Assume state** — spawns `AssumeStateWork` which indexes both live and hot archive buckets, then calls `BucketManager::assumeState()` to set the BucketList to the target state and restart merges. ### AssumeStateWork **File:** `AssumeStateWork.h/.cpp` `Work` subclass spawned at the end of `ApplyBucketsWork`. Holds strong references to all buckets in the HAS (including future buckets from pending merges) to prevent garbage collection during indexing. #### `doWork()`: 1. Spawns `IndexBucketsWork` and `IndexBucketsWork`. 2. Spawns a callback work that calls `BucketManager::assumeState()` and `InvariantManager::checkAfterAssumeState()`. 3. Returns `checkChildrenStatus()`. ### IndexBucketsWork **File:** `IndexBucketsWork.h/.cpp` Template `Work` subclass that indexes bucket files in parallel. For each non-empty, non-indexed bucket, spawns an `IndexWork` child. #### IndexWork (inner class): - Posts indexing to a background thread via `postOnBackgroundThread`. - Tries to load a persisted index file first; if corrupt or outdated, creates a fresh index via `createIndex()`. - On completion, posts result back to main thread and calls `BucketManager::maybeSetIndex()`. ### ApplyBufferedLedgersWork **File:** `ApplyBufferedLedgersWork.h/.cpp` `BasicWork` subclass used at the end of catchup to drain `mSyncingLedgers`. On each `onRun()`: 1. Checks if previous `ConditionalWork` is done. 2. Asks `LedgerApplyManager::maybeGetNextBufferedLedgerToApply()` for the next ledger. 3. Wraps `ApplyLedgerWork` in a `ConditionalWork` that waits for BucketList merge futures. 4. Returns `WORK_SUCCESS` when no more buffered ledgers available. ### ReplayDebugMetaWork **File:** `ReplayDebugMetaWork.h/.cpp` `Work` subclass for offline replay of debug meta files (used in diagnostic scenarios). Iterates sorted debug meta files, optionally gunzips them, and spawns `ApplyLedgersFromMetaWork` (inner helper class) to read `LedgerCloseMeta` entries and apply them via `ApplyLedgerWork`. Can also apply a final `StoredDebugTransactionSet` for the latest ledger. --- ## Key Data Flows ### Online Catchup Flow ``` Herder (consensus) │ ▼ LedgerApplyManagerImpl::processLedger() │ ├─ If sequential with LCL → tryApplySyncingLedgers() → LedgerManager::applyLedger() │ └─ If behind → buffer in mSyncingLedgers │ └─ When checkpoint boundary reached → startOnlineCatchup() │ ▼ CatchupWork (scheduled on WorkScheduler) │ ├─ 1. GetHistoryArchiveStateWork → fetch remote HAS ├─ 2. Compute CatchupRange ├─ 3. downloadVerifyLedgerChain() │ ├─ BatchDownloadWork (ledger header files) │ └─ VerifyLedgerChainWork (hash-chain verification, highest→lowest) ├─ 4a. downloadApplyBuckets() [if applyBuckets()] │ ├─ DownloadBucketsWork │ ├─ verify HAS │ └─ ApplyBucketsWork │ ├─ IndexBucketsWork │ ├─ BucketApplicator (per bucket, level by level) │ └─ AssumeStateWork ├─ 4b. downloadApplyTransactions() [if replayLedgers()] │ └─ DownloadApplyTxsWork (per checkpoint) │ ├─ GetAndUnzipRemoteFileWork │ └─ ApplyCheckpointWork │ └─ ApplyLedgerWork (per ledger) └─ 5. ApplyBufferedLedgersWork → drain mSyncingLedgers ``` ### Offline Catchup Flow Same as online but triggered by `startCatchup()` directly (not by buffered ledgers), mode is `OFFLINE_BASIC` or `OFFLINE_COMPLETE`, no `ApplyBufferedLedgersWork`, and in `OFFLINE_COMPLETE` mode, `DownloadVerifyTxResultsWork` is also run for full validation. --- ## Threading Model - All `LedgerApplyManagerImpl` methods assert `threadIsMain()` — the catchup coordinator runs entirely on the main thread. - The `Work` / `BasicWork` framework is cranked on the main thread's event loop. - `IndexBucketsWork::IndexWork` posts indexing tasks to a background thread pool via `postOnBackgroundThread()`, and posts results back to the main thread via `postOnMainThread()`. - In parallel-close mode (`parallelLedgerClose()`), `tryApplySyncingLedgers()` posts `applyLedger` calls to the ledger-close thread. - `ApplyCheckpointWork` and `ApplyBufferedLedgersWork` use `ConditionalWork` to poll for BucketList merge future resolution before applying ledgers, preventing application while background merges are pending. - `VerifyLedgerChainWork` uses `std::promise` / `std::shared_future` for inter-work communication: the trusted max-ledger hash is passed in via `shared_future`, and the verified min-ledger-prev hash is passed out via `promise`. --- ## Ownership Relationships ``` Application └─ LedgerApplyManagerImpl (unique_ptr, via LedgerApplyManager::create) └─ mCatchupWork: shared_ptr (owned while catchup active) ├─ mDownloadDir: unique_ptr ├─ mLiveBuckets / mHotBuckets: map> ├─ mGetHistoryArchiveStateWork: shared_ptr ├─ mDownloadVerifyLedgersSeq: shared_ptr │ └─ mVerifyLedgers: shared_ptr ├─ mBucketVerifyApplySeq: shared_ptr │ └─ ApplyBucketsWork │ ├─ mIndexBucketsWork: shared_ptr │ │ └─ IndexWork children (per bucket, background thread) │ ├─ mBucketApplicator: unique_ptr │ └─ mAssumeStateWork: shared_ptr ├─ mTransactionsVerifyApplySeq: shared_ptr │ └─ per-checkpoint WorkSequence children │ ├─ GetAndUnzipRemoteFileWork │ └─ ApplyCheckpointWork │ └─ mConditionalWork → ApplyLedgerWork ├─ mApplyBufferedLedgersWork: shared_ptr └─ mCatchupSeq: shared_ptr (final composite) ``` `LedgerApplyManagerImpl` also owns `mSyncingLedgers` (the ledger buffer) independently of `CatchupWork`. --- ## Key Invariants and Error Handling - `CatchupRange::checkInvariants()` ensures at least one of bucket-apply or replay is active, and validates sequencing between them. - Hash-chain verification in `VerifyLedgerChainWork` is done backwards (highest checkpoint first) to propagate trust from the SCP-consensus hash downward. - If `VerifyLedgerChainWork` detects a mismatch with a trusted SCP hash, it sets `mFatalFailurePromise` to true, causing `CatchupWork::fatalFailure()` to return true and `LedgerApplyManagerImpl` to set `mCatchupFatalFailure`, permanently blocking further catchup attempts. - `ApplyCheckpointWork` validates that the resulting LCL hash matches the expected ledger header after each ledger application. - Publish queue flow control in `DownloadApplyTxsWork` prevents the publish queue from growing beyond `PUBLISH_QUEUE_MAX_SIZE` by gating `ApplyCheckpointWork` behind a `ConditionalWork`. - BucketList merge futures are awaited (via `ConditionalWork`) before applying any ledger, both during checkpoint replay and buffered ledger application.