Ac Data Pipeline Orchestration

Running batch data pipelines as a platform rather than as scripts: the DAG as the dependency contract, idempotent reruns and backfills, late-arriving data and upserts, partition strategy as the pipeline's physical index, freshness monitoring and SLAs, lineage and pipeline observability, and the written pipeline record. It explains how batch pipelines are scheduled, rerun, and kept observable; it never recommends an orchestrator and never promises a freshness figure. Use when asking "data pipeline orchestration", "DAG scheduling", "backfill", "idempotent pipeline", "late arriving data", "partition strategy", "data freshness SLA", "pipeline lineage", "rerun failed tasks only", "upsert pattern". Not for market-data collection specifically (ac-market-data-collection), not for the data contract at a boundary (ac-data-contracts), not for unbounded stream processing (ac-stream-processing).

jiayaoqijia Updated 81 repo stars

File contents

jiayaoqijia/cryptoskill commit 04de0acac3

Frequently asked questions

npx get-skill add jiayaoqijia/ac-data-pipeline-orchestration