Airflow

Apache Airflow

NeuralBlitz Updated 1 repo stars

File contents

Apache Airflow

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. It enables defining complex data pipelines as code with rich scheduling and monitoring.

Key Concepts

  • DAGs (Directed Acyclic Graphs)
  • Operators and hooks
  • Task dependencies
  • XCom for communication
  • Sensors for waiting

Common Use Cases

  • ETL pipelines
  • ML pipelines
  • Data orchestration
  • Batch processing
  • Workflow automation

Best Practices

  • Keep DAGs simple and flat
  • Use proper task grouping
  • Implement retry strategies
  • Use connections properly
  • Monitor with Airflow UI

Resources

  • Airflow.apache.org: airflow.apache.org/docs
  • Related Skills: data-engineering, python, kubernetes

NeuralBlitz/Agent-Gateway/tree/main/agent-gateway/skills/user/categories/data-science/airflow commit 063a86c106

Frequently asked questions

npx skillmds@latest add neuralblitz/airflow-2