Managed Service for Apache Airflow (formerly Cloud Composer) DAG troubleshooting guide
This skill provides instructions for troubleshooting Managed Airflow DAGs (DAG
runs and task instances), utilizing gcloud composer, gcloud logging and
gcloud storage commands to fetch remote logs and code.
Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read Usage Attribution. For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix gcloud with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:managed-airflow-dag-troubleshooting)"
and append agent-skills/1.0 (skill:managed-airflow-dag-troubleshooting) to User-Agent headers.
General rules
Provide suggestions on how to troubleshoot the failed jobs. Provide only the
steps that the user can actually take. Ground all troubleshooting advice in
direct findings.
When troubleshooting a failure, follow the following practices to always
provide a deterministic diagnosis:
Fetch relevant logs: Always fetch the logs for a task under
investigation using gcloud logging read; check the logs for specific
error patterns: Python tracebacks, API error codes (e.g., 400, 403, 404,
500), or Airflow signals (e.g., AirflowTaskTimeout).
Fetch task metadata: When troubleshooting a task, fetch the task
state and metadata (execution state, try number, timestamps, and
execution details) using:
gcloud composer environments run {env_name} \
--location {location} \
tasks states-for-dag-run -- -d {dag_id} -r {run_id}
or for an individual task instance:
gcloud composer environments run {env_name} \
--location {location} \
tasks state -- {dag_id} {task_id} {execution_date}
Retrieve and compare DAG source code: Download the remote DAG source
code using gcloud storage cp gs://{bucket_name}/dags/{dag_file}.py .
(find the environment bucket via gcloud composer environments describe {env_name} --location {location} --format="value(config.dagGcsPrefix)"). Compare the parameters in the
code (e.g., table IDs, disk sizes, URI paths) against the error messages
found in the task logs.
Explain code mistakes and potential fixes: Explain mistakes in the
code (if any are actually visible); suggest potential fixes (if they are
very likely to be meaningful); discuss source code availability if
needed - if some source code is unavailable (e.g. imported from a file
other than the main source code file), mention this (you can mention the
package name) - in such a case take into account most likely trigger
rules if they are unknown.
Check for environment-level errors: Query Cloud Logging with gcloud logging read to see if there are high-level environment issues or known
platform errors correlating with the failure (see Known issues
below). You MUST return ALL found issues.
Identify failing tasks in a DAG run: When troubleshooting a failed
DAG run, mention the task that caused a failure (use tasks states-for-dag-run or Cloud Logging to identify failed tasks). Provide
a task instance name. If many tasks failed, mention which task was
critical (mandatory for successful DAG run execution - look into task
dependencies and trigger rules) and focus on this one.
Verify service configurations in code: If logs suggest an issue with
a specific service (e.g., BigQuery, Dataform, Compute Engine), use the
log details to verify the configuration in the DAG source code.
Correlate logs with code: E.g., if BigQuery returns a 404, verify
the dataset ID or table ID in the DAG source code matches reality.
Prioritize known platform issues: Check against Known issues
below. If Cloud Logging queries return matching platform error signals,
prioritize that diagnosis.
Summarize with Evidence (Deterministic Response): Your response must be
specific. Avoid general advice like 'check your permissions.' or 'check the
logs.' Instead, say 'The service account is missing X permission.'
- Problem: State the specific root cause and the exact task instance
ID. Identify if it is a code logic error, a configuration mismatch, or
an environment timeout.
- Evidence: Mandatory. Provide the verbatim text from the log
(
textPayload) or the specific line of code from the DAG that caused
the failure. Do not summarize the evidence; show the data.
- Recommendation: Provide an actionable fix. If it is a code error,
provide the corrected Python snippet. If it is a resource issue, specify
the exact configuration change needed.
DAGs Generated by Orchestration Pipelines: Some DAGs may be generated by
Orchestration Pipelines. A special requirement related to those DAGs is the
need to explain the failure in terms of the logical actions defined in the
pipeline YAML.
- Determine if a DAG is generated by Orchestration Pipelines:
Orchestration Pipeline DAGs deployed by dedicated tools have
bundle_name, version_id, and pipeline_name set in their DAG Run
metadata (DagRun.note that contains JSON metadata). All of them (i.e.
Orchestration Pipeline DAGs deployed by dedicated tools and created
manually) have an op:orchestration_pipeline tag set (DAG properties,
including tags, can be verified in the DAG source code or via gcloud composer environments run {env_name} --location {location} dags list).
- Orchestration Pipeline DAGs deployed by dedicated tools have
additionally the following tags (information in those tags should be
consistent with data in DAG Run attributes mentioned above):
- pipeline name - tag
op:pipeline, e.g. op:pipeline:xyz indicates
a name xyz
- bundle name - tag
op:bundle
- version id - tag
op:version
- Retrieve the resolved pipeline YAML definition from the environment
bucket:
- Determine the YAML file location:
- Retrieve the DAG source code from the environment bucket using
gcloud storage cp gs://{bucket_name}/dags/{dag_file}.py . (or
gcloud storage cat gs://{bucket_name}/dags/{dag_file}.py).
- Inspect the source code for
generate or generate_dags
function calls:
- Scenario 1:
generate call found. The first argument is the
path to the YAML file - relative to the dags folder in
environment's bucket.
- Scenario 2:
generate_dags call found.
- Extract the first argument - this is the data folder. If
it starts with
/home/airflow/gcs/, remove this prefix
to get a path relative to the root of environment's
bucket.
- Extract
bundle_name, version_id, and pipeline_name
(as explained above).
- Construct the path:
{data_directory}/{bundle_name}/versions/{version_id}/{pipeline_name}.yml
(or .yaml).
- Scenario 3: If neither call is found, default to the path:
data/{bundle_name}/versions/{version_id}/{pipeline_name}.yml
(or .yaml) in an environment's bucket.
- Download the YAML file using
gcloud storage cp gs://{bucket_name}/{yaml_path} . (or gcloud storage cat gs://{bucket_name}/{yaml_path}).
- Map the failed Airflow task back to the logical action name using task
instance metadata/notes (e.g.
op_action_name in task note).
- If the failure involves user assets (like Python scripts), check their
path in the action definition. If they are in the environment bucket,
download and read them to debug (
gcloud storage cp gs://{bucket_name}/{asset_path} .). If they are in a custom artifact
bucket (see GCS URIs in logs/config), note the limitation that they
cannot be read directly but analyze based on available logs.
You can assume that environment variables set by default (they can be used
in DAG code, but are not visible in custom environment configuration), e.g.
GCS_BUCKET, are correct - users cannot change them.
"Not found" (404) errors from GCP APIs can be misleading. A "not found"
error might be returned when a resource actually exists, but the caller does
not have permissions to access or view it. If a resource is expected to
exist, suggest verifying proper permissions.
Important constraints & instructions
- Read-Only First: Do NOT attempt to fix the code immediately. You must
first prove the root cause using logs and remote code.
- No Speculation: If logs are empty or code cannot be found, state this
clearly. Always reference error messages as the are.
- Safety: Be careful with secrets. If logs contain sensitive information
(e.g. passwords), redact it in your analysis.
Applying Fixes - only if explicitly requested
When the RCA is complete and a fix is ready:
- Repository Check: If the current workspace does not seem to be the
source of truth for the Managed Airflow environment:
- Ask the user to open the correct repository.
- OR ask if they want to download the remote DAG to the current
workspace to apply the fix (warning them about potential overwrites).
Relevant gcloud commands
Environment & DAG Discovery
List composer environments:
gcloud composer environments list \
--locations=us-central1 \
--format="table(name,location,state)"
Describe environment (get DAGs bucket and config):
gcloud composer environments describe {env_name} \
--location {region} \
--format="value(config.dagGcsPrefix)"
List composer DAGs:
gcloud composer environments run {env_name} \
--location {region} \
dags list
List composer DAG Runs:
gcloud composer environments run {env_name} \
--location {region} \
dags list-runs -- -d {dag_id} --no-backfill
List task instance states for a DAG run:
gcloud composer environments run {env_name} \
--location {region} \
tasks states-for-dag-run -- -d {dag_id} -r {run_id}
Get state of a specific task instance:
gcloud composer environments run {env_name} \
--location {region} \
tasks state -- {dag_id} {task_id} {execution_date}
Log Retrieval
Fetch error logs for a DAG / Task:
gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND labels.dag_id="{dag_id}" AND severity>=ERROR' \
--limit=25 \
--format="table(timestamp,severity,labels.task_id,textPayload)"
Fetch scheduler logs for environment failures:
gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND log_id("airflow-scheduler") AND severity>=ERROR' \
--limit=25 \
--format="table(timestamp,severity,textPayload)"
Code & Asset Retrieval
Download DAG code from GCS:
gcloud storage cp gs://{bucket_name}/dags/{dag_file}.py .
Download pipeline YAML definition or script from GCS:
gcloud storage cp gs://{bucket_name}/{path_to_file} .
Known issues related to DAG runs and task instances
Use gcloud logging read with the queries below to identify specific known
platform failure modes:
1. DAG_RUN_TIMEOUT
Issue summary: The task instance execution was interrupted because a
timeout for a DAG was exceeded. Unfinished tasks were marked as 'SKIPPED' or
failed.
Cloud Logging Query:
gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND log_id("airflow-scheduler") AND textPayload=~"Run .* of .* has timed-out"' --limit=10
2. TASK_QUEUED_TIMEOUT
Issue summary: Task failed because it remained queued longer than the
maximum allowed queue time.
Cloud Logging Query:
gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND log_id("airflow-scheduler") AND textPayload=~"Task requeue attempts exceeded max; marking failed"' --limit=10
Remediation: Consider increasing worker resources (CPU, memory, worker
count) or adjusting [celery]worker_concurrency.
3. TASK_STUCK_IN_QUEUE
Issue summary: Task reached DAG run timeout because task was stuck in
queue for too long.
Cloud Logging Query:
gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND log_id("airflow-scheduler") AND textPayload=~"Task stuck in queued; will try to requeue"' --limit=10
Remediation: Consider increasing the timeout or reducing the load on the
environment.
4. BIGQUERY_JOB_FAILED
Issue summary: Task failed because of a BigQuery job failure inside a
BigQuery operator.
Cloud Logging Query:
gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND (log_id("airflow-worker") OR log_id("airflow-k8s-worker")) AND textPayload:"airflow/providers/google/cloud/operators/bigquery.py" AND textPayload:"Task failed with exception" AND severity=ERROR' --limit=10
Remediation: Inspect the worker logs for the BigQuery Job ID (Job ID: ...) to diagnose the underlying query error or permissions issue.
5. DETECTED_ZOMBIE
Issue summary: The task instance was revoked by the executor due to
missing heartbeats. Task instances send heartbeats periodically (every
job_heartbeat_sec, 5 seconds by default) and if heartbeats are missing for
scheduler_zombie_task_threshold (300 seconds by default), the task is
considered a zombie and marked as failed or up for retry.
Cloud Logging Query:
gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND log_id("airflow-scheduler") AND (textPayload:"Detected zombie job:" OR textPayload:"Detected a task instance without a heartbeat:")' --limit=10
Remediation: This can happen when a worker is overloaded (CPU/memory
starvation) and unable to send heartbeats on time, a worker was terminated
with unfinished tasks (OOM kill/eviction), or the metadata database is
overloaded. Check worker metrics and consider scaling worker CPU/memory.
6. WORKER_OUT_OF_POD_STORAGE
Issue summary: Task instance failed because a worker is running out of
pod storage (ephemeral disk space reached or pod evicted due to storage
limits).
Cloud Logging Query:
gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND (log_id("airflow-worker") OR log_id("airflow-k8s-worker")) AND textPayload:"Pod ephemeral local storage usage exceeds the total limit of containers"' --limit=10
Remediation: Update the worker storage configuration according to the
amount of data being stored or clean up temporary files created during task
execution.
1---2description: Provides guidance for troubleshooting Apache Airflow DAGs (failed DAG runs and task instances) in Managed Service for Apache Airflow (MSAA; formerly Cloud Composer). Use when figuring out reasons for DAG run or task instance failures. Don't use when looking for overall recommendations for Managed Airflow environment performance.3metadata:4 source: https://skillmd.com/skills/h3y6e/managed-airflow-dag-troubleshooting5 category: BigDataAndAnalytics6 github-path: skills/cloud/managed-airflow-dag-troubleshooting7 github-ref: refs/heads/main8 github-repo: https://github.com/google/skills9 github-tree-sha: b917066981f443c92e0c4844364fc4ef766123a310 version: 1.0.111name: managed-airflow-dag-troubleshooting12---1314# Managed Service for Apache Airflow (formerly Cloud Composer) DAG troubleshooting guide1516This skill provides instructions for troubleshooting Managed Airflow DAGs (DAG17runs and task instances), utilizing `gcloud composer`, `gcloud logging` and18`gcloud storage` commands to fetch remote logs and code.1920## Attribution2122**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you23MUST read [Usage Attribution](references/attribution.md). For commands executed24directly in the terminal (never in generated scripts, Terraform, or user code),25prefix `gcloud` with26`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:managed-airflow-dag-troubleshooting)"`27and append `agent-skills/1.0 (skill:managed-airflow-dag-troubleshooting)` to `User-Agent` headers.2829## General rules30311. Provide suggestions on how to troubleshoot the failed jobs. Provide only the32 steps that the user can actually take. Ground all troubleshooting advice in33 direct findings.342. When troubleshooting a failure, follow the following practices to always35 provide a deterministic diagnosis:3637 * **Fetch relevant logs**: Always fetch the logs for a task under38 investigation using `gcloud logging read`; check the logs for specific39 error patterns: Python tracebacks, API error codes (e.g., 400, 403, 404,40 500), or Airflow signals (e.g., `AirflowTaskTimeout`).41 * **Fetch task metadata**: When troubleshooting a task, fetch the task42 state and metadata (execution state, try number, timestamps, and43 execution details) using:4445 ```bash46 gcloud composer environments run {env_name} \47 --location {location} \48 tasks states-for-dag-run -- -d {dag_id} -r {run_id}49 ```5051 or for an individual task instance:5253 ```bash54 gcloud composer environments run {env_name} \55 --location {location} \56 tasks state -- {dag_id} {task_id} {execution_date}57 ```58 * **Retrieve and compare DAG source code**: Download the remote DAG source59 code using `gcloud storage cp gs://{bucket_name}/dags/{dag_file}.py .`60 (find the environment bucket via `gcloud composer environments describe61 {env_name} --location {location}62 --format="value(config.dagGcsPrefix)"`). Compare the parameters in the63 code (e.g., table IDs, disk sizes, URI paths) against the error messages64 found in the task logs.65 * **Explain code mistakes and potential fixes**: Explain mistakes in the66 code (if any are actually visible); suggest potential fixes (if they are67 very likely to be meaningful); discuss source code availability if68 needed - if some source code is unavailable (e.g. imported from a file69 other than the main source code file), mention this (you can mention the70 package name) - in such a case take into account most likely trigger71 rules if they are unknown.72 * **Check for environment-level errors**: Query Cloud Logging with `gcloud73 logging read` to see if there are high-level environment issues or known74 platform errors correlating with the failure (see **Known issues**75 below). You MUST return ALL found issues.76 * **Identify failing tasks in a DAG run**: When troubleshooting a failed77 DAG run, mention the task that caused a failure (use `tasks78 states-for-dag-run` or Cloud Logging to identify failed tasks). Provide79 a task instance name. If many tasks failed, mention which task was80 critical (mandatory for successful DAG run execution - look into task81 dependencies and trigger rules) and focus on this one.82 * **Verify service configurations in code**: If logs suggest an issue with83 a specific service (e.g., BigQuery, Dataform, Compute Engine), use the84 log details to verify the configuration in the DAG source code.85 * **Correlate logs with code**: E.g., if BigQuery returns a 404, verify86 the dataset ID or table ID in the DAG source code matches reality.87 * **Prioritize known platform issues**: Check against **Known issues**88 below. If Cloud Logging queries return matching platform error signals,89 prioritize that diagnosis.90913. **Summarize with Evidence (Deterministic Response):** Your response must be92 specific. Avoid general advice like 'check your permissions.' or 'check the93 logs.' Instead, say 'The service account is missing X permission.'9495 * **Problem:** State the specific root cause and the exact task instance96 ID. Identify if it is a code logic error, a configuration mismatch, or97 an environment timeout.98 * **Evidence:** **Mandatory.** Provide the verbatim text from the log99 (`textPayload`) or the specific line of code from the DAG that caused100 the failure. Do not summarize the evidence; show the data.101 * **Recommendation:** Provide an actionable fix. If it is a code error,102 provide the corrected Python snippet. If it is a resource issue, specify103 the exact configuration change needed.1041054. **DAGs Generated by Orchestration Pipelines:** Some DAGs may be generated by106 Orchestration Pipelines. A special requirement related to those DAGs is the107 need to explain the failure in terms of the logical actions defined in the108 pipeline YAML.109110 * **Determine if a DAG is generated by Orchestration Pipelines**:111 Orchestration Pipeline DAGs deployed by dedicated tools have112 `bundle_name`, `version_id`, and `pipeline_name` set in their DAG Run113 metadata (`DagRun.note` that contains JSON metadata). All of them (i.e.114 Orchestration Pipeline DAGs deployed by dedicated tools and created115 manually) have an `op:orchestration_pipeline` tag set (DAG properties,116 including tags, can be verified in the DAG source code or via `gcloud117 composer environments run {env_name} --location {location} dags list`).118 * Orchestration Pipeline DAGs deployed by dedicated tools have119 additionally the following tags (information in those tags should be120 consistent with data in DAG Run attributes mentioned above):121 * pipeline name - tag `op:pipeline`, e.g. `op:pipeline:xyz` indicates122 a name `xyz`123 * bundle name - tag `op:bundle`124 * version id - tag `op:version`125 * **Retrieve the resolved pipeline YAML definition from the environment126 bucket**:127 * Determine the YAML file location:128 1. Retrieve the DAG source code from the environment bucket using129 `gcloud storage cp gs://{bucket_name}/dags/{dag_file}.py .` (or130 `gcloud storage cat gs://{bucket_name}/dags/{dag_file}.py`).131 2. Inspect the source code for `generate` or `generate_dags`132 function calls:133 * Scenario 1: `generate` call found. The first argument is the134 path to the YAML file - relative to the `dags` folder in135 environment's bucket.136 * Scenario 2: `generate_dags` call found.137 * Extract the first argument - this is the data folder. If138 it starts with `/home/airflow/gcs/`, remove this prefix139 to get a path relative to the root of environment's140 bucket.141 * Extract `bundle_name`, `version_id`, and `pipeline_name`142 (as explained above).143 * Construct the path:144 `{data_directory}/{bundle_name}/versions/{version_id}/{pipeline_name}.yml`145 (or `.yaml`).146 * Scenario 3: If neither call is found, default to the path:147 `data/{bundle_name}/versions/{version_id}/{pipeline_name}.yml`148 (or `.yaml`) in an environment's bucket.149 3. Download the YAML file using `gcloud storage cp150 gs://{bucket_name}/{yaml_path} .` (or `gcloud storage cat151 gs://{bucket_name}/{yaml_path}`).152 * Map the failed Airflow task back to the logical action name using task153 instance metadata/notes (e.g. `op_action_name` in task `note`).154 * If the failure involves user assets (like Python scripts), check their155 path in the action definition. If they are in the environment bucket,156 download and read them to debug (`gcloud storage cp157 gs://{bucket_name}/{asset_path} .`). If they are in a custom artifact158 bucket (see GCS URIs in logs/config), note the limitation that they159 cannot be read directly but analyze based on available logs.1601615. You can assume that environment variables set by default (they can be used162 in DAG code, but are not visible in custom environment configuration), e.g.163 `GCS_BUCKET`, are correct - users cannot change them.1641656. "Not found" (404) errors from GCP APIs can be misleading. A "not found"166 error might be returned when a resource actually exists, but the caller does167 not have permissions to access or view it. If a resource is expected to168 exist, suggest verifying proper permissions.169170### Important constraints & instructions171172* **Read-Only First**: Do NOT attempt to fix the code immediately. You must173 first prove the root cause using logs and remote code.174* **No Speculation**: If logs are empty or code cannot be found, state this175 clearly. Always reference error messages as the are.176* **Safety**: Be careful with secrets. If logs contain sensitive information177 (e.g. passwords), redact it in your analysis.178179### Applying Fixes - only if explicitly requested180181When the RCA is complete and a fix is ready:1821831. **Repository Check**: If the current workspace does not seem to be the184 source of truth for the Managed Airflow environment:185 * Ask the user to **open the correct repository**.186 * OR ask if they want to **download the remote DAG** to the current187 workspace to apply the fix (warning them about potential overwrites).188189## Relevant gcloud commands190191### Environment & DAG Discovery192193* **List composer environments:**194195 ```bash196 gcloud composer environments list \197 --locations=us-central1 \198 --format="table(name,location,state)"199 ```200* **Describe environment (get DAGs bucket and config):**201202 ```bash203 gcloud composer environments describe {env_name} \204 --location {region} \205 --format="value(config.dagGcsPrefix)"206 ```207* **List composer DAGs:**208209 ```bash210 gcloud composer environments run {env_name} \211 --location {region} \212 dags list213 ```214* **List composer DAG Runs:**215216 ```bash217 gcloud composer environments run {env_name} \218 --location {region} \219 dags list-runs -- -d {dag_id} --no-backfill220 ```221* **List task instance states for a DAG run:**222223 ```bash224 gcloud composer environments run {env_name} \225 --location {region} \226 tasks states-for-dag-run -- -d {dag_id} -r {run_id}227 ```228* **Get state of a specific task instance:**229230 ```bash231 gcloud composer environments run {env_name} \232 --location {region} \233 tasks state -- {dag_id} {task_id} {execution_date}234 ```235236### Log Retrieval237238* **Fetch error logs for a DAG / Task:**239240 ```bash241 gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND labels.dag_id="{dag_id}" AND severity>=ERROR' \242 --limit=25 \243 --format="table(timestamp,severity,labels.task_id,textPayload)"244 ```245* **Fetch scheduler logs for environment failures:**246247 ```bash248 gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND log_id("airflow-scheduler") AND severity>=ERROR' \249 --limit=25 \250 --format="table(timestamp,severity,textPayload)"251 ```252253### Code & Asset Retrieval254255* **Download DAG code from GCS:**256257 ```bash258 gcloud storage cp gs://{bucket_name}/dags/{dag_file}.py .259 ```260* **Download pipeline YAML definition or script from GCS:**261262 ```bash263 gcloud storage cp gs://{bucket_name}/{path_to_file} .264 ```265266## Known issues related to DAG runs and task instances267268Use `gcloud logging read` with the queries below to identify specific known269platform failure modes:270271### 1. DAG_RUN_TIMEOUT272273* **Issue summary:** The task instance execution was interrupted because a274 timeout for a DAG was exceeded. Unfinished tasks were marked as 'SKIPPED' or275 failed.276* **Cloud Logging Query:**277278 ```bash279 gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND log_id("airflow-scheduler") AND textPayload=~"Run .* of .* has timed-out"' --limit=10280 ```281282### 2. TASK_QUEUED_TIMEOUT283284* **Issue summary:** Task failed because it remained queued longer than the285 maximum allowed queue time.286* **Cloud Logging Query:**287288 ```bash289 gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND log_id("airflow-scheduler") AND textPayload=~"Task requeue attempts exceeded max; marking failed"' --limit=10290 ```291* **Remediation:** Consider increasing worker resources (CPU, memory, worker292 count) or adjusting `[celery]worker_concurrency`.293294### 3. TASK_STUCK_IN_QUEUE295296* **Issue summary:** Task reached DAG run timeout because task was stuck in297 queue for too long.298* **Cloud Logging Query:**299300 ```bash301 gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND log_id("airflow-scheduler") AND textPayload=~"Task stuck in queued; will try to requeue"' --limit=10302 ```303* **Remediation:** Consider increasing the timeout or reducing the load on the304 environment.305306### 4. BIGQUERY_JOB_FAILED307308* **Issue summary:** Task failed because of a BigQuery job failure inside a309 BigQuery operator.310* **Cloud Logging Query:**311312 ```bash313 gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND (log_id("airflow-worker") OR log_id("airflow-k8s-worker")) AND textPayload:"airflow/providers/google/cloud/operators/bigquery.py" AND textPayload:"Task failed with exception" AND severity=ERROR' --limit=10314 ```315* **Remediation:** Inspect the worker logs for the BigQuery Job ID (`Job ID:316 ...`) to diagnose the underlying query error or permissions issue.317318### 5. DETECTED_ZOMBIE319320* **Issue summary:** The task instance was revoked by the executor due to321 missing heartbeats. Task instances send heartbeats periodically (every322 `job_heartbeat_sec`, 5 seconds by default) and if heartbeats are missing for323 `scheduler_zombie_task_threshold` (300 seconds by default), the task is324 considered a zombie and marked as failed or up for retry.325* **Cloud Logging Query:**326327 ```bash328 gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND log_id("airflow-scheduler") AND (textPayload:"Detected zombie job:" OR textPayload:"Detected a task instance without a heartbeat:")' --limit=10329 ```330* **Remediation:** This can happen when a worker is overloaded (CPU/memory331 starvation) and unable to send heartbeats on time, a worker was terminated332 with unfinished tasks (OOM kill/eviction), or the metadata database is333 overloaded. Check worker metrics and consider scaling worker CPU/memory.334335### 6. WORKER_OUT_OF_POD_STORAGE336337* **Issue summary:** Task instance failed because a worker is running out of338 pod storage (ephemeral disk space reached or pod evicted due to storage339 limits).340* **Cloud Logging Query:**341342 ```bash343 gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="{env_name}" AND (log_id("airflow-worker") OR log_id("airflow-k8s-worker")) AND textPayload:"Pod ephemeral local storage usage exceeds the total limit of containers"' --limit=10344 ```345* **Remediation:** Update the worker storage configuration according to the346 amount of data being stored or clean up temporary files created during task347 execution.