Google Cloud Spark & Dataproc Observability
Starter guide and gcloud lookup map for monitoring, inventorying, and
assessing Google Cloud Managed Service for Apache Spark (Dataproc
Serverless batches and interactive sessions, and Dataproc on Compute
Engine clusters and jobs). For exact gcloud commands, field paths, metric
names, and log filters, see
gcloud Observability Reference.
Skill Routing & Handoff
- Fleet inventory & observability (multi-region discovery, active/failed
workloads, DCU/cost ranking, idle clusters, schedule cadence,
operations/stockouts, quotas, alerts, Monitoring/Logging):
- Skill:
gcp-spark-observability(this skill).
- Skill:
- Root-cause analysis (RCA) of a specific failed or slow batch or job
(driver stack traces, executor OOMs, shuffle spill, stage skew,
sparkApplicationsAPI, GCS event logs):- Skill:
@skill:gcp-spark-troubleshooting. - Handoff: if no
batch_idorjob_idis given (for example, "Why did my last Spark batch fail?"), use this skill first to find the latest failed ID, region, andstateMessage/status.details, then hand off to@skill:gcp-spark-troubleshooting.
- Skill:
- Optimizing a specific batch or job (runtime or DCU regression, GC,
spill, skew, partitioning, Spark property tuning, AQE, autotuning, executor
sizing, code anti-patterns):
- Skill:
@skill:gcp-spark-troubleshooting. - Handoff: use this skill first to identify top DCU consumers, regressed
schedules, or saturated clusters, then hand off one workload at a time
with its ID, region, usage/runtime metrics, and
runtimeConfig/softwareConfigproperties. Do not propose Spark property tuning from fleet metrics alone.
- Skill:
- Authoring or submitting new PySpark/Spark code, notebooks, or creating new
clusters/batches:
- Skill:
gcp-spark.
- Skill:
Non-Obvious CLI & API Gotchas (Read First)
- MCP field projection vs.
gcloud: Dataproc MCP list/get tools (list_batches,get_batch,list_sessions,list_clusters,get_cluster) return minimal summaries and omitruntimeInfo(DCU/shuffle usage),stateMessage,creator, clusterconfig(machine types, lifecycle TTLs, autoscaling), andmetrics.yarnMetrics. Always rungcloud dataproc ... list|describe --format=jsonwhen you need those fields. - Release tracks & location flags:
clusters,jobs,batches,operations,workflow-templates, andautoscaling-policiesare GA and use--region=<REGION>.sessionsandsession-templatesrequire thebetatrack and--location=<REGION>(gcloud beta dataproc sessions list --location=<REGION>).- Always pass
--quiettoasset,metastore,composer, andschedulercommands so disabled APIs do not hang on interactive prompts. Never enable APIs.
- Always specify
--limit& respect server-side--filterrestrictions:- Always pass an explicit
--limit(for example--limit=20to--limit=200) on everygcloud ... list,gcloud asset search-all-resources, andgcloud logging readquery so list calls stay bounded. batches list --filter: acceptsstate,create_time>="<RFC3339>"(snake_case;createTime>=is rejected), andlabels.<key>=<value>.jobs list --filter:status.stateonly acceptsACTIVE,INACTIVE, orNON_ACTIVE(status.state=ERRORis rejected withINVALID_ARGUMENT). Use--state-filter=inactive --limit=100and filterERRORclient-side.operations list --filter:status.stateonly acceptsPENDING,RUNNING, orDONE. Filter server-side withoperationType=CREATE AND status.state=DONE --limit=100and filter client-side whereerror.codeis set.
- Always pass an explicit
- Region resolution:
- Single region: use the user-specified region,
@skill:resolving-mcp-region-configs, orgcloud config get-value dataproc/region(fallbackcompute/region). - Project-wide / multi-region: discover regions with Cloud Asset
Inventory (
gcloud asset search-all-resources --asset-types="dataproc.googleapis.com/Cluster,dataproc.googleapis.com/Batch,dataproc.googleapis.com/Session,dataproc.googleapis.com/Job" --limit=200), then query each discovered region live viagcloud dataproc(Asset Inventory states can be stale). Always state which regions were inspected.
- Single region: use the user-specified region,
- Usage, idle clusters, and safety:
runtimeInfo.approximateUsage(milliDcuSeconds,shuffleStorageGbSeconds) is populated only after a batch/session completes; running workloads exposeruntimeInfo.currentUsage.- $\text{DCU-Hours} = \text{milliDcuSeconds} / 3{,}600{,}000$; $\text{Shuffle-GB-Months} = \text{shuffleStorageGbSeconds} / 2{,}592{,}000$. DCU-hours measure compute usage, not dollar spend (use Cloud Billing export for dollars; never fabricate dollar amounts).
- A
RUNNINGcluster is idle only when it has 0 active Jobs API jobs AND 0 running/pending YARN applications (metrics.yarnMetrics.yarn-apps-runningandyarn-apps-pendinginclusters list), because notebooks, SSHspark-submit, and Spark Connect bypass the Jobs API. - Audit log entries with a non-empty
protoPayload.status(such asPERMISSION_DENIED) are rejected API calls, not cluster operations. - Never cancel, kill, stop, or delete any workload or cluster without explicit user confirmation.
Where to Look (Quick Map)
- Active fleet inventory (all 4 surfaces): query
batches list(RUNNING/PENDING),beta dataproc sessions list(ACTIVE/CREATING),clusters list(RUNNING/CREATING/UPDATING), andjobs list(--state-filter=active) with--limit=100across target regions. - Recent failed batches or jobs:
batches list --filter="state=FAILED" --sort-by=~createTime --limit=20(appendAND labels.<key>=<value>when a workload label is given; inspectstateMessage) andjobs list --state-filter=inactive --limit=100filtered tostatus.state=ERROR(status.details). - Scheduled pipeline cadence:
batches list --sort-by=~createTime --limit=200grouped bylabels.airflow-dag-id/labels.airflow-task-id,labels.goog-dataproc-workflow-template-id, or batch-ID prefix; check median interval and runtime (stateTime - createTime). - Top Serverless compute consumers:
batches list --sort-by=~runtimeInfo.approximateUsage.milliDcuSeconds --limit=20and convert to DCU-hours and Shuffle-GB-Months. - Idle / zombie clusters:
clusters list --filter="status.state=RUNNING" --limit=100(always reportmetrics.yarnMetrics,config.lifecycleConfigTTLsidleDeleteTtl/autoDeleteTime/idleStopTtl, and recentjobs list --cluster=<NAME> --limit=5; recommendgcloud dataproc clusters update <NAME> --region=<REGION> --max-idle=30mor--stop-max-idle=30m/--delete-max-idle=30m). - Failed cluster operations & stockouts:
operations list --filter="operationType=CREATE AND status.state=DONE" --limit=100(always specify--limitand filter client-side whereerror.codeis set), plus Admin Activity audit logs (--limit=50) for zonal stockouts (does not have enough resources available). - Quotas & alerting coverage:
compute regions list --flatten=quotas --limit=100,alpha services quota list --service=dataproc.googleapis.com --limit=100(data_compute_units), andmonitoring policies list --limit=100. - Cloud Monitoring & Logging:
dataproc.googleapis.com/cluster/yarn/*,dataproc.googleapis.com/batch/spark/executors, and Cloud Logging resource typescloud_dataproc_{batch,session,cluster,job}.
See gcloud Observability Reference for all commands and field paths.