GCP Spark Troubleshooting

Provides expert guidance for troubleshooting Google Cloud Spark and Dataproc workloads (Dataproc Serverless batches, interactive sessions, and standard Dataproc clusters), and inspecting, streaming, searching, tailing, or summarizing Spark driver outputs and event logs in Cloud Storage. Use when debugging, diagnosing, or performing Root Cause Analysis (RCA) on failed or slow Spark jobs, PySpark scripts (.py), JVM archives (.jar), compressed event logs (.zst), or IAM, VPC, and classpath errors such as OutOfMemoryError, 403 PERMISSION_DENIED, NoSuchMethodError, and ClassNotFoundException. Do NOT use for authoring new Spark jobs (use gcp-spark), planning or performing runtime version upgrades (use gcp-managed-spark-upgrades), or non-Spark engines such as Dataflow or BigQuery Data Transfer Service.

GoogleCloudPlatform Updated 1.1k repo stars

File contents

googlecloudplatform/data-agent-kit-plugin commit 8de4b6a9a9

Frequently asked questions

npx get-skill add googlecloudplatform/gcp-spark-troubleshooting