GCP Spark

Authors, optimizes, and runs PySpark and Spark code on Google Cloud Managed Spark (Dataproc Serverless, clusters, and Spark Connect). Use when: - Authoring or running Spark/PySpark notebooks in any kernel (Local Kernel with spark-remote templates, Remote Spark / Remote Kernel, ManagedSparkSession, Spark Connect). - Querying, transforming, or writing data with PySpark or Spark SQL, including Apache / BigLake Iceberg, BigQuery, GCS, Spanner, Cloud SQL, AlloyDB, Amazon S3, or Azure Blob Storage / ADLS Gen2. - Building PySpark ETL pipelines, machine learning workloads, or batch jobs. - Managing Dataproc clusters, serverless batches, or interactive sessions. - Optimizing PySpark or Spark SQL queries for performance, memory, and OOM prevention. Do NOT use when: - Auditing multi-region fleet inventory, DCU/cost consumption, idle/zombie clusters, or workload observability (use @skill:gcp-spark-observability). - Troubleshooting failed Spark jobs or Dataproc batches (use @skill:gcp-spark-troubleshooting).

GoogleCloudPlatform Updated 1.1k repo stars

File contents

googlecloudplatform/data-agent-kit-plugin commit 85777b1e75

Frequently asked questions

npx get-skill add googlecloudplatform/gcp-spark