Managed Spark on Google Cloud
[!IMPORTANT]
You MUST follow the Task Execution Workflow when writing spark code.
Notebook Generation Environment Rules
[!CAUTION]
When the task is to generate a Spark notebook (as opposed to executing one), you MUST NOT modify your own local execution environment (i.e. do not run these commands with your shell or other tools). Specifically:
- NEVER create a virtual environment (
python -m venv,virtualenv,conda create,uv venv,poetry init/install, etc.).- NEVER install packages (
pip install,%pip install,uv pip install,conda install,poetry add).- NEVER run environment discovery loops such as activating a venv,
pip list, or probing foripykernel/google-cloud-spark-connectas a precondition for writing the notebook.Assume the user will select the appropriate kernel and dependencies after the notebook is generated.
Task Execution Workflow
Understand user request:
New Spark notebooks target the Local Python kernel and connect to Managed Spark Serverless with
ManagedSparkSession(Spark Connect); do NOT ask the user which kernel to use. Only when the prompt explicitly asks for a Remote Spark kernel or to run against a Dataproc cluster, skip session initialization and use the pre-initializedsparkvariable. Use the Runtime Profile (formerly Session Template) named in the prompt; otherwise ask the user which one to use (scan for existing ones withgcloud dataproc session-templates list --location=<REGION>). Local Python notebooks MUST start with these two cells (plain constants; noos.environ, noSession()objects):PROJECT_ID = "<PROJECT_ID>" REGION = "<REGION>" RUNTIME_PROFILE = "<RUNTIME_PROFILE_ID>"# %pip install -U 'google-cloud-spark-connect[client,interactive]~=2.2' from google.cloud.managed_spark_connect import ManagedSparkSession spark = ( ManagedSparkSession.builder .projectId(PROJECT_ID) .location(REGION) .runtimeProfile(RUNTIME_PROFILE) .getOrCreate() )
Understand schemas: ALWAYS use
@skill:discover-assetsskill orreferences/schema_direct_inspection.mdto understand input and output schemas. Include the schema in your thought process BEFORE generating any code. Do NOT guess column names. Cap GCP data asset discovery attempts at 3 retries max. Unless explicitly specified, assume that the assets are located in the same project. Avoid scanning for assets across other projects as it can take a long time. If an expected dataset or table does not exist, use@skill:discover-assetsto discover all similar tables in the namespace or project.MISSING SOURCE RULE: If the request does not name a table, GCS path or catalog (or a required database instance/connection or
s3a:///abfss:///wasbs:///wasb://URI), you MUST run at most ONE@skill:discover-assetssearch. If it does not return a single clear match, you MUST list the candidates, ask the user which source to use, and STOP. You MUST NOT infer the source by listing buckets, reading existing notebooks or scripts, or runningSHOW NAMESPACES/SHOW TABLES.MINOR TYPO RULE: If there is a minor singular/plural or 1–2 character typo in a table name (e.g.
employeesvsemployee, orloansvsloan), immediately fix the table name in code and proceed without asking for user confirmation, noting the correction in your response.STRICT HALT RULE: If the discovered table names differ from the requested table by more than a minor typo (e.g. completely different words, prefixes, or suffixes), you must IMMEDIATELY report the missing table and a neutral list of all available alternatives in the same namespace to the user without making any recommendations. You MUST ask the user which alternative to use and then STOP EXECUTING your turn. Do NOT write any Spark code or notebooks. Do NOT proceed with code generation, do NOT add fallback logic to code, and do NOT automatically substitute any alternative table (even if its schema seems to match) without explicit user permission.
Verify source accessibility: For Google Cloud Storage (
gs://) paths, verify access/existence usinggcloud storage ls gs://<path-to-dataset>(skipgcloud storage lsfor external cloud URIs such ass3a://,abfss://,wasbs://, orwasb://, which are accessed directly by Spark's Hadoop FileSystem connectors). If accessing or reading a GCS path fails with a storage error e.g., permission errors like403 Forbidden/Forbidden/PermissionDenied, or location errors like404 Not Found/NotFound/FileNotFoundExceptionyou should report the error immediately. Either (1) ask the user what to do next, or (2) if asked to execute a notebook, save the notebook with the error output and recommend next steps to resolve the issue. Do NOT scan all buckets for alternative fallback datasets when encountering GCS errors. NEVER create a new GCS bucket (gcloud storage buckets create) without explicit user confirmation; if a required bucket does not exist, STOP and ask the user.Generate spark code:
Output Format: ALWAYS generate code in Python Notebooks (.ipynb) format unless a standalone script or batch/cluster job is explicitly requested: generate a Python script (
.py) when a PySpark script/job is requested, or a Spark SQL script (.sql) when a Spark SQL script orspark-sqljob/batch is requested (refer toreferences/gcloud_dataproc.mdforspark-sqlsyntax and submission).Spark Session Initialization:
[!IMPORTANT] Initializing a Spark session on Google Cloud can take 2-3 minutes. You MUST inform the user about the potential delay.
[!CAUTION] You MUST NOT create a local Spark session. The following are BANNED: -
SparkSession.builder.master("local")-pyspark.sql.SparkSession.builder.getOrCreate()in notebooks or local Spark Connect scripts - Anytry/exceptfallbacks that revert to a localSparkSession- Silently switching execution modes when an explicitly requested mode fails (e.g. falling back from Spark Connect / interactive session togcloud dataproc batches submitor cluster creation without asking the user).- Remote Kernel Notebooks (
.ipynb):sparkis pre-initialized; use it directly without callingManagedSparkSessionorSparkSession.builder.getOrCreate(). - Local Python Notebooks (
.ipynb) & Spark Connect scripts: You MUST useManagedSparkSessionfromgoogle-cloud-spark-connectto connect to Managed Spark Serverless. - Dataproc Batch & Cluster jobs (
.pysubmitted viagcloud dataproc batches submit pysparkorgcloud dataproc jobs submit pyspark): Use standardSparkSession.builder.appName("<APP_NAME>").getOrCreate()(never.master("local")).
Refer to
references/gcloud_dataproc.mdfor detailed configuration. Minimal initialization for Local Python notebooks / Spark Connect:from google.cloud.managed_spark_connect import ManagedSparkSession spark = ( ManagedSparkSession.builder.projectId("<PROJECT_ID>") .location("<REGION>") .getOrCreate() )- Remote Kernel Notebooks (
Production Logging: In standalone production PySpark batch jobs and scripts (
.py), you MUST use the standard Pythonloggingmodule instead ofprint()statements for job lifecycle, progress, and record counts. Do NOT use theloggingmodule in notebooks (.ipynb); useprint(),.show(), and standard cell outputs instead. Configureloggingfor.pyscripts with timestamps:import logging logging.basicConfig( level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s", )Security & Dynamic Variables: You MUST NOT hardcode plaintext credentials, passwords, or tokens (use Secret Manager — refer to
references/jdbc_cloudsql.mdwhen retrieving credentials from Secret Manager). You MUST NOT hardcode environment-specific variables; parameterize project IDs, dataset IDs, table names, and bucket paths viaos.environ.get(...)or arguments.Missing or Inaccessible Custom JAR Rule (Do Not Improvise):
- If a required custom
gs://.../*.jarURI (e.g., JDBC driver or Socket Factory Uber JAR) is not provided by the user, or if Spark session/job execution fails because the JAR does not exist or is inaccessible in GCS, you MUST STOP and ask the user for thegs://JAR URI(s). - NEVER guess JAR paths, scan arbitrary GCS buckets for
.jarfiles, download JARs from Maven Central (curl/wget) to upload to GCS, or substitutespark.jars.packagesMaven coordinates.
- If a required custom
Read and Write data: ALWAYS load only the connector reference file(s) needed for the prompt:
- BigQuery, GCS, or Pub/Sub:
references/read_write_data.md - BigLake Iceberg Catalog:
references/biglake_iceberg.md - Cloud Spanner:
references/spanner.md - JDBC, Cloud SQL (PostgreSQL / MySQL / SQL Server), AlloyDB &
Secret Manager:
references/jdbc_cloudsql.md - Amazon S3 (
s3a:/// S3-compatible object storage):references/aws_s3.md - Azure Blob Storage & ADLS Gen2 (
abfss://,wasbs://,wasb://):references/azure_storage.md
- BigQuery, GCS, or Pub/Sub:
Machine Learning Tasks: Refer to
@skill:ml-best-practicesskill andreferences/ml_tasks.mdwhen generating Machine Learning code.Spark Optimizations & Broadcast Joins: ALWAYS refer to
references/spark_optimizations.md. When joining a large DataFrame with a small lookup or dimension table, you MUST usebroadcast()(from pyspark.sql.functions import broadcast) on the small table. Treat about 10 MB (Spark's defaultspark.sql.autoBroadcastJoinThreshold) as a guide to what counts as small, not a hard limit.
Verify schema before write: ALWAYS verify that the dataframe and destination schema match, use
df.printSchema()for dataframe schema and refer to@skill:discover-assetsskill orreferences/schema_direct_inspection.mdto verify destination schema.Compile code before executing: For notebooks convert them to python script using
jupyter nbconvert --to script your-notebook.ipynbfirst. Then compile the resulting python script usingpython3 -m py_compile your-script.py. The same can be done for pyspark source code (skippy_compilefor pure.sqlscripts).Execute script or notebook: When requested to run a job, script, session, or execute notebook cells against Managed Spark, refer to
references/gcloud_dataproc.mdon how to execute code on Dataproc Serverless using Spark Connect or Dataproc jobs.Follow up with the user: If a brand new notebook was generated, instruct the user to select the appropriate kernel in the dropdown for cell execution.
Common Mistakes Checklist
[!CAUTION]
Ensure you verify this checklist to avoid mistakes
Before submitting a job, verify:
- All imports present (
col,when,lit,broadcast, etc. frompyspark.sql.functions) -
vector_to_arrayfrom correct module usefrom pyspark.ml.functions import vector_to_array(NOTpyspark.sql.functions) - DataFrame schema matches target Iceberg table verify with
df.printSchema()before writing - CSV files read with
headerandinferSchemawithout these, the header row becomes data and all columns are strings - Driver memory safety (
toPandas()/collect()) NEVER call.toPandas()or.collect()on raw or un-aggregated DataFrames. ALWAYS perform transformations, aggregations (groupBy().agg()), or data reduction (limit(),sample()) in Spark before converting small summaries to Pandas for plotting or display. - No inline pip install in Spark batch/script jobs: NEVER run pip
install or subprocess package installations inside PySpark batch scripts
(the
%pip installsetup cell in generated notebooks is exempt). Pass dependencies using --properties=spark.jars.packages=..., --archives=gs://.../env.tar.gz#environment, --py-files, or a custom --container-image. - Optimization & Pre-Submission Refactoring Verify the job against
references/spark_optimizations.mdand follow the confirmation protocol inreferences/spark_refactoring_guide.md. - No full file reads of executed notebooks: You MUST NOT use
read_fileor generic full file reading tools on executed.ipynbnotebooks; they often contain massive base64-encoded image outputs that cause excessive token consumption. You MUST inspect execution outputs with cell-scoped reading tools (such asnotebook__read_cell,jupyter__read_cell, or specific line slices).
IAM Requirements
The Managed Spark (Dataproc) service account needs:
roles/dataproc.worker: Job executionroles/biglake.admin: BigLake Iceberg table managementroles/bigquery.jobUser: BigQuery query materialization and load jobsroles/bigquery.dataEditor: BigQuery table reads and writes (direct and indirect)roles/bigquery.readSessionUser: BigQuery Storage Read API session creationroles/storage.objectUser: Read/write GCS (data, Iceberg warehouse, and--deps-bucket/temporaryGcsBucket)roles/spanner.databaseUser: Spanner reads and writesroles/secretmanager.secretAccessor: Accessing database passwords and credentials stored in Secret Manager
Spark resource management & observability
Refer to references/gcloud_dataproc.md for detailed guidelines on managing
Spark clusters, jobs, batches, interactive sessions, and Spark Connect sessions.
For fleet-wide and multi-region observability — including active workload
inventory across Batches, Sessions, Clusters, and Jobs, failed batch discovery,
Serverless compute consumption (DCU-Hours and Shuffle-GB-Months), idle or
zombie cluster waste detection, scheduled pipeline cadence health, configuration
auditing, and Cloud Monitoring / Cloud Logging telemetry — use the
@skill:gcp-spark-observability skill.
Code Optimization & Pre-Submission Verification
Before submitting any Spark job or Dataproc batch, enforce the pre-submission
verification protocol in references/spark_refactoring_guide.md: inspect the
code for the anti-patterns catalogued in references/spark_optimizations.md and
obtain explicit user confirmation before applying refactoring.
Troubleshooting & Root Cause Analysis
For troubleshooting failed Spark jobs or Dataproc batches, inspecting/tailing
GCS driver output logs, analyzing Spark event logs, or performing Root Cause
Analysis (RCA), use the @skill:gcp-spark-troubleshooting skill.