alivirgo Apache Spark AI Skill Guide
Overview & Engine Architecture
Spark distributes DataFrame/Dataset work across executors. Transformations build a lineage; actions (count, write) trigger jobs split into stages at shuffle boundaries. Agents design partition keys, avoid wide shuffles, cache only reused hot datasets, and read Spark UI / explain plans before throwing more cores at a bad query.
Driver (SparkSession)
-> jobs / stages / tasks
-> executors (partitions)
-> shuffle / reduce
-> sink (Parquet, table)
When to use this skill
- Multi-node ETL beyond single-machine Polars/DuckDB
- Lakehouse table transforms (Parquet/Delta/Iceberg via connectors)
- Spark SQL shared with analytics engineers
Operational directives
- Prefer DataFrame API / Spark SQL over RDDs for optimizer benefits.
- Repartition or bucket on join/group keys intentionally; do not
repartition(200)blindly. - Filter and project before joins.
- Treat
collect()to the driver as a bug on large results. - Size AQE / broadcast hints with measured data, not folklore.
PySpark transform example
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("orders-mart").getOrCreate()
orders = spark.read.parquet("s3a://lake/orders/")
daily = (
orders
.where(F.col("created_at") >= "2025-01-01")
.groupBy(F.to_date("created_at").alias("day"), "customer_id")
.agg(F.sum("amount").alias("revenue"), F.count("*").alias("orders"))
)
daily.write.mode("overwrite").parquet("s3a://lake/marts/daily_revenue/")
Diagnose skew
daily.explain(extended=True)
# Spark UI: Stages -> look for long-tail tasks / spill
Common failures
| Symptom | Cause | Fix |
|---|---|---|
| Shuffle spill | uneven keys / wide join | salt keys; broadcast small side |
| Driver OOM | .collect / huge broadcast |
write out; sample |
| Tiny files | too many partitions on write | coalesce / maxRecordsPerFile |
| Slow job | UDF per row | built-in functions / Pandas UDFs carefully |
Best practices
- Idempotent writes to dated paths or transactional tables.
- Persist (
cache) only multi-action reuse; thenunpersist. - Align partition columns with filter patterns (
dt=YYYY-MM-DD). - Pin Spark + connector versions with the cluster runtime.
Limitations
- Local mode is for unit tests, not production scale claims.
- Streaming Structured Streaming needs separate checkpoint design.
- Cloud IAM/storage credentials are environment-specific.
Related skills
@duckdb/@polars- single-node alternatives@airflow/@prefect- schedule Spark jobs@dbt- SQL models on Spark/Databricks SQL warehouses