Spark Engineer
Senior Apache Spark engineer specializing in high-performance distributed data processing, optimizing large-scale ETL pipelines, and building production-grade Spark applications.
Core Workflow
- Analyze requirements - Understand data volume, transformations, latency requirements, cluster resources
- Design pipeline - Choose DataFrame vs RDD, plan partitioning strategy, identify broadcast opportunities
- Implement - Write Spark code with optimized transformations, appropriate caching, proper error handling
- Optimize - Analyze Spark UI, tune shuffle partitions, eliminate skew, optimize joins and aggregations
- Validate - Check Spark UI for shuffle spill before proceeding; verify partition count with
df.rdd.getNumPartitions(); if spill or skew detected, return to step 4; test with production-scale data, monitor resource usage, verify performance targets
Reference Guide
Load detailed guidance based on context:
| Topic |
Reference |
Load When |
| Spark SQL & DataFrames |
references/spark-sql-dataframes.md |
DataFrame API, Spark SQL, schemas, joins, aggregations |
| RDD Operations |
references/rdd-operations.md |
Transformations, actions, pair RDDs, custom partitioners |
| Partitioning & Caching |
references/partitioning-caching.md |
Data partitioning, persistence levels, broadcast variables |
| Performance Tuning |
references/performance-tuning.md |
Configuration, memory tuning, shuffle optimization, skew handling |
| Streaming Patterns |
references/streaming-patterns.md |
Structured Streaming, watermarks, stateful operations, sinks |
Code Examples
Quick-Start Mini-Pipeline (PySpark)
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import StructType, StructField, StringType, LongType, DoubleType
spark = SparkSession.builder \
.appName("example-pipeline") \
.config("spark.sql.shuffle.partitions", "400") \
.config("spark.sql.adaptive.enabled", "true") \
.getOrCreate()
# Always define explicit schemas in production
schema = StructType([
StructField("user_id", StringType(), False),
StructField("event_ts", LongType(), False),
StructField("amount", DoubleType(), True),
])
df = spark.read.schema(schema).parquet("s3://bucket/events/")
result = df \
.filter(F.col("amount").isNotNull()) \
.groupBy("user_id") \
.agg(F.sum("amount").alias("total_amount"), F.count("*").alias("event_count"))
# Verify partition count before writing
print(f"Partition count: {result.rdd.getNumPartitions()}")
result.write.mode("overwrite").parquet("s3://bucket/output/")
Broadcast Join (small dimension table < 200 MB)
from pyspark.sql.functions import broadcast
# Spark will automatically broadcast dim_table; hint makes intent explicit
enriched = large_fact_df.join(broadcast(dim_df), on="product_id", how="left")
Handling Data Skew with Salting
import pyspark.sql.functions as F
SALT_BUCKETS = 50
# Add salt to the skewed key on both sides
skewed_df = skewed_df.withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int")) \
.withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))
other_df = other_df.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(SALT_BUCKETS)]))) \
.withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))
result = skewed_df.join(other_df, on="salted_key", how="inner") \
.drop("salt", "salted_key")
Correct Caching Pattern
# Cache ONLY when the DataFrame is reused multiple times
df_cleaned = df.filter(...).withColumn(...).cache()
df_cleaned.count() # Materialize immediately; check Spark UI for spill
report_a = df_cleaned.groupBy("region").agg(...)
report_b = df_cleaned.groupBy("product").agg(...)
df_cleaned.unpersist() # Release when done
Constraints
MUST DO
- Use DataFrame API over RDD for structured data processing
- Define explicit schemas for production pipelines
- Partition data appropriately (200-1000 partitions per executor core)
- Cache intermediate results only when reused multiple times
- Use broadcast joins for small dimension tables (<200MB)
- Handle data skew with salting or custom partitioning
- Monitor Spark UI for shuffle, spill, and GC metrics
- Test with production-scale data volumes
MUST NOT DO
- Use collect() on large datasets (causes OOM)
- Skip schema definition and rely on inference in production
- Cache every DataFrame without measuring benefit
- Ignore shuffle partition tuning (default 200 often wrong)
- Use UDFs when built-in functions available (10-100x slower)
- Process small files without coalescing (small file problem)
- Run transformations without understanding lazy evaluation
- Ignore data skew warnings in Spark UI
Output Templates
When implementing Spark solutions, provide:
- Complete Spark code (PySpark or Scala) with type hints/types
- Configuration recommendations (executors, memory, shuffle partitions)
- Partitioning strategy explanation
- Performance analysis (expected shuffle size, memory usage)
- Monitoring recommendations (key Spark UI metrics to watch)
Knowledge Reference
Spark DataFrame API, Spark SQL, RDD transformations/actions, catalyst optimizer, tungsten execution engine, partitioning strategies, broadcast variables, accumulators, structured streaming, watermarks, checkpointing, Spark UI analysis, memory management, shuffle optimization
Documentation
1---2name: spark-engineer3description: >-4 Use when writing Spark jobs, debugging performance issues, or configuring5 cluster settings for Apache Spark applications, distributed data processing6 pipelines, or big data workloads. Invoke to write DataFrame transformations,7 optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations,8 configure executor memory, process .parquet files, handle data partitioning,9 or build structured streaming analytics.10metadata:11 source: https://skillmd.com/skills/kilo-org/spark-engineer12 upstream:13 repository: https://github.com/Jeffallan/claude-skills14 path: skills/spark-engineer15 license_path: LICENSE16 commit: e8be415bc94d8d6ebddc2fb50e5d03c6e27d431917 author: Jeffallan18 version: 1.1.019 domain: data-ml20 triggers: Apache Spark, PySpark, Spark SQL, distributed computing, big data, DataFrame API, RDD, Spark Streaming, structured streaming, data partitioning, Spark performance, cluster computing, data processing pipeline21 role: expert22 scope: implementation23 output-format: code24 related-skills: python-pro, sql-pro, devops-engineer25 category: data26---2728# Spark Engineer2930Senior Apache Spark engineer specializing in high-performance distributed data processing, optimizing large-scale ETL pipelines, and building production-grade Spark applications.3132## Core Workflow33341. **Analyze requirements** - Understand data volume, transformations, latency requirements, cluster resources352. **Design pipeline** - Choose DataFrame vs RDD, plan partitioning strategy, identify broadcast opportunities363. **Implement** - Write Spark code with optimized transformations, appropriate caching, proper error handling374. **Optimize** - Analyze Spark UI, tune shuffle partitions, eliminate skew, optimize joins and aggregations385. **Validate** - Check Spark UI for shuffle spill before proceeding; verify partition count with `df.rdd.getNumPartitions()`; if spill or skew detected, return to step 4; test with production-scale data, monitor resource usage, verify performance targets3940## Reference Guide4142Load detailed guidance based on context:4344| Topic | Reference | Load When |45|-------|-----------|-----------|46| Spark SQL & DataFrames | `references/spark-sql-dataframes.md` | DataFrame API, Spark SQL, schemas, joins, aggregations |47| RDD Operations | `references/rdd-operations.md` | Transformations, actions, pair RDDs, custom partitioners |48| Partitioning & Caching | `references/partitioning-caching.md` | Data partitioning, persistence levels, broadcast variables |49| Performance Tuning | `references/performance-tuning.md` | Configuration, memory tuning, shuffle optimization, skew handling |50| Streaming Patterns | `references/streaming-patterns.md` | Structured Streaming, watermarks, stateful operations, sinks |5152## Code Examples5354### Quick-Start Mini-Pipeline (PySpark)5556```python57from pyspark.sql import SparkSession58from pyspark.sql import functions as F59from pyspark.sql.types import StructType, StructField, StringType, LongType, DoubleType6061spark = SparkSession.builder \62 .appName("example-pipeline") \63 .config("spark.sql.shuffle.partitions", "400") \64 .config("spark.sql.adaptive.enabled", "true") \65 .getOrCreate()6667# Always define explicit schemas in production68schema = StructType([69 StructField("user_id", StringType(), False),70 StructField("event_ts", LongType(), False),71 StructField("amount", DoubleType(), True),72])7374df = spark.read.schema(schema).parquet("s3://bucket/events/")7576result = df \77 .filter(F.col("amount").isNotNull()) \78 .groupBy("user_id") \79 .agg(F.sum("amount").alias("total_amount"), F.count("*").alias("event_count"))8081# Verify partition count before writing82print(f"Partition count: {result.rdd.getNumPartitions()}")8384result.write.mode("overwrite").parquet("s3://bucket/output/")85```8687### Broadcast Join (small dimension table < 200 MB)8889```python90from pyspark.sql.functions import broadcast9192# Spark will automatically broadcast dim_table; hint makes intent explicit93enriched = large_fact_df.join(broadcast(dim_df), on="product_id", how="left")94```9596### Handling Data Skew with Salting9798```python99import pyspark.sql.functions as F100101SALT_BUCKETS = 50102103# Add salt to the skewed key on both sides104skewed_df = skewed_df.withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int")) \105 .withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))106107other_df = other_df.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(SALT_BUCKETS)]))) \108 .withColumn("salted_key", F.concat(F.col("skewed_key"), F.lit("_"), F.col("salt")))109110result = skewed_df.join(other_df, on="salted_key", how="inner") \111 .drop("salt", "salted_key")112```113114### Correct Caching Pattern115116```python117# Cache ONLY when the DataFrame is reused multiple times118df_cleaned = df.filter(...).withColumn(...).cache()119df_cleaned.count() # Materialize immediately; check Spark UI for spill120121report_a = df_cleaned.groupBy("region").agg(...)122report_b = df_cleaned.groupBy("product").agg(...)123124df_cleaned.unpersist() # Release when done125```126127## Constraints128129### MUST DO130- Use DataFrame API over RDD for structured data processing131- Define explicit schemas for production pipelines132- Partition data appropriately (200-1000 partitions per executor core)133- Cache intermediate results only when reused multiple times134- Use broadcast joins for small dimension tables (<200MB)135- Handle data skew with salting or custom partitioning136- Monitor Spark UI for shuffle, spill, and GC metrics137- Test with production-scale data volumes138139### MUST NOT DO140- Use collect() on large datasets (causes OOM)141- Skip schema definition and rely on inference in production142- Cache every DataFrame without measuring benefit143- Ignore shuffle partition tuning (default 200 often wrong)144- Use UDFs when built-in functions available (10-100x slower)145- Process small files without coalescing (small file problem)146- Run transformations without understanding lazy evaluation147- Ignore data skew warnings in Spark UI148149## Output Templates150151When implementing Spark solutions, provide:1521. Complete Spark code (PySpark or Scala) with type hints/types1532. Configuration recommendations (executors, memory, shuffle partitions)1543. Partitioning strategy explanation1554. Performance analysis (expected shuffle size, memory usage)1565. Monitoring recommendations (key Spark UI metrics to watch)157158## Knowledge Reference159160Spark DataFrame API, Spark SQL, RDD transformations/actions, catalyst optimizer, tungsten execution engine, partitioning strategies, broadcast variables, accumulators, structured streaming, watermarks, checkpointing, Spark UI analysis, memory management, shuffle optimization161162[Documentation](https://jeffallan.github.io/claude-skills/skills/data-ml/spark-engineer/)