Spark And Distributed Processing
Overview
Use this skill for batch workloads that exceed single-node processing and need distributed execution discipline. It helps agents reason about Spark, managed Spark services such as Glue and EMR, partitioning, joins, storage layout, and failure-safe processing.
When to Use
- building or changing
Spark jobs
- choosing between
Spark, Glue, EMR, or smaller engines
- debugging expensive joins, skew, shuffle, or memory issues
- designing batch pipelines over large files or partitioned tables
- implementing transformations against lakehouse tables such as
Iceberg, Delta, or Hudi
Do not use this for lightweight local transforms that fit comfortably in a single process.
Workflow
Confirm distributed execution is actually required.
Check:
- input volume
- latency expectations
- transformation complexity
- file sizes and partition counts
- cost compared with simpler engines
Choose the runtime intentionally.
Spark: direct control and broad ecosystem supportGlue: managed AWS-native Spark executionEMR: broader cluster control for Spark and related engines- short-lived or serverless Spark (
Lambda, serverless Glue, hard timeout ceilings): load spark-serverless-reliability-and-state-management
Design the physical plan, not just the logical one.
Account for:
- partitioning and file layout
- shuffle-heavy joins
- skewed keys
- checkpoint or intermediate persistence
- write mode and idempotency behavior
Keep data contracts visible at the edges.
Validate input assumptions before expensive execution and verify output contracts before publish.
Make backfills and reruns safe.
Historical batch recomputation should define overwrite, merge, or append semantics explicitly.
Common Rationalizations
| Rationalization |
Reality |
| "Spark will handle optimization for us." |
Engine optimizations help, but poor partitioning, skew, and write strategy still create failures or huge cost. |
| "We can just scale the cluster." |
Scaling often masks bad physical design and can still fail on skew or bad shuffles. |
| "Managed Spark means we do not need runtime design." |
Glue and EMR still require deliberate partitioning, retries, and storage strategy. |
Red Flags
Spark is chosen without a scale or latency reason- partitioning and write strategy are undefined
- joins ignore skewed business keys
- rerun behavior for historical loads is unclear
- distributed compute is used where a smaller engine would be simpler and cheaper
Verification
1---2name: spark-and-distributed-processing3description: Guides agents through batch and distributed data processing design. Use when implementing or reviewing Spark-based pipelines, or managed distributed runtimes such as Glue and EMR.4---56# Spark And Distributed Processing78## Overview910Use this skill for batch workloads that exceed single-node processing and need distributed execution discipline. It helps agents reason about `Spark`, managed Spark services such as `Glue` and `EMR`, partitioning, joins, storage layout, and failure-safe processing.1112## When to Use1314- building or changing `Spark` jobs15- choosing between `Spark`, `Glue`, `EMR`, or smaller engines16- debugging expensive joins, skew, shuffle, or memory issues17- designing batch pipelines over large files or partitioned tables18- implementing transformations against lakehouse tables such as `Iceberg`, `Delta`, or `Hudi`1920Do not use this for lightweight local transforms that fit comfortably in a single process.2122## Workflow23241. Confirm distributed execution is actually required.25 Check:26 - input volume27 - latency expectations28 - transformation complexity29 - file sizes and partition counts30 - cost compared with simpler engines31322. Choose the runtime intentionally.33 - `Spark`: direct control and broad ecosystem support34 - `Glue`: managed AWS-native Spark execution35 - `EMR`: broader cluster control for Spark and related engines36 - short-lived or serverless Spark (`Lambda`, serverless `Glue`, hard timeout ceilings): load `spark-serverless-reliability-and-state-management`37383. Design the physical plan, not just the logical one.39 Account for:40 - partitioning and file layout41 - shuffle-heavy joins42 - skewed keys43 - checkpoint or intermediate persistence44 - write mode and idempotency behavior45464. Keep data contracts visible at the edges.47 Validate input assumptions before expensive execution and verify output contracts before publish.48495. Make backfills and reruns safe.50 Historical batch recomputation should define overwrite, merge, or append semantics explicitly.5152## Common Rationalizations5354| Rationalization | Reality |55| --- | --- |56| "Spark will handle optimization for us." | Engine optimizations help, but poor partitioning, skew, and write strategy still create failures or huge cost. |57| "We can just scale the cluster." | Scaling often masks bad physical design and can still fail on skew or bad shuffles. |58| "Managed Spark means we do not need runtime design." | `Glue` and `EMR` still require deliberate partitioning, retries, and storage strategy. |5960## Red Flags6162- `Spark` is chosen without a scale or latency reason63- partitioning and write strategy are undefined64- joins ignore skewed business keys65- rerun behavior for historical loads is unclear66- distributed compute is used where a smaller engine would be simpler and cheaper6768## Verification6970- [ ] The runtime choice is justified against workload needs71- [ ] Partitioning, join strategy, and write semantics are explicit72- [ ] Contracts and quality checks exist at input and output boundaries73- [ ] Backfill and retry behavior are documented for the distributed job
vaquarkhan