# Spark Best Practices

> Apache Spark best practices for PySpark and Scala distributed data processing Use when this capability is needed.

- Skill: `tomevault-io/spark-best-practices` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/spark-best-practices`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/spark-best-practices/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/spark-best-practices

---


# Apache Spark Best Practices

## Performance Optimization

### Broadcast Joins (CRITICAL)
- Use `broadcast(small_df)` for small-large table joins
- Default broadcast threshold: 10MB (`spark.sql.autoBroadcastJoinThreshold`)
- Avoid broadcast for tables > 100MB

### Shuffles (CRITICAL)
- Minimize shuffles: expensive operations
- Use `coalesce()` to reduce partitions without shuffle
- Use `repartition()` only when necessary (causes shuffle)
- Predicate pushdown: filter before joins

### Caching
- Cache DataFrames used multiple times: `df.cache()` or `df.persist()`
- Choose storage level: MEMORY_ONLY, MEMORY_AND_DISK, DISK_ONLY
- Unpersist when done: `df.unpersist()`

## Resource Management

### Executor Configuration
- Executor memory: 80% of available memory per executor
- Executor cores: 4-5 cores per executor (optimal)
- Dynamic allocation: enable for varying workloads

### Partitioning
- Optimal partition size: 100-200MB
- Too few partitions: underutilized cluster
- Too many partitions: task overhead

## Data Processing

### UDFs
- Prefer built-in functions over UDFs
- Use Pandas UDF for vectorized operations
- Avoid Python UDFs (serialization overhead)

### Storage Formats
- Parquet: default for analytics (columnar, compression)
- ORC: alternative to Parquet
- Delta/Iceberg: ACID transactions, time travel

## References
- [Spark Performance Tuning](https://spark.apache.org/docs/latest/tuning.html)

---
> Converted and distributed by [TomeVault](https://tomevault.io/claim/baekenough) — claim your Tome and manage your conversions.
<!-- tomevault:4.0:skill_md:2026-04-12 -->