# Spark Optimization

> Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.

- Skill: `dokhacgiakhoa/spark-optimization` (Agent Skill, multi-file: 12 files)
- Install (CLI): `npx skillmds@latest add dokhacgiakhoa/spark-optimization`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dokhacgiakhoa/spark-optimization/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: Dokhacgiakhoa (https://skillmd.com/u/dokhacgiakhoa)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/dokhacgiakhoa/spark-optimization

---


# Apache Spark Optimization

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

## Do not use this skill when

- The task is unrelated to apache spark optimization
- You need a different domain or tool outside this scope

## Instructions

- Clarify goals, constraints, and required inputs.
- Apply relevant best practices and validate outcomes.
- Provide actionable steps and verification.
- If detailed examples are required, open `resources/implementation-playbook.md`.

## Use this skill when

- Optimizing slow Spark jobs
- Tuning memory and executor configuration
- Implementing efficient partitioning strategies
- Debugging Spark performance issues
- Scaling Spark pipelines for large datasets
- Reducing shuffle and data skew

## Core Concepts

## 🧠 Knowledge Modules (Fractal Skills)

### 1. [1. Spark Execution Model](./sub-skills/1-spark-execution-model.md)
### 2. [2. Key Performance Factors](./sub-skills/2-key-performance-factors.md)
### 3. [Pattern 1: Optimal Partitioning](./sub-skills/pattern-1-optimal-partitioning.md)
### 4. [Pattern 2: Join Optimization](./sub-skills/pattern-2-join-optimization.md)
### 5. [Pattern 3: Caching and Persistence](./sub-skills/pattern-3-caching-and-persistence.md)
### 6. [Pattern 4: Memory Tuning](./sub-skills/pattern-4-memory-tuning.md)
### 7. [Pattern 5: Shuffle Optimization](./sub-skills/pattern-5-shuffle-optimization.md)
### 8. [Pattern 6: Data Format Optimization](./sub-skills/pattern-6-data-format-optimization.md)
### 9. [Pattern 7: Monitoring and Debugging](./sub-skills/pattern-7-monitoring-and-debugging.md)
### 10. [Do's](./sub-skills/dos.md)
### 11. [Don'ts](./sub-skills/donts.md)