# Spark

> Apache Spark distributed computing. Use for big data processing.

- Skill: `majiayu000/spark` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds add majiayu000/spark`
- Raw SKILL.md: https://api.skillmd.com/api/skills/majiayu000/spark/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: majiayu000 (https://skillmd.com/u/majiayu000)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/majiayu000/spark

---


# Apache Spark

Spark is the king of Big Data. v4.0 (2024/2025) makes **Spark Connect** the default, allowing thin clients (like VS Code) to connect to massive clusters easily.

## When to Use

- **Data Engineering**: ETL at Petabyte scale.
- **Streaming**: Structured Streaming for real-time analytics.
- **Legacy ML**: `spark.ml` (though mostly replaced by XGBoost/Torch).

## Core Concepts

### Spark Connect

Decouples client (your laptop) from server (the cluster). Allows using Spark from Go/Rust/TypeScript.

### Catalyst Optimizer

Optimizes your SQL/DataFrame queries before execution.

### RDD

The low-level API. Almost never used directly in modern Spark.

## Best Practices (2025)

**Do**:

- **Use PySpark**: It is now a first-class citizen with Python UDF profiling.
- **Use Delta Lake / Iceberg**: Spark works best with modern table formats.
- **Use `pandas_udf`**: For vectorized Python UDFs.

**Don't**:

- **Don't use `rdd.map`**: It is slow (Python serialization). Use DataFrames.

## References

- [Apache Spark](https://spark.apache.org/)