# Spark

> Apache Spark

- Skill: `neuralblitz/spark` (Agent Skill)
- Install (CLI): `npx skillmds@latest add neuralblitz/spark`
- Raw SKILL.md: https://api.skillmd.com/api/skills/neuralblitz/spark/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: NeuralBlitz (https://skillmd.com/u/neuralblitz)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/neuralblitz/spark

---

# Apache Spark

Apache Spark is a distributed computing system providing APIs for large-scale data processing. It offers in-memory computing, lazy evaluation, and unified API for batch and streaming.

## Key Concepts

- RDDs and DataFrames
- Transformations and actions
- Spark SQL
- Streaming (structured)
- MLlib for machine learning

## Common Use Cases

- Big data processing
- ETL pipelines
- Real-time analytics
- Machine learning at scale
- Data lake processing

## Best Practices

- Use DataFrames over RDDs
- Implement proper partitioning
- Cache judiciously
- Use broadcast variables
- Monitor with Spark UI

## Resources

- Spark.apache.org: spark.apache.org/docs
- Related Skills: data-engineering, scala, python, hadoop