How do I install the Spark Optimization skill?

Run npx skillmds@latest add dokhacgiakhoa/spark-optimization in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Spark Optimization safe to use?

This skill has not completed SkillMD's automated safety review yet. Independent scanners report: SkillSpector: PASS, Skill Scanner: PASS. Capability flags: docs only. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Spark Optimization?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Spark Optimization free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Spark Optimization?

Dokhacgiakhoa (@dokhacgiakhoa) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Spark Optimization by dokhacgiakhoa

Name: Spark Optimization
Rating: 4.2 (5 reviews)
Author: Dokhacgiakhoa

File contents

Apache Spark Optimization

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

Do not use this skill when

The task is unrelated to apache spark optimization
You need a different domain or tool outside this scope

Instructions

Clarify goals, constraints, and required inputs.
Apply relevant best practices and validate outcomes.
Provide actionable steps and verification.
If detailed examples are required, open resources/implementation-playbook.md.

Use this skill when

Optimizing slow Spark jobs
Tuning memory and executor configuration
Implementing efficient partitioning strategies
Debugging Spark performance issues
Scaling Spark pipelines for large datasets
Reducing shuffle and data skew

Core Concepts

🧠 Knowledge Modules (Fractal Skills)

1. 1. Spark Execution Model

2. 2. Key Performance Factors

3. Pattern 1: Optimal Partitioning

4. Pattern 2: Join Optimization

5. Pattern 3: Caching and Persistence

6. Pattern 4: Memory Tuning

7. Pattern 5: Shuffle Optimization

8. Pattern 6: Data Format Optimization

9. Pattern 7: Monitoring and Debugging

10. Do's

11. Don'ts

---name: spark-optimizationdescription: Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.--- # Apache Spark Optimization Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning. ## Do not use this skill when - The task is unrelated to apache spark optimization - You need a different domain or tool outside this scope ## Instructions - Clarify goals, constraints, and required inputs. - Apply relevant best practices and validate outcomes. - Provide actionable steps and verification. - If detailed examples are required, open `resources/implementation-playbook.md`. ## Use this skill when - Optimizing slow Spark jobs - Tuning memory and executor configuration - Implementing efficient partitioning strategies - Debugging Spark performance issues - Scaling Spark pipelines for large datasets - Reducing shuffle and data skew ## Core Concepts ## 🧠 Knowledge Modules (Fractal Skills) ### 1. [1. Spark Execution Model](./sub-skills/1-spark-execution-model.md) ### 2. [2. Key Performance Factors](./sub-skills/2-key-performance-factors.md) ### 3. [Pattern 1: Optimal Partitioning](./sub-skills/pattern-1-optimal-partitioning.md) ### 4. [Pattern 2: Join Optimization](./sub-skills/pattern-2-join-optimization.md) ### 5. [Pattern 3: Caching and Persistence](./sub-skills/pattern-3-caching-and-persistence.md) ### 6. [Pattern 4: Memory Tuning](./sub-skills/pattern-4-memory-tuning.md) ### 7. [Pattern 5: Shuffle Optimization](./sub-skills/pattern-5-shuffle-optimization.md) ### 8. [Pattern 6: Data Format Optimization](./sub-skills/pattern-6-data-format-optimization.md) ### 9. [Pattern 7: Monitoring and Debugging](./sub-skills/pattern-7-monitoring-and-debugging.md) ### 10. [Do's](./sub-skills/dos.md) ### 11. [Don'ts](./sub-skills/donts.md)

Frequently asked questions

Spark Optimization