Optimizing Pyspark Jobs

Optimize slow or failing PySpark and Spark SQL jobs — partitioning and repartitioning, data skew, shuffles, broadcast joins, caching, Adaptive Query Execution, and avoiding driver collects and Python UDFs. Use when a Spark job is slow, spills, OOMs, has skewed tasks, runs a huge shuffle, or a stage hangs on a few straggler tasks.

Unknown-333 Updated 26 repo stars
Awesome Data Engineering Skills — data-engineering skills for your AI coding agent

File contents

Unknown-333/awesome-data-engineering-skills commit 35a6c77913

Frequently asked questions

npx get-skill add unknown-333/optimizing-pyspark-jobs