Dask Parallel and Distributed Computing
Scale pandas/NumPy workflows beyond memory and across clusters.
When to Use
- Datasets exceed available RAM
- Need to parallelize pandas or NumPy operations
- Processing multiple files efficiently (CSVs, Parquet)
- Building custom parallel workflows
- Distributing workloads across multiple cores/machines
Dask Collections
| Collection |
Like |
Use Case |
| DataFrame |
pandas |
Tabular data, CSV/Parquet |
| Array |
NumPy |
Numerical arrays, matrices |
| Bag |
list |
Unstructured data, JSON logs |
| Delayed |
Custom |
Arbitrary Python functions |
Key concept: All collections are lazy—computation happens only when you call .compute().
Lazy Evaluation
| Function |
Behavior |
Use |
dd.read_csv() |
Lazy load |
Large CSVs |
dd.read_parquet() |
Lazy load |
Large Parquet |
| Operations |
Build graph |
Chain transforms |
.compute() |
Execute |
Get final result |
Key concept: Dask builds a task graph of operations, optimizes it, then executes in parallel. Call .compute() once at the end, not after every operation.
Schedulers
| Scheduler |
Best For |
Start |
| threaded |
NumPy/Pandas (releases GIL) |
Default |
| processes |
Pure Python (GIL bound) |
scheduler='processes' |
| synchronous |
Debugging |
scheduler='synchronous' |
| distributed |
Monitoring, scaling, clusters |
Client() |
Distributed Scheduler
| Feature |
Benefit |
| Dashboard |
Real-time progress monitoring |
| Cluster scaling |
Add/remove workers |
| Fault tolerance |
Retry failed tasks |
| Worker resources |
Memory management |
Chunking Concepts
DataFrame Partitions
| Concept |
Description |
| Partition |
Subset of rows (like a mini DataFrame) |
| npartitions |
Number of partitions |
| divisions |
Index boundaries between partitions |
Array Chunks
| Concept |
Description |
| Chunk |
Subset of array (n-dimensional block) |
| chunks |
Tuple of chunk sizes per dimension |
| Optimal size |
~100 MB per chunk |
Key concept: Chunk size is critical. Too small = scheduling overhead. Too large = memory issues. Target ~100 MB.
DataFrame Operations
Supported (parallel)
| Category |
Operations |
| Selection |
filter, loc, column selection |
| Aggregation |
groupby, sum, mean, count |
| Transforms |
apply (row-wise), map_partitions |
| Joins |
merge, join (shuffles data) |
| I/O |
read_csv, read_parquet, to_parquet |
Avoid or Use Carefully
| Operation |
Issue |
Alternative |
iterrows |
Kills parallelism |
map_partitions |
apply(axis=1) |
Slow |
map_partitions |
Repeated compute() |
Inefficient |
Single compute() at end |
sort_values |
Expensive shuffle |
Avoid if possible |
Common Patterns
ETL Pipeline
scan_* or read_* (lazy load)- Chain filters and transforms
- Single
.compute() or .to_parquet()
Multi-File Processing
| Pattern |
Description |
| Glob patterns |
dd.read_csv('data/*.csv') |
| Partition per file |
Natural parallelism |
| Output partitioned |
to_parquet('output/') |
Custom Operations
| Method |
Use Case |
map_partitions |
Apply function to each partition |
map_blocks |
Apply function to each array block |
delayed |
Wrap arbitrary Python functions |
Best Practices
| Practice |
Why |
| Don't load locally first |
Let Dask handle loading |
| Single compute() at end |
Avoid redundant computation |
| Use Parquet |
Faster than CSV, columnar |
| Match partition to files |
One partition per file |
| Check task graph size |
len(ddf.__dask_graph__()) < 100k |
| Use distributed for debugging |
Dashboard shows progress |
Common Pitfalls
| Pitfall |
Solution |
| Loading with pandas first |
Use dd.read_* directly |
| compute() in loops |
Collect all, single compute() |
| Too many partitions |
Repartition to ~100 MB each |
| Memory errors |
Reduce chunk size, add workers |
| Slow shuffles |
Avoid sorts/joins when possible |
vs Alternatives
| Tool |
Best For |
Trade-off |
| Dask |
Scale pandas/NumPy, clusters |
Setup complexity |
| Polars |
Fast in-memory |
Must fit in RAM |
| Vaex |
Out-of-core single machine |
Limited operations |
| Spark |
Enterprise, SQL-heavy |
Infrastructure |
Resources
1---2name: dask3description: Use when "Dask", "parallel computing", "distributed computing", "larger than memory", or asking about "parallel pandas", "parallel numpy", "out-of-core", "multi-file processing", "cluster computing", "lazy evaluation dataframe"4---5
6# Dask Parallel and Distributed Computing
7
8Scale pandas/NumPy workflows beyond memory and across clusters.
9
10## When to Use
11
12- Datasets exceed available RAM
13- Need to parallelize pandas or NumPy operations
14- Processing multiple files efficiently (CSVs, Parquet)
15- Building custom parallel workflows
16- Distributing workloads across multiple cores/machines
17
18---
19
20## Dask Collections
21
22| Collection | Like | Use Case |
23|------------|------|----------|
24| **DataFrame** | pandas | Tabular data, CSV/Parquet |
25| **Array** | NumPy | Numerical arrays, matrices |
26| **Bag** | list | Unstructured data, JSON logs |
27| **Delayed** | Custom | Arbitrary Python functions |
28
29**Key concept**: All collections are lazy—computation happens only when you call `.compute()`.
30
31---
32
33## Lazy Evaluation
34
35| Function | Behavior | Use |
36|----------|----------|-----|
37| `dd.read_csv()` | Lazy load | Large CSVs |
38| `dd.read_parquet()` | Lazy load | Large Parquet |
39| Operations | Build graph | Chain transforms |
40| `.compute()` | Execute | Get final result |
41
42**Key concept**: Dask builds a task graph of operations, optimizes it, then executes in parallel. Call `.compute()` once at the end, not after every operation.
43
44---
45
46## Schedulers
47
48| Scheduler | Best For | Start |
49|-----------|----------|-------|
50| **threaded** | NumPy/Pandas (releases GIL) | Default |
51| **processes** | Pure Python (GIL bound) | `scheduler='processes'` |
52| **synchronous** | Debugging | `scheduler='synchronous'` |
53| **distributed** | Monitoring, scaling, clusters | `Client()` |
54
55### Distributed Scheduler
56
57| Feature | Benefit |
58|---------|---------|
59| Dashboard | Real-time progress monitoring |
60| Cluster scaling | Add/remove workers |
61| Fault tolerance | Retry failed tasks |
62| Worker resources | Memory management |
63
64---
65
66## Chunking Concepts
67
68### DataFrame Partitions
69
70| Concept | Description |
71|---------|-------------|
72| **Partition** | Subset of rows (like a mini DataFrame) |
73| **npartitions** | Number of partitions |
74| **divisions** | Index boundaries between partitions |
75
76### Array Chunks
77
78| Concept | Description |
79|---------|-------------|
80| **Chunk** | Subset of array (n-dimensional block) |
81| **chunks** | Tuple of chunk sizes per dimension |
82| **Optimal size** | ~100 MB per chunk |
83
84**Key concept**: Chunk size is critical. Too small = scheduling overhead. Too large = memory issues. Target ~100 MB.
85
86---
87
88## DataFrame Operations
89
90### Supported (parallel)
91
92| Category | Operations |
93|----------|------------|
94| **Selection** | `filter`, `loc`, column selection |
95| **Aggregation** | `groupby`, `sum`, `mean`, `count` |
96| **Transforms** | `apply` (row-wise), `map_partitions` |
97| **Joins** | `merge`, `join` (shuffles data) |
98| **I/O** | `read_csv`, `read_parquet`, `to_parquet` |
99
100### Avoid or Use Carefully
101
102| Operation | Issue | Alternative |
103|-----------|-------|-------------|
104| `iterrows` | Kills parallelism | `map_partitions` |
105| `apply(axis=1)` | Slow | `map_partitions` |
106| Repeated `compute()` | Inefficient | Single `compute()` at end |
107| `sort_values` | Expensive shuffle | Avoid if possible |
108
109---
110
111## Common Patterns
112
113### ETL Pipeline
114
1151. `scan_*` or `read_*` (lazy load)
1162. Chain filters and transforms
1173. Single `.compute()` or `.to_parquet()`
118
119### Multi-File Processing
120
121| Pattern | Description |
122|---------|-------------|
123| Glob patterns | `dd.read_csv('data/*.csv')` |
124| Partition per file | Natural parallelism |
125| Output partitioned | `to_parquet('output/')` |
126
127### Custom Operations
128
129| Method | Use Case |
130|--------|----------|
131| `map_partitions` | Apply function to each partition |
132| `map_blocks` | Apply function to each array block |
133| `delayed` | Wrap arbitrary Python functions |
134
135---
136
137## Best Practices
138
139| Practice | Why |
140|----------|-----|
141| Don't load locally first | Let Dask handle loading |
142| Single compute() at end | Avoid redundant computation |
143| Use Parquet | Faster than CSV, columnar |
144| Match partition to files | One partition per file |
145| Check task graph size | `len(ddf.__dask_graph__())` < 100k |
146| Use distributed for debugging | Dashboard shows progress |
147
148---
149
150## Common Pitfalls
151
152| Pitfall | Solution |
153|---------|----------|
154| Loading with pandas first | Use `dd.read_*` directly |
155| compute() in loops | Collect all, single compute() |
156| Too many partitions | Repartition to ~100 MB each |
157| Memory errors | Reduce chunk size, add workers |
158| Slow shuffles | Avoid sorts/joins when possible |
159
160---
161
162## vs Alternatives
163
164| Tool | Best For | Trade-off |
165|------|----------|-----------|
166| **Dask** | Scale pandas/NumPy, clusters | Setup complexity |
167| **Polars** | Fast in-memory | Must fit in RAM |
168| **Vaex** | Out-of-core single machine | Limited operations |
169| **Spark** | Enterprise, SQL-heavy | Infrastructure |
170
171## Resources
172
173- Docs: <https://docs.dask.org/>
174- Best Practices: <https://docs.dask.org/en/stable/best-practices.html>
175- Examples: <https://examples.dask.org/>
eyadsibai