vamseeachanta Mode Selection (+2)
Mode Selection
| Workload Type | Recommended Mode | Reason |
|---|---|---|
| File I/O | THREAD_POOL |
IO-bound, threads avoid GIL issues |
| Data parsing | THREAD_POOL |
Pandas releases GIL during IO |
| CPU computation | PROCESS_POOL |
Bypasses GIL for true parallelism |
| Network requests | ASYNC |
Best for many concurrent connections |
| Simple operations | SEQUENTIAL |
Overhead may exceed benefit |
Worker Count
import os
# IO-bound (reading files, network)
io_workers = os.cpu_count() * 2
# CPU-bound (heavy computation)
cpu_workers = os.cpu_count()
# Memory-constrained (large files)
memory_workers = max(2, os.cpu_count() // 2)
Batch Size
- Small files (<1MB): Large batches (500-1000)
- Medium files (1-100MB): Medium batches (50-100)
- Large files (>100MB): Small batches (10-20) or one at a time