Big data processing for research
"Big" starts where the current tool breaks: the dataset that no
longer fits in RAM, the loop over files that no longer finishes
overnight. The escalation path matters more than any framework -
each step up costs complexity, debuggability and reproducibility,
so take the smallest step that works.
The escalation ladder
- Optimize in place first: columnar formats with predicate
pushdown (Parquet - rseng-scientific-file-formats), dtype
downcasting, reading only needed columns; profile before scaling
(rseng-performance-profiling) - many "big data" problems are
memory-layout problems.
- Out-of-core on one machine: chunked iteration (read-process-
write per chunk), memory-mapped arrays, or Dask's lazy
collections on a single node. One machine with streaming
processing handles far more than intuition suggests, with none
of the distributed complexity.
- Embarrassingly parallel batch: independent per-file/per-chunk
jobs as cluster job arrays (rseng-hpc-computing) or a workflow
engine (rseng-workflows) - the RIGHT answer for most research
sweeps, and simpler than any framework.
- Distributed frameworks: Dask (scales NumPy/pandas/xarray idioms;
native in the Pangeo geoscience stack) or Spark (SQL-flavored
tabular pipelines, industry-standard cluster tooling) when
computation genuinely needs cross-partition coordination:
shuffles, joins, global aggregations over larger-than-node data.
Skipping straight to step 4 is the classic mistake: a distributed
job that could have been a job array is slower to build, harder to
debug and harder to reproduce.
Patterns that make batch processing trustworthy
- Idempotent tasks: running a task twice yields the same result -
write to output paths derived from inputs and parameters, never
append blindly.
- Restartable pipelines: skip work whose outputs already exist
(checkpointing at the task level), so a failure at file 90,000
costs minutes, not the weekend. Workflow engines give this for
free (rseng-workflows).
- Fail loudly per item, not globally: quarantine failing inputs
with logged reasons and continue; a summary of 37 failures beats
a crash at the first.
- Validate at the boundaries: schema/sanity checks on ingest and
before final aggregation - silent corruption scales with the data
(rseng-data-management).
- Deterministic partitioning and seeds where randomness exists, so
reruns are comparable.
Framework-specific footguns
- Lazy evaluation (Dask, Spark) means errors surface at compute
time, far from their cause: materialize small samples early while
developing; keep transformations testable on in-memory subsets
(rseng-testing) - the same code path at toy scale is the unit test.
- Partition sizing dominates performance: too many tiny partitions
drown in scheduling overhead, too few lose parallelism; target
the framework's recommended per-partition sizes and re-partition
after heavy filters.
- Shuffles (joins, groupbys across partitions) are the expensive
operations - restructure to avoid them where possible, and
broadcast small tables instead of joining large-to-large.
- Cluster resources: match worker memory to partition size,
and on shared clusters run the framework's scheduler inside the
allocation (rseng-hpc-computing) rather than assuming the machine.
- Record framework and cluster configuration with results -
distributed runs are part of the provenance
(rseng-reproducible-environments, rseng-ai-declaration for
AI-assisted pipeline work).
Working with this skill
This skill is source-independent: its authority is the framework
documentation and community practice linked below.
Learn more (verified):
Related skills
Check whether any of these applies before moving on:
- rseng-green-computing - distributed runs carry energy cost
- rseng-hpc-computing - job arrays and cluster allocations
- rseng-performance-profiling - profile before scaling out
- rseng-scientific-file-formats - Parquet and chunked stores enable it
- rseng-testing - test transforms on in-memory subsets
- rseng-workflows - restartable pipelines via engines
1---2name: rseng-big-data-processing3description: Covers processing research data that outgrows one machine's memory: out-of-core and chunked computation, Dask for scaling the scientific Python stack, Spark for distributed tabular pipelines, lazy evaluation, partitioning strategies, idempotent and restartable batch jobs, and knowing when NOT to distribute. Use when datasets no longer fit in memory, when the user mentions Dask, Spark, out-of-core or larger-than-memory data, when a pandas/NumPy workflow hits memory limits, or when designing batch pipelines over many files. (Cluster job submission and job arrays are rseng-hpc-computing; pipeline orchestration engines are rseng-workflows; profile first with rseng-performance-profiling.)4license: CC-BY-4.05---67# Big data processing for research89"Big" starts where the current tool breaks: the dataset that no10longer fits in RAM, the loop over files that no longer finishes11overnight. The escalation path matters more than any framework -12each step up costs complexity, debuggability and reproducibility,13so take the smallest step that works.1415## The escalation ladder16171. Optimize in place first: columnar formats with predicate18 pushdown (Parquet - rseng-scientific-file-formats), dtype19 downcasting, reading only needed columns; profile before scaling20 (rseng-performance-profiling) - many "big data" problems are21 memory-layout problems.222. Out-of-core on one machine: chunked iteration (read-process-23 write per chunk), memory-mapped arrays, or Dask's lazy24 collections on a single node. One machine with streaming25 processing handles far more than intuition suggests, with none26 of the distributed complexity.273. Embarrassingly parallel batch: independent per-file/per-chunk28 jobs as cluster job arrays (rseng-hpc-computing) or a workflow29 engine (rseng-workflows) - the RIGHT answer for most research30 sweeps, and simpler than any framework.314. Distributed frameworks: Dask (scales NumPy/pandas/xarray idioms;32 native in the Pangeo geoscience stack) or Spark (SQL-flavored33 tabular pipelines, industry-standard cluster tooling) when34 computation genuinely needs cross-partition coordination:35 shuffles, joins, global aggregations over larger-than-node data.3637Skipping straight to step 4 is the classic mistake: a distributed38job that could have been a job array is slower to build, harder to39debug and harder to reproduce.4041## Patterns that make batch processing trustworthy4243- Idempotent tasks: running a task twice yields the same result -44 write to output paths derived from inputs and parameters, never45 append blindly.46- Restartable pipelines: skip work whose outputs already exist47 (checkpointing at the task level), so a failure at file 90,00048 costs minutes, not the weekend. Workflow engines give this for49 free (rseng-workflows).50- Fail loudly per item, not globally: quarantine failing inputs51 with logged reasons and continue; a summary of 37 failures beats52 a crash at the first.53- Validate at the boundaries: schema/sanity checks on ingest and54 before final aggregation - silent corruption scales with the data55 (rseng-data-management).56- Deterministic partitioning and seeds where randomness exists, so57 reruns are comparable.5859## Framework-specific footguns6061- Lazy evaluation (Dask, Spark) means errors surface at compute62 time, far from their cause: materialize small samples early while63 developing; keep transformations testable on in-memory subsets64 (rseng-testing) - the same code path at toy scale is the unit test.65- Partition sizing dominates performance: too many tiny partitions66 drown in scheduling overhead, too few lose parallelism; target67 the framework's recommended per-partition sizes and re-partition68 after heavy filters.69- Shuffles (joins, groupbys across partitions) are the expensive70 operations - restructure to avoid them where possible, and71 broadcast small tables instead of joining large-to-large.72- Cluster resources: match worker memory to partition size,73 and on shared clusters run the framework's scheduler inside the74 allocation (rseng-hpc-computing) rather than assuming the machine.75- Record framework and cluster configuration with results -76 distributed runs are part of the provenance77 (rseng-reproducible-environments, rseng-ai-declaration for78 AI-assisted pipeline work).7980## Working with this skill8182This skill is source-independent: its authority is the framework83documentation and community practice linked below.8485Learn more (verified):86 - https://www.dask.org - Dask87 - https://spark.apache.org - Apache Spark88 - https://pangeo.io - Pangeo community practice for large-scale89 scientific data90 - https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009757 -91 Ten Simple Rules for large-scale data processing9293<!-- related-skills:begin -->9495## Related skills9697Check whether any of these applies before moving on:9899- rseng-green-computing - distributed runs carry energy cost100- rseng-hpc-computing - job arrays and cluster allocations101- rseng-performance-profiling - profile before scaling out102- rseng-scientific-file-formats - Parquet and chunked stores enable it103- rseng-testing - test transforms on in-memory subsets104- rseng-workflows - restartable pipelines via engines105106<!-- related-skills:end -->
fdiblen