collapse: Fast Data Transformation
Overview
collapse provides C/C++-based high-performance grouped and weighted statistics. 50-100x faster than dplyr for grouped operations, matches data.table speed while working with any data frame type (tibbles, data.tables, xts).
Core principle: Fast aggregation, transformation, and panel data operations through vectorized C code.
References
Read references/API.md before writing code.
references/API.md - Complete function referencereferences/collapse-for-tidyverse-users.md - Migration guide and patternsreferences/collapse-documentation.md - Core concepts and usagereferences/collapse-and-sf.md - Working with spatial datareferences/collapse-object-handling.md - Data structure handling
When to Use
Use collapse when:
- Dataset >100k rows
- Weighted statistics required
- Panel data (between/within transformations)
- Time series lags/diffs/growth rates
- Performance bottleneck in dplyr pipeline
Don't use:
- Small datasets (<10k rows) - dplyr is clearer
- Need arbitrary grouped functions (use dplyr)
- Working with sf (use sf and dplyr)
- Need reference semantics/in-place modification (use data.table)
- Complex joins (data.table's keyed/rolling/non-equi joins better)
vs Alternatives:
| Scenario |
Use This |
| Large grouped stats |
collapse |
| Weighted computations |
collapse |
| sf manipulation |
dplyr |
| Reference semantics |
data.table |
| Complex joins |
data.table |
| Arbitrary group functions |
dplyr |
Quick Reference
| Task |
Function/Example |
| Grouped stats |
fmean(), fsum(), fsd(), fmedian() |
| Aggregation |
collap(df, ~ by, list(fmean, fsd)) |
| Transform |
ftransform(), fmutate() |
| Selection |
fselect(), fsubset() (~100x faster) |
| Time series |
flag(), fdiff(), fgrowth() |
| Panel data |
fwithin(), fbetween(), qsu() |
| Grouping |
fgroup_by(), GRP() |
Core Pattern
library(collapse)
# Basic: grouped mean (50-100x faster than dplyr)
data |> fgroup_by(category) |> fmean()
# Weighted aggregation
data |> fgroup_by(region) |> fmean(w = weight_col)
# Multiple stats at once
collap(data, ~ category, list(fmean, fsd, fmedian))
# TRA transformations (key differentiator - single C pass)
data |> fgroup_by(id) |> fmean(TRA = "-") # Demean: subtract group mean
data |> fgroup_by(id) |> fsd(TRA = "/") # Scale: divide by group SD
data |> fgroup_by(id) |> fmean(TRA = "fill") # Fill: replace NA with group mean
# See references/API.md for full TRA options ("-", "/", "fill", "-+", "replace")
Common Mistakes
| Mistake |
Fix |
Using group_by() with collapse functions |
Use fgroup_by() or pass g = GRP(groupvar) |
collap() applies to ALL numeric columns |
Explicitly select columns before calling |
Expecting na.rm = FALSE default |
collapse defaults to na.rm = TRUE |
fwithin()/fbetween() collapse rows |
They return same # rows (centered/group means) |
| Global options affect behavior |
Set arguments explicitly in package code |
Ignoring sort = FALSE speedup |
Add sort = FALSE when order doesn't matter (3x faster) |
Advanced
See references/ for API reference, vignette content (tidyverse comparison, sf integration, object handling, development guidelines), and panel data patterns.
Validator: lib/r-validators/numerical-validator.R
Resources: Docs
1---2name: r-collapse3description: Use when code loads or uses collapse (library(collapse), collapse::), performing fast grouped or weighted statistics in R, or seeking faster alternatives to dplyr aggregation4---56# collapse: Fast Data Transformation78## Overview910**collapse provides C/C++-based high-performance grouped and weighted statistics.** 50-100x faster than dplyr for grouped operations, matches data.table speed while working with any data frame type (tibbles, data.tables, xts).1112**Core principle:** Fast aggregation, transformation, and panel data operations through vectorized C code.1314## References1516Read `references/API.md` before writing code.1718- `references/API.md` - Complete function reference19- `references/collapse-for-tidyverse-users.md` - Migration guide and patterns20- `references/collapse-documentation.md` - Core concepts and usage21- `references/collapse-and-sf.md` - Working with spatial data22- `references/collapse-object-handling.md` - Data structure handling2324## When to Use2526**Use collapse when:**2728- Dataset >100k rows29- Weighted statistics required30- Panel data (between/within transformations)31- Time series lags/diffs/growth rates32- Performance bottleneck in dplyr pipeline3334**Don't use:**3536- Small datasets (<10k rows) - dplyr is clearer37- Need arbitrary grouped functions (use dplyr)38- Working with sf (use sf and dplyr)39- Need reference semantics/in-place modification (use data.table)40- Complex joins (data.table's keyed/rolling/non-equi joins better)4142**vs Alternatives:**4344| Scenario | Use This |45| ------------------------- | ---------- |46| Large grouped stats | collapse |47| Weighted computations | collapse |48| sf manipulation | dplyr |49| Reference semantics | data.table |50| Complex joins | data.table |51| Arbitrary group functions | dplyr |5253## Quick Reference5455| Task | Function/Example |56| ----------------- | ----------------------------------------- |57| **Grouped stats** | `fmean()`, `fsum()`, `fsd()`, `fmedian()` |58| **Aggregation** | `collap(df, ~ by, list(fmean, fsd))` |59| **Transform** | `ftransform()`, `fmutate()` |60| **Selection** | `fselect()`, `fsubset()` (~100x faster) |61| **Time series** | `flag()`, `fdiff()`, `fgrowth()` |62| **Panel data** | `fwithin()`, `fbetween()`, `qsu()` |63| **Grouping** | `fgroup_by()`, `GRP()` |6465## Core Pattern6667```r68library(collapse)6970# Basic: grouped mean (50-100x faster than dplyr)71data |> fgroup_by(category) |> fmean()7273# Weighted aggregation74data |> fgroup_by(region) |> fmean(w = weight_col)7576# Multiple stats at once77collap(data, ~ category, list(fmean, fsd, fmedian))7879# TRA transformations (key differentiator - single C pass)80data |> fgroup_by(id) |> fmean(TRA = "-") # Demean: subtract group mean81data |> fgroup_by(id) |> fsd(TRA = "/") # Scale: divide by group SD82data |> fgroup_by(id) |> fmean(TRA = "fill") # Fill: replace NA with group mean83# See references/API.md for full TRA options ("-", "/", "fill", "-+", "replace")84```8586## Common Mistakes8788| Mistake | Fix |89| ------------------------------------------ | -------------------------------------------------------- |90| Using `group_by()` with collapse functions | Use `fgroup_by()` or pass `g = GRP(groupvar)` |91| `collap()` applies to ALL numeric columns | Explicitly select columns before calling |92| Expecting `na.rm = FALSE` default | collapse defaults to `na.rm = TRUE` |93| `fwithin()`/`fbetween()` collapse rows | They return same # rows (centered/group means) |94| Global options affect behavior | Set arguments explicitly in package code |95| Ignoring `sort = FALSE` speedup | Add `sort = FALSE` when order doesn't matter (3x faster) |9697## Advanced9899See `references/` for API reference, vignette content (tidyverse comparison, sf integration, object handling, development guidelines), and panel data patterns.100101**Validator:** `lib/r-validators/numerical-validator.R`102103**Resources:** [Docs](https://sebkrantz.github.io/collapse/)
arthurgailes