collapse-r

name

collapse-r

description

Expert guidance for the collapse R package (v2.x) — fast, class-agnostic data transformation and statistical computing in R, and the core of the fastverse. Covers the Fast Statistical Functions (fmean/fsum/fmedian/fmode/fnth/fndistinct … with their g/w/TRA arguments), grouping objects (GRP, qF, qG, group, fgroup_by, gsplit), the dplyr-like verbs (fsubset/fselect/fmutate/fsummarise/ across/roworder/frename/fslice/fcount), join(), pivot(), collap(), panel and time-series operations (flag/L, fdiff/D, fgrowth/G, fcumsum, findex_by, psmat), summaries (qsu, descr, qtab, pwcor, varying, namlab/vlabels), list processing (rsplit, unlist2d, get_elem, rapply2d, t_list), memory-efficient programming helpers (setv, whichv, %==%, %+=%, %iin%, alloc, copyv, setop, ss), quick conversions (qDF/qDT/qM/qTBL/mctl/mrtl), and namespace masking via set_collapse(). Use this skill WHENEVER the user is manipulating, aggregating, reshaping, joining, summarising, or transforming data in R and collapse or the fastverse is available or already used in the project; whenever they mention collapse, fastverse, set_collapse, any f-prefixed verb, TRA, GRP, fgroup_by, qDT/qM, collap, pivot, or the L/D/G/W/B/STD/HDW operators; whenever they need grouped/weighted statistics, panel or within/between transformations, or fast joins/reshapes on large data; and whenever they are writing an R package that uses collapse as a computational backend. Prefer this over ad-hoc dplyr/base R so code matches the established fastverse idiom — also when the user just says "aggregate/reshape/join/clean this in R" without naming a package.

collapse — fast data transformation and statistical computing in R

collapse is a C/C++ package that makes statistical operations vectorized across columns and across groups at once. That single idea is what separates it from dplyr/data.table: instead of splitting data by group and applying a function, you pass the grouping to the function. fmean(x, g) computes all group means in one C pass. Everything else — the verbs, the joins, the reshapes — exists to feed those functions efficiently.

It is also class-agnostic: the same code works on vectors, matrices, data.frames, data.tables, tibbles, grouped_df, sf, pseries/pdata.frame, xts/zoo, and units objects, preserving their attributes. You never need to convert.

Current version is 2.1.7. Check with packageVersion("collapse"); join(), pivot(), and set_collapse() require ≥ 2.0.0, fslice()/groupv() ≥ 2.1.0, and GRP(drop = FALSE)/fcount(drop=) ≥ 2.1.7.

Decide the namespace before writing a line

This is the highest-leverage decision and the most common source of broken code. By default collapse exports f-prefixed names only: fsubset, fselect, fmutate, fsummarise, fgroup_by, frename, ftransform. Bare mutate, subset, select, group_by, summarise, n() are not available from a plain library(collapse).

# --- Analysis / research scripts: mask, so the code reads like dplyr ---
library(fastverse)                                    # or library(collapse)
set_collapse(mask = c("manip", "helper", "special"),  # the standard trio
             nthreads = 4, sort = FALSE)
fastverse_conflicts()                                 # report what got masked

• "manip" → subset, select, mutate, summarise, transform, group_by, rename, slice, compute, index_by, ungroup, …
• "helper" → unique, duplicated, count, quantile, range, dist, droplevels, interaction, match, nrow, ncol, …
• "special" → n() (= GRPN), table() (= qtab), and a fast %in%.
• "all" additionally masks the statistical and transformation functions — sum, mean, median, min, max, sd, var, mode, first, last, scale, diff, cumsum, between, within, … (59 names in total). Powerful, but it silently turns every sum()/mean() inside fmutate/fsummarise into a vectorized grouped call, and shadows base scale/diff/within project-wide. Only use it deliberately, and reach for base::sum() when you want the split-apply behaviour back.

# --- Package code: NEVER call set_collapse(mask=) — it leaks into user sessions ---
#' @importFrom collapse fsubset fmutate fsum GRP get_vars add_vars

If you are editing an existing project, read the top of a nearby script and mirror its choice. Writing mutate() into a script that never called set_collapse(mask = "manip") produces a dplyr call (or an error), silently losing all the performance the code was written for.

Do not call set_collapse() on the user's behalf

set_collapse() is global and session-wide: it changes defaults for every later call, in the user's code and in every other package that uses collapse. Options like sort and na.rm change results, not just speed, and mask rewrites the exported namespace and moves collapse to the top of the search path. None of that is yours to decide silently.

So: write code that works under whatever settings are already in force. Add a set_collapse() call only when

• the user asked for it ("make this faster", "set up masking", "use 4 threads"), or
• you are writing a new standalone script and are setting its header — in which case say so in one line, and prefer mirroring a sibling script's header over inventing one, or
• the project already sets it and you are matching that convention.

Never add it to package code (see references/programming.md), never inside a function without on.exit() restoring the old value, and do not sprinkle nthreads/na.rm/sort into individual calls as a reflex — reach for them when the job is large enough to care. The complete option list is in references/efficiency.md.

House style: full names, piped, one verb per line

Two rules, both about the next person to read the code.

1. Write the full function names. collapse ships terse aliases — slt, sbt, gby, smr, mtt, tfm, tfmv, settfm, settfmv, gv, gvr, nv, av, rnm, iby, ix, itn. Do not write them. Use fselect, fsubset, fgroup_by, fsummarise, fmutate, ftransform, ftransformv, settransform, settransformv, get_vars, get_vars(..., regex = TRUE), num_vars, add_vars, frename, findex_by, findex, finteraction.

2. Pipe; do not nest. Every collapse verb takes the data as its first argument, so chains read top-to-bottom with one verb per line. Nesting them inside-out is how collapse code gets its undeserved reputation for being cryptic.

smr(sbt(d, x > 0, id, v), m = fmean(v))     # don't — aliases, and inside-out

d |>                                         # do
  fsubset(x > 0, id, v) |>
  fsummarise(m = fmean(v))

The two together are what make a pipeline skimmable — each line is one verb, and each verb says what it does:

exports |>
  fsubset(year > 2010, country, sector, value) |>
  fgroup_by(country, sector) |>
  fsummarise(value = fsum(value)) |>
  roworder(-value)

Use the native |>. Reach for magrittr's %>% only when you actually need the . placeholder (an argument other than the first, or referring to the data twice in one step) — and when a step needs . twice, an intermediate variable is usually clearer than a %>% {...} block.

The aliases are identical in behaviour, so none of this changes results — set_collapse(remove = "shorthand") deletes them from the namespace entirely if you want the rule enforced rather than remembered. You still need to recognize them when reading existing code; the alias table is in references/data-manipulation.md.

Two names that look like shorthands but are not, and should be used freely: ss() (the programmer's [, not an alias for fsubset) and .c() (non-standard concatenation).

3. Write the direct form, not the defensive one. The most common way to write bad collapse code is not getting it wrong — it is reaching past the simple call for something more elaborate:

Reflex	Direct form
get_vars(d, c("a", "b"))	fselect(d, a, b) — quote names only when they come from a variable
fsummarise(across(cols, fmean))	fgroup_by(g) |> get_vars(cols) |> fmean() when every column gets the same statistic
value / fsum(value, g, TRA = "fill")	fsum(value, g, TRA = "/") — one pass, no extra copy
join(x, y, on = "id", validate = "1:1")	join(x, y, on = "id") — validation costs a pass; add it when the check is wanted, not by reflex
get_vars(d, cols) / d$total	get_vars(d, cols) %c/% d$total, or settransformv(d, cols, `/`, total)
rowSums(!is.na(get_vars(d, cols))) > 0	!missing_cases(d, cols, prop = 1)
zero-filling NA weights so fmedian() stops erroring	let it error, or drop the rows deliberately

The point is not brevity for its own sake — each left-hand form allocates something the right-hand form doesn't, and hides a decision the reader would want to see.

The core mechanic: g, w, TRA

Every Fast Statistical Function — fsum fprod fmean fmedian fmode fvar fsd fmin fmax fnth ffirst flast fnobs fndistinct — has the same signature:

f<stat>(x, g = NULL, w = NULL, TRA = NULL, na.rm = TRUE, use.g.names = TRUE, nthreads = 1)

• x — vector, matrix, data.frame, or grouped_df. One function, all shapes.
• g — grouping: a vector, list of vectors, factor, qG, or GRP object.
• w — weights. fmean(x, w = pop) is a weighted mean; fmedian, fmode, fnth, fvar, fsd are all weighted too.
• TRA — instead of returning one value per group, sweep the statistic back onto the data at full length. This is the feature people miss most.

fmean(mtcars$mpg, mtcars$cyl)                 # aggregate:  3 group means
fmean(mtcars$mpg, mtcars$cyl, TRA = "fill")   # expand:     32 values, group mean
fmean(mtcars$mpg, mtcars$cyl, TRA = "-")      # centre by group mean
fsum(x, list(country, year), TRA = "/")       # share of group total — no join needed

TRA operations: "replace_na"/"na", "fill", "replace", "-", "-+", "/", "%", "+", "*", "%%", "-%%". "fill" replaces missing values too, "replace" keeps them missing. Add set = TRUE to modify by reference, no copy: fmedian(df$x, df$g, TRA = "fill", set = TRUE).

Named shortcuts for the common cases: fbetween/B (= TRA = "fill" of the mean), fwithin/W (= TRA = "-"), fscale/STD (grouped standardization), fhdwithin/HDW (centre on multiple factors / partial out a regression).

Eager vectorization inside fmutate/fsummarise

collapse reads the expression you type. If it contains a Fast Statistical Function, the whole expression is executed once with the grouping passed down; otherwise it falls back to split-apply-combine via gsplit().

mtcars |> fgroup_by(cyl) |> fsummarise(x = fmean(mpg) + fmin(qsec))  # fmean(mpg,g) + fmin(qsec,g)
mtcars |> fgroup_by(cyl) |> fsummarise(x = fmean(mpg) +  min(qsec))  # grouped mean + OVERALL min (!)
mtcars |> fgroup_by(cyl) |> fsummarise(x =  mean(mpg) +  min(qsec))  # split-apply-combine, per group

This is intentional and lets you vectorize complex expressions, but it means mixing prefixed and unprefixed statistics in one expression is almost always a bug. Only expressions you type out are inspected — a statistic hidden inside myfun <- function(x) fmean(x) + fmin(x) is not vectorized.

Prefer to avoid fgroup_by for mutate-style work entirely — internal grouping is faster and reads fine:

dat |> fmutate(med = fmedian(mpg, list(cyl, vs, am), TRA = "fill"))   # best
dat |> fgroup_by(cyl, vs, am, return.groups = FALSE) |> fmutate(...)  # if reusing the grouping

Which function do I need?

Task	Function
Filter rows (+ select columns in the same call)	fsubset(d, cond, col1, col2)
Filter by position, programmatically	ss(d, i, j, check = FALSE)
Select / drop columns	fselect(d, a, b, -c); by name/index/predicate get_vars, by regex get_vars(d, re, regex = TRUE), by type num_vars, cat_vars, char_vars
Add / modify columns	fmutate (sequential, across), ftransform (parallel), settransform (by reference)
Keep only computed columns	fcompute(d, ...) ; fmutate(..., .keep = "used")
Apply a function to many columns	across() inside fmutate/fsummarise; ftransformv(d, cols, FUN, ..., apply = FALSE); or dapply
Aggregate	fgroup_by() then fsummarise(), or f<stat>(d, g), or collap() for multi-type
Weighted statistics	the w = argument: fmean, fmedian, fnth, fmode, fsd, fvar, fquantile, collap, qsu, pwcor
Centre / demean / standardize by group	fwithin/W, fbetween/B, fscale/STD; multi-factor fhdwithin/HDW
Share / percentage of a group total	f<stat>(x, g, TRA = "/") — no join, no fgroup_by
Sort rows / columns	, ; ,

Gotchas that bite immediately

1. na.rm = TRUE is the default everywhere — the opposite of base R. Set set_collapse(na.rm = FALSE) if the data has no missing values (it is also faster).
2. join() takes only the first match by default (multiple = FALSE), does not sort (sort = FALSE preserves x order), and does not validate (validate = "m:m"). For a dplyr-left_join-like cartesian result you must pass multiple = TRUE. Use validate = "1:1" / require = list(x = 0.9) to assert what you expect.
3. fsubset's ... are columns to keep, not extra conditions: fsubset(d, x > 1 & y < 2, id, x, y). Combine conditions with &.
4. No tidyselect. No starts_with(), where(), c_across(), no ~ .x lambdas. Use get_vars(d, "^pt_", regex = TRUE) for regex, across(is.numeric, fmean) for predicates, and plain function names or \(x) ... for anonymous functions.
5. across() is not a real function — it is an expression parsed by fmutate/fsummarise. collapse::across() does not work.
6. fsubset() is not allowed on grouped data and roworder() messages about it — this is deliberate: subset and sort before grouping.
7. qDF/qDT/qM drop non-essential attributes by default. Pass keep.attr = TRUE if you need them.
8. set_collapse(sort = FALSE) changes result ordering globally, including inside packages that use collapse. Great for speed, but pass a sorted factor (qF(g, sort = TRUE)) where output order matters.
9. Infix operators evaluate strictly left to right — a %+=% b * 2 is not what you think. Parenthesize.
10. Operators L/D///// add name prefixes to matrix/data.frame columns (, ). Their counterparts (, , , …) do not, unless .

references/gotchas.md has the full list, including panel/time handling and attribute-preservation surprises.

Reference files

Read the one that matches the task rather than guessing at arguments:

• references/statistical-functions.md — the Fast Statistical Functions in detail, the full TRA table, weights, fscale/fbetween/fwithin/fhdwithin, dapply/BY, flm/fFtest, fquantile/frange/fdist.
• references/data-manipulation.md — every verb with its real signature: subsetting, selection, mutate/transform/compute, across, summarise, ordering, renaming, unique/count, quick conversions.
• references/join-pivot-aggregate.md — join() semantics and diagnostics, pivot() in all three directions (including labels), collap() for multi-type and weighted aggregation, rowbind().
• references/time-series-panel.md — flag/fdiff/fgrowth/fcumsum, why passing t matters on irregular panels, indexed frames, psmat/psacf, seqid/timeid, xts/plm interop.
• references/summaries-and-lists.md — qsu, descr, qtab, pwcor, varying, labels (vlabels/namlab/setrelabel), and the list-processing toolkit (rsplit, unlist2d, get_elem, rapply2d, t_list).
• references/efficiency.md — writing fast collapse code: sequencing, minimal grouping, sort/na.rm/nthreads, controlling what gets materialized, computing by reference (chained set = TRUE ops, iterative fitting loops, the aliasing trap), reusing one ordering across several order statistics, and vectorized grouped regression. Measured timings throughout.

A companion skill, fastverse-r, covers the rest of the stack that library(fastverse) attaches — data.table, kit and magrittr — and the division of labour between them and collapse. Consult it when the task needs something collapse does not have (rolling windows, rolling/non-equi joins, conditional := updates, row-wise statistics across columns, topn, vectorized switches, fread/fwrite), or when deciding whether a given step belongs in collapse or in a data.table [i, j, by].

Writing efficient code

collapse usually offers several routes to the same answer, and they differ mainly in efficiency. The habit worth building is to think about what the machine has to do and craft the minimal solution — not to reach for the first thing that works. Full treatment with measured timings in references/efficiency.md.

Sequence operations so less data moves. Subset and select in the same call, before grouping: fsubset(d, cond, only, the, needed, cols) |> fgroup_by(g) |> fmean() (~1.65x faster than filtering wide and selecting later). collapse enforces part of this — fsubset() refuses grouped data and roworder() messages about it.

Use the minimal grouping the job needs. Don't reach for fgroup_by() unless several different functions share one grouping; there is no .by = argument because internal grouping is the .by:

fmutate(d, med = fmedian(v, list(cyl, vs, am), TRA = "fill"))   # best
d |> fgroup_by(g, return.groups = FALSE) |> fmutate(...)        # if reusing the grouping

Turn on the three options once, at the top. Unsorted grouping and no missing-value check are faster; nthreads parallelizes ungrouped reductions (~4–7x) and work across data-frame columns (~1.6x), though little for a single grouped vector:

set_collapse(sort = FALSE, nthreads = 4, na.rm = FALSE)

sort = FALSE changes result order to first-appearance; na.rm = FALSE is only safe when the data has no NAs. Statistical functions have no sort argument — pass qF(g, sort = TRUE) where sorted output is required regardless.

Materialize only what you need. use.g.names = FALSE, return.groups = FALSE, return.order = FALSE, na.exclude = FALSE — a lean GRP object was ~1.9x faster than a full one here. Same instinct for intermediates: settransform, setv, %+=%, set = TRUE where the copy is pure waste.

Prefer indices to logical vectors. whichv(x, 0) / x %==% 0 returns positions; setv(x, 0, NA) replaces without allocating a mask at all.

Let the C code do the work. If you are writing lapply over groups, sapply(split(...)), or a join purely to attach a group aggregate, there is almost certainly a g/TRA/w/o argument that removes it. Two idioms worth knowing by heart, both in references/efficiency.md: reusing one radixorder() across several order statistics (~4.5x), and fsum(y, g, w = x) as an inner product for grouped regression slopes.

For anything not covered here, the package's own documentation is unusually complete: help("collapse-documentation") is a hierarchical index of every function, and ?fsummarise, ?TRA, ?join, ?pivot, ?GRP all have extensive worked examples. The JSS article (doi:10.18637/jss.v116.i01) is the concise reference.