r-eda

name	r-eda
description	Run end-to-end exploratory and analytical R workflows from raw data to analysis-ready datasets, reproducible pipelines, reports, and model-ready outputs. Use when the user wants the full analysis pipeline scaffolded or executed.

R EDA

Run exploratory work as an auditable pipeline from intake to report-ready artifacts.

Quick start

1. If the workspace does not exist, start with /init-r-workspace.
2. Read references/pipeline-checklist.md.
3. If the source work includes metadata registries, designated IDs, hierarchy slots, dataset grain, relationship inference, or joined-dataset creation, also read references/metadata-registry-hierarchy-workflows.md before building transformations.
4. If the source is geospatial, shapefile/GeoJSON/GPKG-backed, coordinate-column-backed, or WKT-backed, also read references/geospatial-sf-workflows.md before reshaping or joining.
5. If the source is API-backed, JSON-backed, paginated, rate-limited, or NDJSON/streaming-backed, also read references/api-ingest-with-httr2-jsonlite.md before writing ingestion helpers.
6. If the workflow will publish or consume versioned datasets or derived artifacts through pins, also read references/pins-artifact-boards.md before wiring the handoff surface.
7. If the source is a large local in-memory table and the workflow will rely on data.table, tidytable, or dtplyr, also read references/high-performance-tabular-workflows.md before shaping the dataset.
8. If the source is large, multi-file, or repeatedly reused, also read references/arrow-parquet-large-data.md before intake.
9. If the workflow should use DuckDB as a local analytical warehouse over files, Arrow objects, or staged intermediate tables, also read references/duckdb-local-analytics.md before writing transformations.
10. If the source data already lives in a database or warehouse, also read references/database-backed-analysis.md before writing transformations.
11. If the workflow must rerun reproducibly across analysts, machines, or scheduled executions, also read references/renv-analysis-environments.md before the project grows large.
12. If the workflow must rerun reproducibly across datasets, models, and rendered reports, also read references/targets-quarto-pipelines.md before scripting the orchestration layer.
13. If the workflow must fan out across many independent files, groups, parameter sets, or simulations, also read references/parallel-analysis-with-future-furrr.md before parallelizing the loop.
14. If the workflow is slow enough that implementation choices need evidence, also read references/performance-profiling-and-benchmarking.md before rewriting helpers or micro-optimizing.
15. Map the source data, primary exposure, primary outcome, time/status fields, covariates, and API or geometry identifiers before building transformations.
16. Build one explicit analysis dataset pipeline.
17. For larger-than-memory or repeated-use sources, convert to a partitioned Parquet working set early and keep work lazy in Arrow until the analyst-facing slice is small.
18. For DuckDB-backed sources, keep the query lazy in dbplyr or the DuckDB relational layer, materialize repeated intermediates deliberately, and collect only the small analyst-facing slice.
19. For database-backed sources, keep the query lazy in dbplyr, materialize remote intermediates with compute() only when repeated work warrants it, and collect only the small analyst-facing slice.
20. Add sanity checks, summary tables, visuals, and a model-ready dataset in that order.
21. Hand off to /r-model-builder or /r-tables-reporting when the modeling or reporting layer needs deeper specialization.

Workflow

1. Intake and variable inventory

• Read the source file(s) or connect to the source database deliberately.
• Inspect names, example values, and missingness.
• Write down the role of each analysis variable before recoding.
• If the source is too large for comfortable in-memory work, open it as an Arrow Dataset or a lazy database table and keep the working set remote until it is narrow.

2. Build the analysis dataset

• Keep transformations in one explicit pipeline when feasible.
• Preserve labels or a lookup object for reporting.
• Keep only the variables needed for the current analysis layer.
• For large raw text data, create a Parquet working layer before repeated analyst loops.
• For database-backed data, start from dbplyr::tbl() and let filtering, selection, grouping, and summarising stay in SQL as long as possible.
• Delay collect() until filtering and column reduction have already made the working set small.

2b. Keep geospatial geometry contracts explicit

• Current st_read() documentation says sf can read simple-feature layers from files or databases. Keep the source path, layer name, and any database query explicit when ingesting shapefiles, GeoJSON, GPKG, or database-backed geometry tables.
• Current st_as_sf() documentation says point geometries can be constructed from coordinate columns or WKT columns with an explicit crs. Convert tabular coordinates or WKT into sf at the ingestion boundary instead of leaving geometry creation implicit downstream.
• Current st_transform() documentation says CRS conversion is explicit and st_can_transform() can test whether a transformation path exists. Keep CRS alignment visible before spatial joins, overlays, or web-map display.
• Current validity documentation says st_is_valid() checks geometry validity and st_make_valid() repairs invalid geometry. Validate polygon-heavy layers before joins, summaries, or app-facing mapping surfaces.
• Current st_join() documentation keeps join semantics explicit through the spatial predicate, left, and largest. Choose the predicate and join posture deliberately so downstream summaries can explain why features did or did not match.
• Current st_simplify() documentation keeps dTolerance and preserveTopology explicit. Team inference: preserve one analysis-grade geometry layer and, when needed, derive a simpler display layer for app rendering rather than mutating the only geometry surface in place.

2c. Keep API request, response, and pagination contracts explicit

• Current httr2 documentation says requests are built by starting with request() and then modifying URL, headers, body, retry, and throttling behavior before req_perform(). Keep the request builder explicit so ingestion logic can explain exactly which endpoint, query parameters, and authentication headers produced the data.
• Current req_url_query() and req_url_path_append() documentation keeps path and query mutation explicit. Build endpoint paths and query parameters deliberately instead of pasting full URLs together ad hoc.
• Current req_body_json() documentation says JSON request bodies are encoded explicitly and can be modified by named components. Keep POST/PATCH payload construction visible rather than mixing request-body creation into a generic helper that also performs the call.
• Current req_retry() documentation says retries apply to transient failures such as 429 and 503, with truncated exponential backoff and jitter by default. Keep retry posture explicit whenever the source is rate-limited or intermittently unavailable.
• Current req_throttle() documentation says throttling uses a token-bucket model with explicit capacity and fill_time_s. Treat source-side rate limits as part of the ingest contract instead of as ad hoc Sys.sleep() calls.
• Current resp_body_json() documentation says parsed JSON can remain unsimplified or be simplified into vectors and frames. Keep the simplification boundary explicit so nested API payloads do not silently flatten into lossy tabular forms.
• Current jsonlite documentation says fromJSON() / toJSON() handle ordinary JSON while stream_in() / stream_out() support line-by-line stream processing, and flatten() is available for nested data frames. Team doctrine: keep one raw-response or nested-list contract long enough to inspect the schema before flattening for analysis.

2d. Keep local high-performance tabular stages explicit

• Current fread() documentation says ingest controls such as select, drop, nThread, and integer64 stay explicit while parsing large delimited files efficiently. Keep that intake posture visible instead of reading every column and recoding later.
• Current setDT() documentation says conversion to data.table happens by reference. Convert once at the intake boundary when the same local table will feed many grouped summaries or joins.
• Current reference-semantics and := documentation says column updates happen by reference and copy() prevents side effects when a second branch needs an independent object. Keep the copy boundary explicit when one source object feeds multiple analysis variants.
• Current setkey() documentation says keys reorder rows by reference and help binary-search subsets and joins. Keep ordering and join-key assumptions visible whenever downstream plots, tables, or exports care about row order.
• Current tidytable documentation keeps grouped mutate() and summarize() work available through .by, while current dtplyr documentation says lazy_dt() captures dplyr intent and only materializes when collected or converted. Keep the engine choice explicit so a reviewer can tell whether the pipeline is by-reference data.table, tidy-style tidytable, or lazy-translated dtplyr.
• Current dtplyr translation guidance keeps show_query() available. Inspect the generated data.table code when grouped summaries, ordering, or joins become hard to reason about.
• Team doctrine: choose one high-performance tabular engine for each pipeline stage and name the conversion boundary when the workflow moves back to tibble, Arrow, DuckDB, or a model-ready frame.

2e. Use DuckDB deliberately as a local analytical warehouse

• Reach for DuckDB when the job needs a local analytical SQL engine over Parquet, CSV, Arrow objects, or staged intermediate tables without standing up a separate database server.
• Current DuckDB documentation frames the R package as a DBI connector for an embedded analytical database and exposes file-facing helpers like tbl_file() plus registration helpers for data frames and Arrow data sources.
• Keep the same remote-data discipline you would use with another dbplyr backend: narrow first, inspect the generated query, materialize repeated intermediates intentionally, and collect only the analysis-ready slice.
• Current Arrow documentation says to_duckdb() creates a DuckDB table backed by the Arrow object with no copied data until collect(), compute(), or another query runs. That makes DuckDB a good next layer when Arrow filtering is no longer enough and you need joins, window functions, or richer SQL aggregation.
• Current Arrow documentation also says to_arrow() can move a DuckDB-backed pipeline back toward Arrow, but collect() or compute() on that result can only be called once unless you materialize it first with as_arrow_table().
• Current duckplyr documentation positions it as a drop-in replacement for dplyr on DuckDB, with fallback to R when translation is unavailable. Treat that fallback boundary as a real materialization boundary. A practical team doctrine is to prefer dbplyr when you need explicit SQL inspection and predictable remote semantics, and reach for duckplyr when closer dplyr compatibility is the bigger gain.
• Current DuckDB concurrency documentation says one process can both read and write, while multiple processes can read in READ_ONLY mode but multi-process writes are not supported automatically. Treat a file-backed DuckDB database as an analytical working store first, not as a shared transactional write surface.

2f. Keep pin boards and artifact versions explicit

• Current pins documentation says boards come from explicit constructors such as board_temp(), board_local(), board_folder(), board_connect(), and other board helpers. Keep the board choice explicit because it changes who can read the artifact and how durable the handoff is.
• Current pin_write() / pin_read() documentation keeps type, title, description, metadata, versioned, tags, urls, version, and hash explicit. Treat those artifact details as part of the analysis contract whenever downstream workflows need reproducible retrieval.
• Current pin_versions() documentation keeps version listing, deletion, and pruning explicit. Keep version lifecycle deliberate when a board stores evolving curated datasets or derived artifacts.
• Team doctrine: treat a pin board as an artifact exchange surface and keep board name, artifact name, version, and metadata visible enough that downstream work can explain exactly which artifact it consumed.

3. Add sanity checks

• Confirm row counts, key subgroup counts, and missingness impact.
• Check exposure/outcome cross-tabs or distributions before modeling.
• Keep these checks in the script so reruns stay auditable.
• On large datasets, start with Arrow-side counts and grouped summaries before materializing data frames.
• On database-backed datasets, inspect the generated query and query plan before blaming R-side code for slow grouped summaries or joins.

4. Produce exploratory outputs

• Build at least one summary table by the key grouping variable.
• Add visuals that reveal distribution shape, sparsity, and exposure/outcome relationships.
• Keep plot objects named and reusable.

5. Build the model-ready layer and report artifacts

• Construct the modeling dataset explicitly.
• Keep metrics and predictions aligned with the chosen outcome family.
• Save report outputs and interpretation notes in predictable locations.
• If a database-backed analysis must write curated outputs back to the database, keep the write step explicit and transactional.

5b. Keep the analysis environment reproducible with renv

• In shared or long-lived analytical workspaces, initialize renv early enough that the environment reflects the real project rather than a developer’s ad hoc global library.
• Current renv::snapshot() documentation says implicit mode follows packages discovered by dependencies(), explicit follows DESCRIPTION, and all records every package in the project library. For analysis workspaces built around scripts, Quarto files, and Shiny code, implicit is usually the better default; this is a team doctrine / inference grounded in the official snapshot-type definitions.
• Snapshot after deliberate dependency changes, and restore before rerunning the workflow on a fresh machine, CI runner, or deployment target.
• Run renv::status() when collaborators, CI, or scheduled jobs drift from local behavior. Current renv docs recommend resolving reported issues before further work.
• Keep renv.lock committed alongside _targets.R, Quarto documents, and data-ingest scripts so the pipeline and its package environment move together.
• If the repository serves multiple operational modes, such as exploratory analysis plus a Shiny app runtime, renv profiles can separate those environments. Keep the mode split deliberate.
• Current renv Docker guidance notes that OS libraries, compilers, and other system details still matter. For highly portable analytical runs, pair the project package environment with a system-level runtime contract.

6. Orchestrate repeated work with targets and Quarto

• Keep orchestration in _targets.R, and keep data, model, and reporting helpers in sourced files so the pipeline remains readable. Current targets documentation highlights tar_source() plus tar_option_set() as the standard pattern.
• Use tar_option_set() to declare shared packages and defaults such as storage format, execution controller, and target-level behavior.
• Keep file boundaries explicit. Use file targets for raw inputs and rendered artifacts, and keep downstream targets honest about which files they actually consume.
• For repeated or large intermediate data, choose storage deliberately. Current targets guidance notes that non-default formats such as qs can be more efficient than rds, while format = "file" is a strong fit when data should stay on disk and downstream steps should only load narrow slices.
• Use tar_render() for R Markdown, tar_quarto() for a Quarto document or project, and tar_quarto_rep() / tar_render_rep() when one report must render over a parameter grid.
• For a single parameterized render, pass one list of parameters. Current tar_render() and tar_quarto() docs note that these parameter arguments are evaluated when the target runs, which means they can depend on upstream targets.
• For repeated parameterized renders, keep one row per render in the parameter data frame and make output-file naming explicit. Current tar_quarto_rep() docs assume you do not set output-dir in _quarto.yml.
• Keep literate-programming limitations visible. Current tarchetypes docs note that child documents are not tracked for changes, dependency detection can miss tar_read() or tar_load() calls inside user-defined functions, and the literate-programming targets work with local files.
• For heavy pipelines, current targets guidance points to crew controllers, storage = "worker", retrieval = "worker", and deployment = "main" as levers for balancing throughput, bandwidth, and quick local tasks.
• Use tar_manifest(), tar_visnetwork(), tar_progress(), and tar_watch() to inspect, explain, and monitor the pipeline instead of treating the DAG as hidden machinery.

6b. Parallelize only self-sufficient work with future and furrr

• Current future documentation says evaluation happens in a local environment and, for asynchronous backends, globals are exported or frozen for the worker. Treat each parallel task as a self-sufficient job with a deliberate input payload instead of letting workers inherit a huge implicit analysis state.
• Current future documentation also says the total size of exported globals is checked against a configurable threshold. If parallel work fails only at launch time, inspect the globals footprint before rewriting the analysis logic.
• Current multisession documentation says work is resolved in separate R sessions and that later changes to a global variable do not affect an already launched future. For cross-platform analyst workflows, multisession is usually the steadier default when interactive safety matters.
• Current multicore documentation says forked processing is unavailable on Windows and can be unstable in environments such as RStudio. Keep that backend out of default interactive analyst workflows unless the runtime is known to support it safely.
• Current future guidance says properly written code should not assume where or when futures are resolved. Team doctrine: choose the plan in the top-level script, pipeline controller, or app entry point so the parallel posture stays inspectable.
• Current furrr documentation says future_map() and friends parallelize map-style iteration and that furrr_options() controls globals, packages, random-number seeding, and chunking via scheduling or chunk_size. Keep those controls explicit when throughput, reproducibility, or worker overhead matters.
• Current furrr chunking guidance says the default strategy places one chunk on each worker. Increase chunk count when task durations vary substantially, and prefer larger chunks when per-task overhead dominates the work.
• Current furrr gotchas note that grouped data frames can block the sharding behavior that makes furrr worthwhile. When iterating over grouped or nested data, ungroup or split into an explicit list-column/input list before parallel mapping.
• Current furrr gotchas also warn that forked multicore plans and graphics devices can interact badly. When parallel work renders plots or touches graphics devices, separate-session backends are the steadier posture.

6c. Profile real bottlenecks before optimizing

• Current profvis documentation says it visualizes sampled profiling data from Rprof(), with a default sampling interval of 10 ms. When a workflow feels slow, begin with a representative end-to-end or section-level profile so the hot path is evidence-based.
• Current profvis guidance also says profiles are not deterministic because they are sampled. Repeat important profiles and look for recurring hot helpers instead of overfitting to one run.
• Current bench::mark() documentation says benchmarked expressions run at least twice and can record both timing and memory allocation. Once the slow region is known, benchmark the competing narrow implementations with explicit check, min_time, or iterations settings when the comparison will inform a refactor.
• Team doctrine: profile the real pipeline first, then benchmark the narrow alternative. Keep benchmark inputs representative enough that the result still resembles the analyst workload that triggered the investigation.

Resources

• references/pipeline-checklist.md
• references/metadata-registry-hierarchy-workflows.md
• references/geospatial-sf-workflows.md
• references/api-ingest-with-httr2-jsonlite.md
• references/pins-artifact-boards.md
• references/high-performance-tabular-workflows.md
• references/arrow-parquet-large-data.md
• references/duckdb-local-analytics.md
• references/database-backed-analysis.md
• references/renv-analysis-environments.md
• references/targets-quarto-pipelines.md
• references/parallel-analysis-with-future-furrr.md
• references/performance-profiling-and-benchmarking.md