data-analysis
End-to-end empirical data analysis workflow for R or Python projects.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
القائمة
End-to-end empirical data analysis workflow for R or Python projects.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
Find and assess datasets for a research question.
Repository-wide consistency audit for skills, hooks, rules, and docs.
Systematic literature review workflow using parallel librarian agents.
Structured interview and project-spec workflow for new research ideas.
Proofreading workflow for academic manuscripts and papers.
Verify that paper claims match analysis outputs before submission.
| name | data-analysis |
| description | End-to-end empirical data analysis workflow for R or Python projects. |
Run an end-to-end data analysis in R or Python: load, explore, analyze, and produce publication-ready output.
Input: $ARGUMENTS — a dataset path (e.g., data/county_panel.csv) or a description of the analysis goal (e.g., "regress wages on education with state fixed effects using CPS data").
Determine language from $ARGUMENTS or ask the user:
tidyverse, fixest, lm, .R context → R trackpandas, statsmodels, sklearn, .py or .ipynb context → Python track.csv/.parquet with no language cue → use request_user_input in Plan mode for a single-choice prompt if available; otherwise ask the user which language to use:
If $ARGUMENTS points to a PDF, scraped HTML page, government portal URL, or other non-tabular source, extract it into a tidy dataset before Phase 1. Skip this phase when the input is already a flat file (.csv, .parquet, .dta, .rds, .feather, .json).
.pdf, .html, .htm, .xml| Source | Package | Pattern |
|---|---|---|
| PDF tables | tabulizer, pdftools | tabulizer::extract_tables("file.pdf", pages = 3) |
| Scanned PDFs (OCR) | tesseract | tesseract::ocr("file.png") |
| HTML tables | rvest | read_html(url) %>% html_table() |
| Generic scrape | rvest, httr2 | read_html(url) %>% html_elements(".css-selector") |
| FRED | fredr | fredr(series_id = "UNRATE") |
| BLS | blsAPI | blsAPI(payload) |
| Census / ACS | tidycensus | get_acs(geography, variables, year) |
| World Bank | WDI | WDI(indicator = "NY.GDP.MKTP.CD", country = "all") |
| Source | Package | Pattern |
|---|---|---|
| PDF text | pdfplumber | pdfplumber.open(path).pages[0].extract_text() |
| PDF tables | camelot | camelot.read_pdf(path, pages='3') |
| Scanned PDFs | pytesseract, Pillow | pytesseract.image_to_string(Image.open(...)) |
| HTML tables | pandas | pd.read_html(url)[0] |
| Generic scrape | bs4, httpx | BeautifulSoup(httpx.get(url).text, "html.parser") |
| FRED | fredapi | Fred(api_key).get_series("UNRATE") |
| Census / ACS | census | Census(api_key).acs5.state(...) |
Before saving, create each parent directory if it is missing: scripts/ingest, data/processed, and quality_reports.
scripts/ingest/[name]_ingest.R (or .py) — distinct from the analysis script so the slow extraction step does not rerun every analysis.data/processed/[name].csv or .parquet.quality_reports/ingest_[name].md: source URL/file, date pulled, page range or CSS selectors, row count, dropped rows and reason.data/processed/ — not the raw source.If the source is ambiguous (which table, which page, which date range), ask once before extracting. Use request_user_input in Plan mode only when there are 2-3 bounded single-choice options; otherwise ask conversationally. Do not guess on legal or government documents — the wrong table is worse than no table.
Before saving any script, data extract, diagnostic, analysis object, or review report, create the parent directory if it is missing. At minimum, the workflow may need:
scripts/ingestscripts/Rscripts/pythondata/processedoutput/diagnosticsoutput/analysisquality_reportsrules/r-code-conventions.md for all standardsscripts/R, output/analysis, output/diagnostics, and quality_reports before saving into themscripts/R/ with descriptive namesoutput/saveRDS() for every computed objectr-reviewer protocol on the generated script before presenting resultslibrary(), never require())set.seed(42)dir.create(..., recursive = TRUE, showWarnings = FALSE) for scripts/R, output/analysis, output/diagnostics, and quality_reportssummary(), missingness rates, variable typesoutput/diagnostics/fixest; cross-section: use lm/glmRun the design-specific diagnostics referees expect to see for the design being claimed. This phase is mandatory for any analysis with a causal interpretation. Detect the design from the analysis goal and the model call (feols(... | id + t) with treatment-timing -> DiD; rdrobust -> RDD; feols(y ~ ... | 0 | endog ~ instr) -> IV). Ask once only if the design is genuinely ambiguous.
Save every diagnostic figure to output/diagnostics/ and every numerical result to RDS for quality-gate and write-paper to consume. Create output/diagnostics before saving.
DiD / event study:
fixest::iplot() (or ggiplot::ggiplot())fixest::feols(y ~ sunab(g, t) | id + t), did::att_gt() (Callaway-Sant'Anna), or DIDmultiplegt::did_multiplegt_dyn() (de Chaisemartin); compare to TWFE headlinebacondecomp::bacon() when TWFE is the headline estimateRDD:
rddensity::rddensity(X, c = cutoff) for the McCrary-type density discontinuity test — report t-stat and p-valuerdrobust::rdbwselect(y, x, c) for optimal bandwidth; re-estimate at 0.5x, 1x, 2x the optimalrdrobust::rdrobust(z, x, c) for each covariate zIV:
fixest::feols(y ~ controls | fe | endog ~ instr) — report the effective F via fitstat(., "ivf1"); Olea-Pflueger via ivDiag::ivDiag()ivDiag::AR_test() or ivmodel::AR.test()fixest::wald() when instruments > 1Synthetic control:
tidysynth::generate_placebos() for in-space placebo permutation; plot treatment-vs-donor effect pathsRCT / OLS with exogenous treatment:
modelsummary::datasummary_balance(~ treat, data = df)ri2::conduct_ri() for the primary outcomeMatching / propensity score:
cobalt::bal.tab()Detected design -> run the corresponding block automatically. Multiple designs (e.g., DiD with IV robustness) -> run all relevant blocks. The user does not need to ask for these; they are the default for any causal claim.
Tables: modelsummary (preferred) or stargazer — export .tex and .html
Figures: ggplot2 with project theme; explicit ggsave(width = X, height = Y); save as .pdf and .png; add bg = "transparent" only if output is for Beamer slides
saveRDS() for all key objectsagents/r-reviewer.md and apply that review protocol to the generated script. If the user explicitly requested delegated review, you may run the r-reviewer as a separate subagent.# ============================================================
# [Descriptive Title]
# Author: [from project context]
# Purpose: [What this script does]
# Inputs: [Data files]
# Outputs: [Figures, tables, RDS files]
# ============================================================
# 0. Setup ----
library(tidyverse)
library(fixest)
library(modelsummary)
set.seed(42)
dir.create("scripts/R", recursive = TRUE, showWarnings = FALSE)
dir.create("output/analysis", recursive = TRUE, showWarnings = FALSE)
dir.create("output/diagnostics", recursive = TRUE, showWarnings = FALSE)
dir.create("quality_reports", recursive = TRUE, showWarnings = FALSE)
# 1. Data Loading ----
# 2. Exploratory Analysis ----
# 3. Main Analysis ----
# 4. Tables and Figures ----
# 5. Export ----
scripts/python, output/analysis, output/diagnostics, and quality_reports before saving into themscripts/python/ with descriptive namesoutput/joblib.dump() for model objects; .to_parquet() for DataFramespathlib.Path for all file paths — never hardcode absolute pathsnp.random.seed(42) and random.seed(42)Path("output/analysis").mkdir(parents=True, exist_ok=True)pandasdf.describe(), df.isnull().sum(), df.dtypesmatplotlib/seabornoutput/diagnostics/df.describe().to_csv("output/diagnostics/summary_stats.csv")smf.ols("y ~ x", data=df).fit(cov_type="HC3")PanelOLS from linearmodels with cluster-robust SEsMirror the R-track diagnostics with Python tooling. This phase is mandatory for any analysis with a causal interpretation. Detect the design from the analysis goal and the model call (PanelOLS(..., entity_effects=True, time_effects=True) with treatment timing -> DiD; rdrobust Python port -> RDD; IV2SLS -> IV). Ask once only if the design is genuinely ambiguous.
Save diagnostic figures to output/diagnostics/ and numerical results to .pkl or .parquet. Create output/diagnostics before saving.
DiD / event study:
linearmodels.PanelOLS interaction terms — plot leads and lagsdifferences::ATTgt (Callaway-Sant'Anna port) or pyfixest Sun-Abraham; compare to the TWFE headlineRDD:
rdrobust Python port: rdrobust.rddensity(X, c=cutoff) — report McCrary-type t and prdrobust.rdbwselect(...) for bandwidth; re-estimate at 0.5x, 1x, 2xIV:
linearmodels.IV2SLS(...).fit() — first-stage F via results.first_stage / results.diagnosticslinearmodels Wald tools)Synthetic control:
pysyncon for in-space placebo permutationRCT / OLS with exogenous treatment:
scipy.stats.ttest_ind or stargazer)numpy.random.permutation loopMatching:
causalinference or manual computationDetection and "no need to ask the user" rules match the R track.
Tables: Format with pandas and export via .to_latex() or stargazer (Python port)
Figures: matplotlib/seaborn; explicit fig.savefig(path, dpi=300, bbox_inches="tight"); save as .pdf and .png
joblib.dump(model, "output/model.pkl") for fitted modelsdf_results.to_parquet("output/results.parquet") for DataFrames# ============================================================
# [Descriptive Title]
# Author: [from project context]
# Purpose: [What this script does]
# Inputs: [Data files]
# Outputs: [Figures, tables, pickle/parquet files]
# ============================================================
import random
import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
import matplotlib.pyplot as plt
import seaborn as sns
import joblib
from pathlib import Path
# Seeds
np.random.seed(42)
random.seed(42)
# Output directories
Path("output/analysis").mkdir(parents=True, exist_ok=True)
Path("output/diagnostics").mkdir(parents=True, exist_ok=True)
Path("output/figures").mkdir(parents=True, exist_ok=True)
Path("scripts/python").mkdir(parents=True, exist_ok=True)
Path("quality_reports").mkdir(parents=True, exist_ok=True)
# 1. Data Loading
# 2. Exploratory Analysis
# 3. Main Analysis
# 4. Tables and Figures
# 5. Export
[ ] All imports at top
[ ] Random seeds set (numpy + stdlib)
[ ] All paths use pathlib.Path — no hardcoded strings
[ ] Output directories created with mkdir(exist_ok=True)
[ ] Figures saved with explicit dpi=300, bbox_inches="tight"
[ ] Model objects saved with joblib.dump()
[ ] DataFrames saved as parquet
[ ] Comments explain WHY, not WHAT