| name | scholar-data |
| description | Comprehensive open data directory (100+ datasets across 14 categories) with auto-fetch capability, plus data collection instrument design, variable dictionaries, data management, IRB materials, and web/digital data collection for social science studies. Covers GSS, PSID, ACS, CPS, ESS, WVS, Afrobarometer, Eurobarometer, DHS, PISA, OECD, Eurostat, WHO, OpenAlex, Harvard Dataverse, ICPSR, Zenodo, OSF, and many more. Use when the user needs to find or download data for a research question, design a survey or interview protocol, scrape websites or collect social media data, plan data management, navigate IRB, or produce a data blueprint. Works best after /scholar-design and before /scholar-eda. |
| tools | Read, WebSearch, Write, Bash |
| argument-hint | [dataset|survey|interview|irb|manage|vignette|scrape|web|api|social media] [topic or research question] [optional: population, journal, design] |
| user-invocable | true |
Scholar Data Collection & Management
You are an expert social science methodologist helping design rigorous data collection instruments, identify optimal data sources, and build reproducible data management systems. All outputs target top-tier journals (ASR, AJS, Demography, Science Advances, NHB, NCS).
Arguments
The user has provided: $ARGUMENTS
Parse to determine:
- Primary task (select workflow below)
- Research topic and target population
- Stage: finding data vs. designing new collection vs. managing existing data
Dispatch Table
Route to the appropriate workflow based on arguments:
| Keyword(s) in arguments | Workflow to run |
|---|
dataset, data source, secondary, existing data, find data, what data | WORKFLOW 0 — Secondary Data Directory |
variable, measure, operationalize, construct, blueprint | WORKFLOW 1 — Variable Dictionary |
survey, questionnaire, scale, Qualtrics, Prolific, MTurk | WORKFLOW 2 — Survey Instrument Design |
vignette, conjoint, factorial, experiment | WORKFLOW 2 (Steps 1–5 + Step 6 Experimental Modules) |
list experiment, sensitive, endorsement experiment | WORKFLOW 2 Step 6b |
interview, qualitative, protocol, ethnography, focus group | WORKFLOW 3 — Qualitative Protocol |
admin, administrative, records, linkage, Census, IPUMS | WORKFLOW 4 — Administrative and Secondary Data |
IRB, ethics, consent, CITI, human subjects | WORKFLOW 5 — IRB and Research Ethics |
manage, codebook, clean, pipeline, DMP, data sharing, git | WORKFLOW 6 — Data Management Pipeline |
scrape, web scraping, crawl, HTML, API, social media, Twitter, Reddit, news, digital data, online data | WORKFLOW 7 — Web Scraping and Digital Data Collection |
Run all relevant workflows if multiple apply. Always end with Save Output.
Create output directories:
OUTPUT_ROOT="${OUTPUT_ROOT:-output}"
mkdir -p "${OUTPUT_ROOT}/logs"
Process Logging (REQUIRED) — Reasoning · Action · Observation trace:
This skill emits an append-only RAO trace at ${OUTPUT_ROOT}/logs/trace-scholar-data-<date>.ndjson — the source of truth. The human-readable process-log-scholar-data-<date>.md is rendered from it. Full protocol + privacy rule: _shared/process-logger.md.
At each meaningful step (a decision, a script/tool run, a gate call, a subagent dispatch), append one record. emit-trace.sh derives seq from the file, so no state is tracked across the stateless Bash blocks:
bash "${SCHOLAR_SKILL_DIR:-.}/scripts/gates/emit-trace.sh" --skill scholar-data --step "<label>" \
--reasoning "<the WHY — stated rationale, 1–2 lines>" \
--action "<the WHAT — tool/script/gate call + key args>" \
--observation "<the RESULT — verdict/metric/count/error/file ref>" --status ok
At the end (Save Output), render the human-readable log and self-check:
OUTPUT_ROOT="${OUTPUT_ROOT:-output}"
bash "${SCHOLAR_SKILL_DIR:-.}/scripts/gates/render-trace.sh" "${OUTPUT_ROOT}/logs/trace-scholar-data-$(date +%Y-%m-%d).ndjson"
bash "${SCHOLAR_SKILL_DIR:-.}/scripts/gates/trace-coverage-check.sh" "${OUTPUT_ROOT}" --skill scholar-data
Privacy (C-01 / LOCAL_MODE): the trace carries aggregate metrics, verdicts, counts, and file refs ONLY — never raw data rows, verbatim quotes, or PII.
Step 0 — Data Safety Sidecar Check (Tier B)
If $ARGUMENTS includes paths to existing user data files (e.g., the user wants a variable dictionary for an already-downloaded CSV, or is extracting a codebook from a local .dta), consult .claude/safety-status.json BEFORE any Read call. scholar-data is a Tier B skill — it does not implement the full LOCAL_MODE dispatch contract, so it refuses files whose sidecar status is NEEDS_REVIEW:*, HALTED, or LOCAL_MODE. See _shared/tier-b-safety-gate.md for the full policy.
This step is a no-op when $ARGUMENTS contains no file paths (e.g., the user is searching for open datasets with a topic query), or when the project was not initialized via /scholar-init (no sidecar exists). The PreToolUse hook (scripts/gates/pretooluse-data-guard.sh) remains the mechanical backstop either way.
SIDECAR=".claude/safety-status.json"
if [ -f "$SIDECAR" ] && command -v jq >/dev/null 2>&1; then
UNSAFE=""
for F in $FILE_ARGS; do
[ -f "$F" ] || continue
ABS=$(python3 -c 'import os,sys; print(os.path.realpath(sys.argv[1]))' "$F" 2>/dev/null \
|| realpath "$F" 2>/dev/null || readlink -f "$F" 2>/dev/null || echo "$F")
STATUS=$(jq -r --arg k "$ABS" '.[$k] // empty' "$SIDECAR")
[ -z "$STATUS" ] && STATUS=$(jq -r --arg k "$F" '.[$k] // empty' "$SIDECAR")
case "$STATUS" in
CLEARED|ANONYMIZED|OVERRIDE|"") ;;
NEEDS_REVIEW:*) UNSAFE="${UNSAFE}
- $F → $STATUS (run: /scholar-init review)" ;;
HALTED) UNSAFE="${UNSAFE}
- $F → HALTED (off-limits)" ;;
LOCAL_MODE) UNSAFE="${UNSAFE}
- $F → LOCAL_MODE (use /scholar-analyze or /scholar-eda — scholar-data is Tier B and does not implement LOCAL_MODE)" ;;
*) UNSAFE="${UNSAFE}
- $F → $STATUS (unrecognized; resolve via /scholar-init review)" ;;
esac
done
if [ -n "$UNSAFE" ]; then
cat >&2 <<HALTMSG
⛔ HALT — scholar-data Step 0 refused the following file(s):
$UNSAFE
HALTMSG
exit 1
fi
fi
WORKFLOW 0: Secondary Data Source Directory
Use when the user needs to identify the best existing dataset for a research question. Always run this workflow first if no data source has been specified.
Step 0a. Match Topic to Dataset
Sociology / stratification / inequality / labor markets:
| Dataset | Coverage | Key variables | Access | Timeline |
|---|
| GSS (General Social Survey) | US, 1972–present, cross-sectional + panel (2016+) | Attitudes, SES, religion, race, family, work | IPUMS / NORC; free | Immediate |
| PSID (Panel Study of Income Dynamics) | US, 1968–present, household panel | Income, wealth, employment, health, housing | psid.isr.umich.edu; free registration | 1–2 weeks |
| NLSY79 / NLSY97 | US birth cohorts, longitudinal | Labor, education, health, cognition, family | BLS; free registration | Immediate |
| Add Health (NSFAH) | US adolescents, 4 waves 1994–2018 | Health, networks, neighborhoods, SES | ICPSR; free registration | 1–2 weeks |
| SIPP (Survey of Income and Program Participation) | US, household panel | Income, poverty, program participation | Census; free | Immediate |
| ACS (American Community Survey) | US, annual, 1% / 5% samples | Demographics, income, housing, language | IPUMS USA; free | Immediate |
| CPS (Current Population Survey) | US, monthly, annual ASEC supplement | Employment, income, demographics | IPUMS CPS; free | Immediate |
Demography / family / health:
| Dataset | Coverage | Key variables | Access | Timeline |
|---|
| HRS (Health and Retirement Study) | US 50+, biennial panel 1992– | Health, cognition, wealth, retirement, death | ICPSR; free registration | 1–2 weeks |
| NHANES | US, cross-sectional cycles | Physical health, biomarkers, diet, SES | CDC; free | Immediate |
| NHIS (National Health Interview Survey) | US, annual | Health status, access to care | CDC; free | Immediate |
| Vital Statistics (NCHS) | US births and deaths | Birth certificate, death certificate vars | CDC; free | Immediate |
| World Population Prospects | Global, UN projections | Fertility, mortality, migration by country | UN; free | Immediate |
Education:
| Dataset | Coverage | Key variables | Access | Timeline |
|---|
| NELS / ELS / HSLS (NCES) | US high school cohorts | Academic achievement, SES, post-secondary | NCES; free | Immediate |
| NAEP | US, grade 4/8/12, national + state | Math and reading scores | NCES; free | Immediate |
| College Scorecard | US institutions | Earnings, completion, debt, costs | ED.gov; free | Immediate |
| IPEDS | US institutions | Enrollment, graduation, faculty, finances | NCES; free | Immediate |
Political behavior / civic:
| Dataset | Coverage | Key variables | Access | Timeline |
|---|
| ANES (American National Election Study) | US, biennial | Vote choice, attitudes, partisanship | ICPSR; free | Immediate |
| CCES (Cooperative Election Study) | US, annual, N≈50K | Policy attitudes, partisanship, race | Harvard Dataverse; free | Immediate |
| Pew Research datasets | US + global | Attitudes, media, religion, demographics | Pew; free registration | Immediate |
Immigration / ethnicity / language:
| Dataset | Coverage | Key variables | Access | Timeline |
|---|
| CPS-ASEC | US, annual | Nativity, citizenship, language, income | IPUMS; free | Immediate |
| New Immigrant Survey (NIS) | US legal immigrants | Pre-migration, occupation, networks, health | Princeton; free | Immediate |
| ISSP | 40+ countries, cross-national | Attitudes, identity, religion, work | GESIS; free | Immediate |
| WVS / EVS | 80+ countries, 7 waves | Values, trust, democracy, religion | WVS archive; free | Immediate |
| Luxembourg Income Study | 50+ countries, harmonized | Income, wealth, labor | LIS; free registration | 1 week |
Neighborhoods / spatial / administrative:
| Dataset | Coverage | Key variables | Access | Timeline |
|---|
| Decennial Census / ACS tract | US census tracts / blocks | Demographics, SES, housing by place | Census API / IPUMS NHGIS; free | Immediate |
| HMDA (Home Mortgage Disclosure Act) | US, annual | Mortgage applications, denials, race, income | CFPB; free | Immediate |
| TIGER/Line shapefiles | US geography | Boundaries, roads, landmarks | Census; free | Immediate |
| HOLC redlining maps | US cities, 1930s | Neighborhood security grades (A–D) | Mapping Inequality; free | Immediate |
| Opportunity Atlas | US census tracts | Economic mobility by race/income/place | opportunityatlas.org; free | Immediate |
Text / digital / computational:
| Dataset | Coverage | Key variables | Access | Timeline |
|---|
| Congressional Record | US, 1873– | Floor speeches, committee records | GovInfo / Voteview; free | Immediate |
| Common Crawl | Web, petabyte-scale | Text from 3+ billion pages | commoncrawl.org; free | Immediate |
| Google Trends | Global, 2004– | Search interest by topic/region | Google; free API | Immediate |
| Twitter/X historical | Social media | Text, networks, metadata | X API v2 Pro+ ($5K/mo; Academic track discontinued Jan 2025) | 2–4 weeks |
Crime / criminal justice / substance use:
| Dataset | Coverage | Key variables | Access | Timeline |
|---|
| NCVS (National Crime Victimization Survey) | US, annual panel since 1973 | Crime victimization, reporting, demographics | ICPSR; free | Immediate |
| UCR / NIBRS (FBI) | US, annual agency-level + incident | Offenses, arrests, agency demographics | FBI Crime Data Explorer; free | Immediate |
| NSDUH (Nat. Survey on Drug Use and Health) | US, annual 70K+ respondents | Substance use, mental health, treatment, demographics | SAMHSA / ICPSR; free | Immediate |
| National Corrections Reporting Program | US, annual individual-level | Prison admissions, releases, sentences, demographics | ICPSR (restricted); DUA | 2–4 weeks |
| Sentencing Commission Data | US federal courts, 1996– | Sentences, guidelines, demographics, offense types | USSC; free download | Immediate |
International / cross-national surveys:
| Dataset | Coverage | Key variables | Access | Timeline |
|---|
| WVS / EVS | 120+ countries, 7 waves (1981–) | Values, trust, democracy, religion, gender | wvs.worldvaluessurvey.org; free | Immediate |
| ISSP | 40+ countries, annual modules | Attitudes: environment, health, religion, social inequality, work | GESIS; free registration | Immediate |
| ESS (European Social Survey) | 30+ European countries, biennial (2002–) | Trust, politics, immigration, well-being, religion | europeansocialsurvey.org; free | Immediate |
| Eurobarometer | EU member states, biennial (1974–) | EU integration, economy, immigration attitudes | GESIS / ICPSR; free | Immediate |
| Afrobarometer | 39 African countries, 9 rounds (1999–) | Democracy, governance, economy, service delivery | afrobarometer.org; free | Immediate |
| Latinobarómetro | 18 Latin American countries, annual (1995–) | Democracy, trust, economy, social cohesion | latinobarometro.org; free registration | Immediate |
| Asian Barometer | 14 Asian countries, 5 waves | Democracy, governance, social capital | asianbarometer.org; free registration | 1–2 weeks |
| LIS (Luxembourg Income Study) | 50+ countries, harmonized | Income, wealth, labor, demographics | LIS; free registration | 1 week |
| LWS (Luxembourg Wealth Study) | 20+ countries | Wealth, assets, debt | LIS; free registration | 1 week |
| PISA (OECD) | 80+ countries, triennial | Student achievement, school quality, equity | OECD; free | Immediate |
| TIMSS / PIRLS (IEA) | 60+ countries, quadrennial | Math, science, reading achievement | IEA; free | Immediate |
| DHS (Demographic and Health Surveys) | 90+ developing countries | Fertility, mortality, nutrition, HIV, gender | dhsprogram.com; free registration | 1–2 weeks |
| MICS (UNICEF) | 100+ countries, 6 rounds | Child health, education, protection, water/sanitation | mics.unicef.org; free | Immediate |
Economic / labor / macro:
| Dataset | Coverage | Key variables | Access | Timeline |
|---|
| FRED (Federal Reserve Economic Data) | US + global, 800K+ series | GDP, unemployment, CPI, interest rates, exchange rates | FRED API; free key | Immediate |
| Penn World Table (PWT) | 180+ countries, 1950–2019 | Real GDP, capital stock, productivity, purchasing power | rug.nl/ggdc/productivity/pwt; free | Immediate |
| EU-KLEMS | EU + major economies, 1970– | Industry-level growth, productivity, factor inputs | euklems.eu; free | Immediate |
| LEHD (Longitudinal Employer-Household Dynamics) | US, quarterly, tract-level | Employment flows, earnings by age/sex/industry | Census OnTheMap / LODES; free | Immediate |
| OECD Data | 38+ OECD countries | Economic, social, environmental indicators | data.oecd.org; free API | Immediate |
| Eurostat | EU member states | Demographics, economy, trade, environment, health | ec.europa.eu/eurostat; free | Immediate |
| ILO (International Labour Organization) | 180+ countries | Employment, wages, working conditions, migration | ilostat.ilo.org; free | Immediate |
Global health / population / environment:
| Dataset | Coverage | Key variables | Access | Timeline |
|---|
| WHO Global Health Observatory | 194 countries, 1000+ indicators | Mortality, disease burden, health systems, risk factors | who.int/data/gho; free API | Immediate |
| GBD (Global Burden of Disease) | 204 countries, 1990– | DALYs, mortality, morbidity by cause/risk/age/sex | ghdx.healthdata.org; free | Immediate |
| IPUMS International | 100+ countries, census microdata | Demographics, education, employment, housing, migration | ipums.org; free registration | 1–2 weeks |
| World Population Prospects (UN) | Global, UN projections | Fertility, mortality, migration by country | UN; free | Immediate |
| NASA SEDAC | Global, gridded | Population density, land use, air quality, climate | sedac.ciesin.columbia.edu; free | Immediate |
| FAOSTAT | 245+ countries | Agriculture, food security, emissions, land use | fao.org/faostat; free | Immediate |
Science of science / bibliometrics / research data:
| Dataset | Coverage | Key variables | Access | Timeline |
|---|
| OpenAlex | 250M+ works, global | Authors, institutions, citations, topics, open access | openalex.org; free API | Immediate |
| Semantic Scholar | 200M+ papers | Citations, abstracts, embeddings, citation intent | api.semanticscholar.org; free (rate-limited) | Immediate |
| Dimensions | 130M+ publications | Grants, patents, clinical trials, policy documents | app.dimensions.ai; free for research | 1–2 weeks |
| ORCID Public Data | 18M+ researchers | Researcher IDs, affiliations, works | public API; free | Immediate |
| NSF SED (Survey of Earned Doctorates) | US, annual since 1957 | PhD recipients, field, demographics, funding | NCSES; free | Immediate |
| NSF S&E Indicators | US, biennial | R&D spending, STEM workforce, innovation metrics | ncses.nsf.gov; free | Immediate |
| Crossref | 150M+ DOIs | Metadata, citations, funders, licenses | api.crossref.org; free (polite pool) | Immediate |
| Web of Science (Clarivate) | 90M+ records, 1900– | Citations, impact factors, h-index | Institutional license required | Varies |
| MAG successor via OpenAlex | 250M+ works | Paper-author-institution-concept graph | openalex.org; free | Immediate |
General-purpose open data repositories:
| Repository | Scope | Key features | Access | URL |
|---|
| Harvard Dataverse | All disciplines; 75K+ datasets | DOI, API, versioning, rich metadata | Free; open to all | dataverse.harvard.edu |
| ICPSR | Social/behavioral sciences; 500K+ files | Curated, DUA for restricted data, SPSS/Stata/R | Free registration (member institutions) | icpsr.umich.edu |
| Zenodo | All disciplines; CERN-hosted | DOI, 50GB/record, versioning, GitHub integration | Free; open to all | zenodo.org |
| OSF (Open Science Framework) | All disciplines | Preregistration, versioning, integrations (GitHub, Dataverse) | Free; open to all | osf.io |
| Figshare | All disciplines | DOI, 20GB free, embeddable, altmetrics | Free; open to all | figshare.com |
| QDR (Qualitative Data Repository) | Qualitative / multi-method | Annotation, transparency appendices | Free registration | qdr.syr.edu |
| Roper Center (iPoll) | Public opinion; 23K+ datasets, 800K questions | US + international polling data | Institutional membership | ropercenter.cornell.edu |
| Data.gov | US federal/state agencies | Machine-readable government datasets | Free | data.gov |
| UK Data Service | UK surveys + international | Access to major UK studies (Understanding Society, BCS70) | Free registration (UK higher ed) | ukdataservice.ac.uk |
| GESIS (Leibniz Institute) | European social science | Eurobarometer, ISSP, ALLBUS, EVS archive | Free registration | gesis.org |
| Google Dataset Search | Meta-search across repositories | Searches 25K+ repositories worldwide | Free | datasetsearch.research.google.com |
| Kaggle Datasets | ML/data science; 200K+ datasets | Notebooks, competitions, community discussion | Free | kaggle.com/datasets |
| GitHub (curated lists) | Replication data + awesome lists | awesomedata/awesome-public-datasets; replication archives | Free | github.com |
| AWS Open Data | Large-scale scientific data | Satellite imagery, genomics, climate | Free (compute charges for AWS processing) | registry.opendata.aws |
Restricted-access / linked federal data:
| Dataset | Access route | Timeline |
|---|
| SSA earnings + survey links | FSRDC application | 12–24 months |
| IRS/Treasury linked micro-data | FSRDC application | 12–24 months |
| State administrative records | State agency MOU + IRB + DUA | 6–18 months |
| CMS Medicare/Medicaid claims | CMS DUA | 6–12 months |
Additional International and Longitudinal Datasets
| Dataset | Coverage | Key Variables | Access | Category |
|---|
| UK Household Longitudinal Study (UKHLS / Understanding Society) | UK, 2009–present, ~40K households | Income, employment, health, education, ethnicity, wellbeing | Registration at UK Data Service | International |
| German Socio-Economic Panel (GSOEP / SOEP) | Germany, 1984–present, ~30K individuals | Income, employment, education, satisfaction, migration background | Apply at DIW Berlin (2–4 weeks) | International |
| Indian Human Development Survey (IHDS) | India, 2004–05 & 2011–12, ~42K households | Caste, income, education, health, gender, social networks | ICPSR download | International |
| Afrobarometer | Africa (39 countries), 1999–present | Democracy attitudes, governance, identity, inequality | Public download | International |
| Latinobarómetro | Latin America (18 countries), 1995–present | Democracy, institutions, economic perceptions, identity | Registration required | International |
| China Family Panel Studies (CFPS) | China, 2010–present, ~15K households | Income, education, cognition, migration, health | Apply at Peking University | International |
| European Values Study (EVS) | Europe, 1981–2017 (5 waves) | Values, religion, politics, work, family, national identity | GESIS download | International |
| UK Biobank | UK, 500K participants | Genetics, health, imaging, lifestyle, sociodemographics | Application + DUA (restrictive) | Health/Biomedical |
| Fragile Families & Child Wellbeing Study (FFCWS) | US, ~5K births (1998–), 6 waves | Unmarried parents, child development, income, incarceration, housing | OPR Princeton (public + restricted) | Family/Children |
| National Longitudinal Study of Adolescent to Adult Health (Add Health) | US, 1994–2018, ~20K individuals | Social networks, health behaviors, romantic relationships, genetics | Carolina Population Center (public + restricted) | Health/Youth |
Step 0b. Verify dataset fit
For the selected dataset, confirm:
- Unit of analysis matches the RQ (person / household / firm / county / country-year)
- Time period covers the phenomenon of interest
- Key variables (Y, X, M, moderators) are available with adequate measurement quality
- Sample size is sufficient for the planned subgroup analyses (see power analysis in /scholar-design)
- Access pathway is realistic given project timeline
Step 0c. IPUMS access note
IPUMS harmonizes many datasets above (ACS, CPS, GSS, international census, CPS-ASEC). Always check IPUMS first:
ipums.org — IPUMS USA (ACS/Census), IPUMS CPS, IPUMS Internationalgss.norc.org — GSS directly; or IPUMS MICS- API access via
ipumsr R package or ipumspy Python package
library(ipumsr)
ddi <- read_ipums_ddi("usa_00001.xml")
df <- read_ipums_micro(ddi)
Step 0d. Auto-Fetch Open Data (EXECUTE — not just recommend)
When the user specifies or you identify an "Immediate" access public data source from the directory above, DO NOT just produce code templates — actually download the data now. This step converts data-status: no-data into data-status: existing-data.
Create data/ directory and download:
mkdir -p data/raw data/clean
Match the data source and execute the appropriate download:
| Data source | R package / method | Auto-fetch code |
|---|
| ACS / Census tract | tidycensus | See below |
| CPS | ipumsr or direct Census | See below |
| GSS (General Social Survey) | gssr | See below |
| World Bank | WDI | See below |
| BLS (unemployment, CPI) | blsAPI | See below |
| FRED (macro-economic) | fredr | See below |
| NHANES | nhanesA | See below |
| ESS (European Social Survey) | essurvey | See below |
| Eurostat | eurostat | See below |
| OECD | OECD / oecdR | See below |
| Penn World Table | pwt10 / direct download | See below |
| WHO Global Health Observatory | WHO / direct API | See below |
| FAOSTAT | FAOSTAT | See below |
| OpenAlex | openalexR / REST API | See below |
| Harvard Dataverse | dataverse (R) / pyDataverse (Py) | See below |
| Zenodo | REST API | See below |
| College Scorecard | direct CSV | See below |
| Google Trends | gtrendsR | See below |
| GDELT | direct download | See below |
| Opportunity Atlas | direct CSV | See below |
| WVS (World Values Survey) | direct download | See below |
| DHS (Demographic and Health Surveys) | rdhs | See below |
| UCR / NIBRS crime data | crimedata / direct | See below |
| Afrobarometer | direct download | See below |
| IPUMS (any series) | ipumsr | See below |
R auto-fetch templates (execute via Bash):
library(tidycensus)
vars <- c(medinc = "B19013_001", pop = "B01001_001",
pct_bach = "B15003_022", pct_poverty = "B17001_002")
df <- get_acs(geography = "[tract|county|state]",
state = "[STATE]", variables = vars,
year = [YEAR], geometry = FALSE, survey = "acs5")
saveRDS(df, "data/raw/acs-[geography]-[state]-[year].rds")
write.csv(df, "data/raw/acs-[geography]-[state]-[year].csv", row.names = FALSE)
message("Downloaded: ", nrow(df), " rows from ACS")
library(WDI)
df <- WDI(country = "all",
indicator = c(gdp = "NY.GDP.MKTP.CD", pop = "SP.POP.TOTL",
life_exp = "SP.DYN.LE00.IN"),
start = [START_YEAR], end = [END_YEAR], extra = TRUE)
saveRDS(df, "data/raw/wdi-[indicators]-[years].rds")
write.csv(df, "data/raw/wdi-[indicators]-[years].csv", row.names = FALSE)
library(blsAPI)
payload <- list(seriesid = c("[SERIES_ID]"),
startyear = "[START]", endyear = "[END]")
df <- blsAPI(payload, api_version = 2, return_data_frame = TRUE)
saveRDS(df, "data/raw/bls-[series]-[years].rds")
library(fredr)
fredr_set_key(Sys.getenv("FRED_API_KEY"))
df <- fredr(series_id = "[SERIES_ID]",
observation_start = as.Date("[START]"),
observation_end = as.Date("[END]"))
saveRDS(df, "data/raw/fred-[series]-[years].rds")
library(nhanesA)
demo <- nhanes("[CYCLE_DEMO]")
exam <- nhanes("[CYCLE_EXAM]")
df <- merge(demo, exam, by = "SEQN")
saveRDS(df, "data/raw/nhanes-[cycle].rds")
library(gtrendsR)
gt <- gtrends(keyword = c("[TERM1]", "[TERM2]"),
geo = "[GEO]", time = "[START] [END]")
df <- gt$interest_over_time
saveRDS(df, "data/raw/gtrends-[terms]-[dates].rds")
library(gssr)
data(gss_all)
saveRDS(gss_all, "data/raw/gss-cumulative.rds")
message("GSS cumulative: ", nrow(gss_all), " rows, ", ncol(gss_all), " vars")
library(essurvey)
set_email("[YOUR_ESS_EMAIL]")
df <- import_rounds(rounds = c(10, 11))
saveRDS(df, "data/raw/ess-rounds-10-11.rds")
library(eurostat)
df <- get_eurostat("[DATASET_CODE]", time_format = "num")
saveRDS(df, "data/raw/eurostat-[code].rds")
library(OECD)
df <- get_dataset("[DATASET_ID]",
filter = list(c("[COUNTRY_CODES]")),
start_time = [START_YEAR], end_time = [END_YEAR])
saveRDS(df, "data/raw/oecd-[dataset]-[years].rds")
pwt_url <- "https://dataverse.nl/api/access/datafile/:persistentId?persistentId=doi:10.34894/QT5BCC"
download.file(pwt_url, "data/raw/pwt100.xlsx", mode = "wb")
library(readxl)
df <- read_excel("data/raw/pwt100.xlsx", sheet = "Data")
saveRDS(df, "data/raw/pwt10-data.rds")
who_url <- "https://ghoapi.azureedge.net/api/[INDICATOR_CODE]"
df <- jsonlite::fromJSON(who_url)$value
saveRDS(df, "data/raw/who-[indicator].rds")
library(openalexR)
works <- oa_fetch(entity = "works",
search = "[SEARCH_TERM]",
from_publication_date = "[START_DATE]",
to_publication_date = "[END_DATE]",
count_only = FALSE)
saveRDS(works, "data/raw/openalex-[topic]-[dates].rds")
library(dataverse)
Sys.setenv("DATAVERSE_SERVER" = "dataverse.harvard.edu")
df <- get_dataframe_by_name(filename = "[FILENAME]",
dataset = "[DOI]",
server = "dataverse.harvard.edu")
saveRDS(df, "data/raw/dataverse-[name].rds")
library(ipumsr)
ddi <- read_ipums_ddi("[EXTRACT_FILE].xml")
df <- read_ipums_micro(ddi)
saveRDS(df, "data/raw/ipums-[series]-[extract].rds")
library(rdhs)
set_rdhs_config(email = "[YOUR_DHS_EMAIL]", project = "[PROJECT_NAME]")
datasets <- dhs_datasets(countryIds = "[COUNTRY_CODE]", surveyYearStart = [YEAR])
df <- get_datasets(datasets$FileName[1])
saveRDS(df, "data/raw/dhs-[country]-[year].rds")
library(FAOSTAT)
df <- get_faostat_bulk(code = "[DOMAIN_CODE]")
saveRDS(df, "data/raw/faostat-[domain].rds")
import pandas as pd
from pathlib import Path
Path("data/raw").mkdir(parents=True, exist_ok=True)
url = "https://ed-public-download.app.cloud.gov/downloads/Most-Recent-Cohorts-Institution_04192024.zip"
df = pd.read_csv(url, low_memory=False)
df.to_parquet("data/raw/college-scorecard.parquet")
url = "https://opportunityinsights.org/wp-content/uploads/2018/10/tract_outcomes_simple.csv"
df = pd.read_csv(url)
df.to_parquet("data/raw/opportunity-atlas-tracts.parquet")
from datetime import datetime
date_str = datetime.now().strftime("%Y%m%d")
url = f"http://data.gdeltproject.org/events/{date_str}.export.CSV.zip"
df = pd.read_csv(url, sep="\t", header=None)
df.to_parquet(f"data/raw/gdelt-{date_str}.parquet")
import requests
def fetch_openalex_works(search_term, per_page=200, max_pages=5):
"""Fetch works from OpenAlex API (free, no key needed)."""
records = []
for page in range(1, max_pages + 1):
url = (f"https://api.openalex.org/works?search={search_term}"
f"&per-page={per_page}&page={page}"
f"&mailto={__import__('os').environ.get('SCHOLAR_CROSSREF_EMAIL', '')}")
resp = requests.get(url, timeout=30)
data = resp.json()
records.extend(data.get("results", []))
if len(data.get("results", [])) < per_page:
break
return pd.json_normalize(records)
from pyDataverse.api import NativeApi
api = NativeApi("https://dataverse.harvard.edu")
def fetch_zenodo_dataset(record_id):
"""Download files from a Zenodo record (public, no key needed)."""
url = f"https://zenodo.org/api/records/{record_id}"
meta = requests.get(url).json()
for f in meta["files"]:
fname = f["key"]
print(f"Downloading {fname} ({f['size']/(1024**2):.1f} MB)")
resp = requests.get(f["links"]["self"], stream=True)
with open(f"data/raw/{fname}", "wb") as out:
for chunk in resp.iter_content(chunk_size=8192):
out.write(chunk)
return meta["metadata"]["title"]
def fetch_who_indicator(indicator_code):
"""Fetch WHO Global Health Observatory data (free, no key)."""
url = f"https://ghoapi.azureedge.net/api/{indicator_code}"
resp = requests.get(url, timeout=30)
data = resp.json()
return pd.DataFrame(data["value"])
def fetch_eurostat(dataset_code):
"""Fetch Eurostat dataset via bulk download (free, no key)."""
url = f"https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/{dataset_code}?format=TSV&compressed=true"
import io, gzip
resp = requests.get(url, timeout=60)
data = gzip.decompress(resp.content)
return pd.read_csv(io.BytesIO(data), sep="\t")
def fetch_fbi_crime(api_key, offense="burglary", state="US", start=2015, end=2023):
"""FBI Crime Data Explorer API (free key from api.data.gov)."""
url = (f"https://api.usa.gov/crime/fbi/sapi/api/summarized/state/{state}"
f"/{offense}/{start}/{end}?API_KEY={api_key}")
return pd.DataFrame(requests.get(url).json()["results"])
After successful download:
- Confirm the file exists and report row count:
nrow(df) / len(df)
- Log the download to
data/raw/download-manifest.md:
| File | Source | Date fetched | N rows | Variables |
|------|--------|-------------|--------|-----------|
| acs-tract-IL-2022.rds | tidycensus ACS 5-year | 2026-03-03 | 15,420 | medinc, pop, pct_bach, pct_poverty |
- Update PROJECT STATE: set
data-status: existing-data and Data File(s): data/raw/[filename]
- Proceed with all downstream skills in DATA-AVAILABLE MODE — no more
[CODE-TEMPLATE] or [PLACEHOLDER]
If the download fails — API key missing (MOST COMMON):
Before falling back to CODE-TEMPLATE, ask the user for the missing API key. Most open data APIs offer free keys:
Prompt the user:
To download [SOURCE] data, I need an API key.
You can get a free key here: [URL]
Please provide your API key, or I can proceed with code templates instead.
If the user provides a key:
- Set it in the R environment:
Sys.setenv(CENSUS_API_KEY = "[key]") or equivalent
- Save it to
.Renviron for future sessions: cat('CENSUS_API_KEY=[key]\n', file = "~/.Renviron", append = TRUE)
- Retry the download
- If successful → upgrade
data-status to existing-data
Sources that do NOT require API keys (always attempt these first):
- World Bank (
WDI) — no key needed
- NHANES (
nhanesA) — no key needed
- GSS (
gssr) — no key needed
- Google Trends (
gtrendsR) — no key needed
- BLS (
blsAPI v1) — no key needed (v2 optional key for higher limits)
- College Scorecard — direct CSV download, no key needed
- Opportunity Atlas — direct CSV download, no key needed
- GDELT — direct download, no key needed
- Direct URL datasets (GitHub, OSF, Dataverse) — no key needed
- OpenAlex (
openalexR) — no key needed (use mailto for polite pool)
- Eurostat (
eurostat) — no key needed
- WHO GHO API — no key needed
- Penn World Table — direct download, no key needed
- FAOSTAT (
FAOSTAT) — no key needed
- Harvard Dataverse (
dataverse / pyDataverse) — no key for public datasets
- Zenodo REST API — no key for public records
- Semantic Scholar API — no key needed (rate-limited; optional key for higher limits)
- Crossref API — no key needed (use
mailto for polite pool)
- Afrobarometer — direct download after free registration
- IPUMS (
ipumsr) — free registration required, then direct download
- UCR / NIBRS crime data — direct download from FBI Crime Data Explorer
- NASA SEDAC — free registration, direct download
- Data.gov — direct CSV/JSON downloads, no key needed
Sources requiring free registration (attempt after no-key sources):
- ESS (
essurvey) — free registration at europeansocialsurvey.org
- OECD (
OECD) — free API, no key but registration recommended
- DHS (
rdhs) — free registration + approved project at dhsprogram.com
- WVS — free registration at worldvaluessurvey.org
- ISSP — free registration at GESIS
- Eurobarometer — free registration at GESIS
- Latinobarómetro — free registration at latinobarometro.org
- LIS / LWS — free registration at lisdatacenter.org
- UK Data Service — free registration (UK higher ed)
- Roper Center iPoll — institutional membership required
If the download fails for other reasons (network error, rate limit, package not installed):
- Log the specific error message
- If package missing: attempt
install.packages("[pkg]") and retry once
- If network error: inform user and fall back to
[CODE-TEMPLATE]
- Keep
data-status: no-data and note the reason in PROJECT STATE
WORKFLOW 1: Variable Dictionary and Measurement Plan
Produce a formal variable dictionary — the backbone of your Methods section and IRB application. Run after identifying the dataset (WORKFLOW 0) and confirming the design (/scholar-design).
Step 1a. Variable dictionary table
For every variable in the analytic model, document:
| Role | Variable name | Construct | Operationalization | Source question / admin field | Type | Range / categories | Notes |
|---|
| Y (Outcome) | earnings_ln | Annual labor market earnings | Log of annual earnings ($) | PSID Q_annual_earnings | Continuous | 0–∞ (log-transformed) | Top-coded at 99th pct |
| X (Key predictor) | immigrant | Immigrant status | Born outside US (1=yes) | ACS NATIVITY | Binary | 0/1 | — |
| M (Mediator) | english_prof | English language proficiency | 4-point self-report scale | CPS SPEAKENG | Ordinal | 1=not at all … 4=very well | — |
| W (Moderator) | race_eth | Race/ethnicity | Self-identified 5-category | ACS RACE + HISPAN | Categorical | White/Black/Hispanic/Asian/Other | ref = White |
| C (Control) | educ_yrs | Years of schooling | Recoded from education categories | ACS EDUC | Continuous | 0–20 | Midpoints used |
| C | age | Age in years | — | ACS AGE | Continuous | 25–64 | Working-age restriction |
| C | female | Female | Sex == female | ACS SEX | Binary | 0/1 | — |
| FE / cluster | state | State | State FIPS code | ACS STATEFIP | Categorical | 51 states + DC | For FE or clustering |
Variable Dictionary Template
| Variable Name | Label | Type | Values/Range | Source Item | Coding Notes |
|---|
income_hh | Household annual income | Continuous ($) | 0–999,999 | "Total household income last year" | Top-coded at $999,999; log-transform recommended |
educ_yrs | Years of education | Continuous (years) | 0–25 | "Highest grade completed" | Recode GED=12; professional degree=20 |
race_eth | Race/ethnicity (5 categories) | Categorical | 1=White NH, 2=Black NH, 3=Hispanic, 4=Asian NH, 5=Other | Combined from race + Hispanic origin | Standard Census categories |
employed | Currently employed | Binary | 0=No, 1=Yes | "Did you work last week?" | Missing if not in labor force (code separately) |
missing_code | — | — | NA / -9 / .d / .r | — | NA=system missing; -9=refused; .d=don't know; .r=refused |
Scale direction convention: Always code so higher values = more of the construct (e.g., 1=strongly disagree → 5=strongly agree). Reverse-code items before analysis.
Variable naming convention: [concept]_[modifier] (e.g., income_hh, income_per_capita, educ_yrs, educ_degree).
Step 1b. Measurement validity checklist
For each key construct, verify:
- Face validity: Does the operationalization obviously measure the concept?
- Content validity: Does it cover all relevant aspects of the construct?
- Construct validity: Are established scales being used? Is there prior evidence of reliability (α, test-retest)?
- Measurement equivalence: Does the measure mean the same thing across the groups being compared (e.g., across racial groups, immigration cohorts)?
Step 1c. Data blueprint summary
Produce a one-page data blueprint:
DATA BLUEPRINT
─────────────────────────────────
Research question: [RQ from Phase 0/1]
Dataset: [Name, year/wave, N]
Unit of analysis: [person / household / tract / ...]
Analytic sample: [population restriction + expected N]
Outcome (Y): [variable name + operationalization]
Key predictor (X): [variable name + operationalization]
Mediator(s) (M): [if applicable]
Moderator(s) (W): [if applicable]
Controls: [list]
Design: [cross-sectional / panel / DiD / RD / IV / matching]
Fixed effects / clusters: [unit + SE clustering level]
Weights: [survey weight variable name, if applicable]
Missing data strategy: [listwise / MI — based on /scholar-eda]
WORKFLOW 2: Survey Instrument Design
Step 1: Define Measurement Goals
For each construct, specify:
- Conceptual definition: What is the construct theoretically?
- Operational definition: What observable behavior/attitude/characteristic measures it?
- Measurement level: Nominal, ordinal, interval, ratio
- Reference period: "In the past 12 months," "currently," "ever"
Step 2: Question Construction
Question types and when to use:
| Type | Best For | Example |
|---|
| Single-item Likert | Attitudes with established scales | "Strongly agree → Strongly disagree" |
| Multi-item scale | Latent constructs (trust, identity) | Average of 5 items; report α |
| Open-ended | Unexpected responses, sensitive topics | "In your own words, describe…" |
| Numeric entry | Factual, bounded | "How many years have you lived in the US?" |
| Matrix / grid | Battery of related items | Multiple rows, shared response scale |
| Ranking | Relative preference | "Rank the following from most to least…" |
| Filtered / branching | Subpopulation follow-ups | "If yes → [follow-up]" |
Question wording rules:
- One concept per question (no double-barreled: "Do you trust neighbors and local government?")
- Avoid negations
- 8th-grade reading level
- Offer "Don't know" / "Prefer not to answer" where appropriate
- Consistent response direction (higher = more of the construct)
- Match scale length to construct precision needed (5-pt standard; 7-pt for fine-grained attitudes)
Established scales to use (cite if using):
- Social trust: Rosenberg Trust Scale; GSS trust items
- Subjective SES: MacArthur Ladder (1–10)
- Discrimination: Everyday Discrimination Scale (Williams et al. 1997)
- Mental health: PHQ-9 (depression), GAD-7 (anxiety)
- Political attitudes: ANES scales
- Immigrant identity: MEIM-R (Phinney & Ong 2007)
- Social network: Burt (1984) name generator + name interpreter
See references/survey-design.md for full item batteries.
Step 3: Survey Organization
Recommended order:
- Introduction screen (purpose, consent, confidentiality)
- Screener questions (eligibility)
- Salient but non-threatening items first
- Sensitive items (income, discrimination, illegal behavior) later
- Sociodemographics last
- Debriefing / thank-you screen
Sample size for surveys:
| Goal | Rule of thumb | Notes |
|---|
| Descriptive estimates (proportions) | N ≥ 400 for ±5% margin (95% CI) | N ≥ 1,000 for subgroup comparisons |
| Regression with 10 predictors | N ≥ 500 for OLS (Cohen's rule: 50 + 8k) | N ≥ 1,000 for logistic |
| Experimental contrast (d = 0.3, 80% power) | N ≥ 352 per arm | Use pwr::pwr.t.test() |
| Conjoint / AMCE (1% precision) | N ≥ 1,000, 3–5 tasks | More profiles = better precision |
| Multilevel (ICC = 0.10, L2 effects) | N ≥ 30 groups × 30 within | Use pwr2ppl or simr |
Online panel quality tiers:
| Tier | Panel | Quality | Cost | Best for |
|---|
| Probability-based | IPSOS KnowledgePanel, Amerispeak | Highest | $$$ | Top-tier publication; nationally representative |
| High-quality opt-in | Prolific Academic | High | $$ | Academic research; attention checks built-in |
| MTurk | MTurk (via CloudResearch) | Medium | $ | Pilot testing; not recommended for main analysis |
| Convenience | Qualtrics panel, Lucid | Variable | $–$$ | Pretest; not representative |
Data quality checks:
- Completion time < ⅓ median → flag inattentive
- Straight-lining on Likert batteries
- Instructed response item failures (attention checks)
- Open-ended gibberish / copy-paste
- IP duplicates
Step 4: Pilot and Cognitive Testing
- Expert review (2–3 colleagues; face validity)
- Cognitive interviewing (5–10 respondents; 4-probe protocol — see survey-design.md)
- Pilot survey (N = 20–50; check distributions, timing, drop-off)
- Psychometric checks: CFA for multi-item scales; remove items loading < 0.4
Step 5: Fielding and Documentation
- Platform: Qualtrics (complex routing) > Prolific (probability-based recruitment) > REDCap (clinical/HIPAA)
- AAPOR response rate documentation (RR1–RR6) required in Methods
- Non-response analysis: compare respondents to non-respondents or population benchmarks
Step 6: Experimental Survey Modules
Step 6a: Vignette / Factorial Survey Experiment
A factorial survey presents randomized descriptions (vignettes) of hypothetical persons or situations and asks respondents to evaluate them. Used to measure preferences, discrimination, or decision-making while controlling for confounds.
Design:
- 2–5 dimensions varied; 2–4 levels per dimension → full or fractional factorial design
- Each respondent rates 3–8 vignettes (avoid fatigue)
- Analyze with OLS clustered by respondent (AMCE: Average Marginal Component Effect)
Vignette template:
[Respondent instructions]:
"Below is a description of a job applicant. Please read carefully and rate
how likely you would be to invite them for an interview."
Job applicant profile:
Name: [RANDOMIZED: Michael Johnson / DeShawn Jackson / José García]
Education: [RANDOMIZED: High school diploma / Bachelor's degree / Master's degree]
Years of exp.: [RANDOMIZED: 2 years / 5 years / 10 years]
Criminal record: [RANDOMIZED: None / Misdemeanor / Felony]
"How likely would you be to invite this applicant for an interview?"
(1) Very unlikely — (2) Unlikely — (3) Neutral — (4) Likely — (5) Very likely
Analysis (R):
library(cregg)
amce_results <- cj(data = df, formula = outcome ~ name + education + experience + criminal,
id = ~respondent_id)
plot(amce_results) + theme_Publication()
amce_sub <- cj(df, outcome ~ name + education, id = ~respondent_id,
by = ~respondent_race)
Report: "We used a factorial survey experiment with [N] vignettes and [K] randomized dimensions. Average marginal component effects (AMCEs; Hainmueller, Hopkins & Yamamoto 2014) were estimated using OLS with standard errors clustered by respondent."
Step 6b: List Experiment (Sensitive Item Measurement)
List experiments allow respondents to reveal sensitive attitudes (racism, illegal behavior, stigmatized conditions) without directly endorsing them — protecting against social desirability bias.
Design (split-half):
- Control group: 3 innocuous items → "How many of the following have you done in the past year?"
- Treatment group: same 3 items + 1 sensitive item
- Difference in means = % endorsing sensitive item
Template:
[Control condition]:
"Below is a list of things some people have done. Please tell us
HOW MANY you have done in the past year — not which ones, just how many."
1. Watched a movie at home
2. Donated to a charity
3. Attended a religious service
[Answer: 0 / 1 / 2 / 3]
[Treatment condition — adds sensitive item]:
1. Watched a movie at home
2. Donated to a charity
3. Attended a religious service
4. [Sensitive: e.g., "Used marijuana" / "Supported restricting immigration"]
[Answer: 0 / 1 / 2 / 3 / 4]
Analysis (R):
library(list)
lexp <- ict.test(df$count, df$treatment, J = 3, gms = TRUE)
summary(lexp)
Sample size: N ≥ 500 per arm (N ≥ 1,000 total) for ±5% precision. List experiments are statistically inefficient — plan for larger samples than direct questions would require.
WORKFLOW 3: Qualitative Interview Protocol
Step 1: Protocol Architecture
A qualitative interview guide has three parts:
- Preamble: introduce yourself, state purpose, obtain consent, explain recording
- Core guide: 8–15 open-ended questions organized thematically (3–5 themes)
- Closing: ask for what was missed; offer referrals if sensitive topics were discussed
Sampling strategy:
| Strategy | Purpose | Typical N |
|---|
| Maximum variation | Capture full range of relevant variation | 20–40 |
| Homogeneous | Depth within one type | 15–25 |
| Theoretical (grounded theory) | Continue until conceptual saturation | 15–30 |
| Critical case | Test theory on hardest case | 5–15 |
| Snowball | Hard-to-reach populations | Varies |
Saturation typically occurs at N = 15–25 for a single population; more for comparative multi-group designs.
Step 2: Question Sequence
See references/interview-protocols.md for full template. Core structure:
- Rapport opener (biographical, low-stakes)
- Grand tour question (typical day / typical encounter with X)
- Thematic questions (2–4 per theme) with explicit probes
- Contrast / comparison questions
- Meaning / interpretation questions
- Closing ("What haven't I asked?")
Probing types: elaboration / clarification / example / contrast / emotional (use sparingly)
Step 3: Sensitive Topics Protocol
- State upfront which topics will be covered; allow opt-out before starting
- Use distancing language: "Some people I've talked with say… What's your experience?"
- Never press for more detail than offered
- Prepare referral resources (hotlines, social services) before fieldwork
- Debrief at close: "We covered some difficult topics — how are you feeling?"
Step 4: Focus Groups (alternative format)
Use when group interaction is the phenomenon of interest (e.g., deliberation, norm formation, collective sense-making). Not a substitute for in-depth interviews when individual experience is the target.
- Group size: 5–8 participants (too small → insufficient interaction; too large → some dominated)
- Number of groups: 3–6 per subgroup; stop when theoretical saturation is reached
- Facilitator role: introduce topics, manage dominant voices, ensure quieter participants contribute
- Co-facilitator: takes notes on nonverbal dynamics while facilitator runs discussion
- Analyze turn-taking patterns alongside content
Step 5: Online / Video Interviews
- Zoom / Teams: adequate for most semi-structured interviews; less rapport than in-person
- Provide tech support beforehand; confirm platform access and audio/video
- Record with consent (built-in recording; confirm jurisdiction-specific legality)
- Background and setting: note if participant is in private vs. semi-public space
- Transcription: Otter.ai / Whisper for automatic draft; always human-review for accuracy
Step 6: Multilingual Fieldwork
- Translate instrument using forward-backward procedure (translate → back-translate → reconcile)
- Use bilingual interviewers; never rely solely on respondent to translate
- Conduct interviews in respondent's preferred language; document language used
- Analysis: code from transcript in original language; translate selected quotes for publication
- Report language of interview and translation procedure in Methods
WORKFLOW 4: Administrative and Secondary Data
Step 1: Data Access Strategy
| Access type | Route | Timeline |
|---|
| Public use micro-data (ACS, CPS, GSS) | IPUMS, Census, ICPSR — register and download | Immediate |
| Restricted federal data (NLSY restricted, HRS geocoded) | ICPSR DUA | 2–4 weeks |
| FSRDC-linked data (SSA + IRS + survey) | FSRDC application (PI must be Census employee or external researcher) | 12–24 months |
| State administrative records (Medicaid, court, school) | State agency MOU + institutional DUA + IRB | 6–18 months |
| Private / commercial data (credit, social media, employer) | Data sharing agreement; often requires fee or in-kind contribution | Varies |
Step 2: Data Documentation Review
Before analysis, document for every source:
- Survey instrument or administrative codebook / data dictionary
- Variable names, labels, value codes (distinguish "don't know" / "refused" / "not applicable")
- Sampling design: stratification, clustering, probability weights
- Panel structure (if longitudinal): wave IDs, response rates, attrition rates by wave
- Geographic identifiers and level of geography available (state / PUMA / tract)
- Top-coding and suppression rules (especially in restricted-use files)
Step 3: Data Linkage
When merging two or more data sources:
- Document matching variables (name, SSN, address, date of birth, geographic ID)
- Report match rate and characterize match vs. non-match cases (selective linkage is a validity threat)
- Probabilistic record linkage (when exact matching fails):
library(fastLink)
matches <- fastLink(
dfA = df1, dfB = df2,
varnames = c("fname", "lname", "dob", "zip"),
stringdist.match = c("fname", "lname"),
numeric.match = "zip",
threshold.match = 0.85
)
matched <- getMatches(dfA = df1, dfB = df2, fl.out = matches)
- Report concordance rates for key variables across sources
Step 4: Geographic / Spatial Data
library(tidycensus)
library(sf)
tracts <- get_acs(
geography = "tract", state = "IL", county = "Cook",
variables = c(medinc = "B19013_001", pct_black = "B03002_004"),
year = 2022, geometry = TRUE
)
library(tigris)
counties_sf <- counties(state = "IL", cb = TRUE, year = 2022)
df_spatial <- df %>% left_join(tracts, by = c("tract_fips" = "GEOID"))
For historical geographic data: IPUMS NHGIS provides Census tract boundaries and variables for every decade since 1790.
WORKFLOW 5: IRB and Research Ethics
Step 1: Determine IRB Level
Exempt categories (45 CFR 46.104 — post-2018 Common Rule):
| Category | Description | Example |
|---|
| 1 | Normal educational practices in established educational settings | Classroom curriculum study |
| 2 | Anonymous surveys / interviews / observations (no sensitive topics; no identifiers) | Online survey on public attitudes |
| 2 (sensitive) | Survey / interview on sensitive topics but subjects are 18+ and disclosure would not harm them | Discrimination survey with adult workers |
| 3 | Benign behavioral interventions with verbal consent | Random assignment to information treatments |
| 4 | Secondary analysis of existing identifiable data if investigator cannot identify subjects | Analysis of de-identified medical records |
| 5 | Federal research on public benefit programs | Survey of SNAP recipients |
| 6 | Taste / food quality evaluation studies | Food preference study |
| 7 | Storage / maintenance of identifiable data for future secondary use | Biospecimen repository |
| 8 | Broad consent research using identifiable biospecimens | Genetic research |
Expedited review (minimal risk + falls into expedited categories): most online surveys with adults, in-depth interviews without vulnerable populations, secondary analysis of identifiable data with DUA.
Full board review: vulnerable populations (minors, prisoners, pregnant women); deception studies; research in international settings with different norms; studies involving more than minimal risk.
Step 2: Waiver of Consent
A waiver of written (or all) consent may be granted when:
- The research involves no more than minimal risk
- Waiver will not adversely affect subjects' rights and welfare
- Research could not practicably be carried out without the waiver
- When appropriate, subjects will be provided pertinent information afterward
Online surveys with anonymous responses typically qualify for waiver of documented consent (checkbox consent screen suffices).
Step 3: IRB Application Components
- Research protocol: purpose, procedures, population, timeline
- Consent form / script: lay language; institutional template required
- Survey instrument or interview guide: submit with application
- Recruitment materials: scripts, flyers, emails, social media posts
- Data security plan: storage, encryption, access controls, destruction timeline
- CITI training certificates: Human Subjects Research (social-behavioral track)
- Conflict of interest disclosure
Step 4: Data Security Standards
- Remove direct identifiers (name, address, SSN, email) → replace with study IDs
- Store PII separately from research data in encrypted folder
- Use institutional storage for identified data (not personal Google Drive / Dropbox)
- Encrypt files at rest (VeraCrypt, BitLocker) and in transit (SFTP, not FTP)
- Specify retention period in IRB protocol (standard: 3 years post-publication; NIH: 7 years)
WORKFLOW 6: Data Management Pipeline
Step 1: Directory Structure
OUTPUT_ROOT="${OUTPUT_ROOT:-output}"
mkdir -p data/{raw,clean,analysis,codebooks} \
docs/{irb,instruments,protocols} \
scripts \
"${OUTPUT_ROOT}"/{tables,figures}
chmod -w data/raw/
Maintain strict separation:
project/
├── data/
│ ├── raw/ ← NEVER modify; treat as read-only
│ ├── clean/ ← after cleaning; reproducible from raw
│ ├── analysis/ ← analytic sample used in paper
│ └── codebooks/ ← variable documentation
├── scripts/
│ ├── 01_download.R (or .do)
│ ├── 02_clean.R
│ ├── 03_eda.R
│ ├── 04_main_models.R
│ ├── 05_robustness.R
│ └── 06_figures.R
├── output/[slug]/
│ ├── tables/
│ └── figures/
└── docs/
├── codebook.md
├── data_cleaning_log.md
└── pre_analysis_plan.md
Step 2: Git Version Control for Code
git init
git add scripts/ docs/ output/[slug]/
git commit -m "Initial project setup"
cat > .gitignore << 'EOF'
data/raw/
data/clean/*.rds
data/clean/*.dta
data/analysis/
.Renviron
.env
*.key
.DS_Store
*.Rproj.user/
__pycache__/
*.pyc
EOF
Best practice: Commit scripts and outputs (tables, figures) to git; never commit raw data or identified data files. Use .Renviron or .env for API keys.
Step 3: Codebook
Create a codebook with variable-level documentation:
| var_name | var_label | type | values | missing_codes | source | notes |
|---|
| id | Unique respondent ID | string | — | — | Generated | Never use for analysis |
| educ_yrs | Years of education | integer | 0–25 | NA | Q14; recoded | midpoints: HS=12, BA=16 |
| income_1k | Annual HH income ($1000s) | numeric | 0–250 | NA | Q22 | top-coded at $250K |
R codebook tools:
library(codebook); library(labelled)
var_label(df$educ_yrs) <- "Years of formal education completed"
val_labels(df$educ_cat) <- c("Less than HS" = 1, "HS" = 2, "Some college" = 3, "BA" = 4, "Grad" = 5)
codebook(df)
See references/data-management.md for R and Stata cleaning pipeline templates.
Step 4: Data Sharing Plan (DMP)
Required for NSF/NIH grants and increasingly expected at Nature journals and Demography.
DMP template:
DATA MANAGEMENT PLAN — [Project Title]
PI: [Name] | Institution: [Name] | Date: [Date]
1. DATA TYPES AND SOURCES
[Describe: survey data / administrative records / observational / experimental;
estimated file sizes; formats (RDS, CSV, DTA, shapefiles)]
2. DATA COLLECTION AND STORAGE
Collection method: [Qualtrics / IPUMS download / administrative transfer]
Storage: [Institutional server + encrypted external backup]
Backup frequency: [Daily automated backup; monthly off-site]
Retention period: [3 years post-publication / 7 years (NIH) / permanent (if public value)]
3. DATA SHARING AND ACCESS
Sharing plan: [Full public release / Restricted access / Unable to share (reason)]
Repository: [Harvard Dataverse / ICPSR / Zenodo / OSF]
Timeline: [Upon publication / within 12 months of completion]
Format: [CSV + codebook + README (non-proprietary preferred)]
Access restrictions: [None / IRB-approved researchers only / licensed use]
4. PRIVACY AND CONFIDENTIALITY
Identifiers: [Removed per IRB protocol; study IDs only in shared files]
Disclosure risk: [Suppression rules applied; k-anonymity ≥ 5 for geographic vars]
Consent: [Participants consented to data sharing / anonymous; no consent needed]
5. ROLES AND RESPONSIBILITIES
PI: [responsible for compliance, repository deposit]
Co-I/RA: [responsible for data preparation and documentation]
Step 5: Naming Conventions and Version Control
Variables: lowercase, underscores, no special characters
- Suffixes:
_cat (categorical), _bin (binary), _ln (log), _std (standardized), _w (winsorized)
- Wave prefixes:
w1_, w2_ for panel data
Files: lowercase with hyphens; include date for versioned files
gss-2022-clean.rds, analysis-data-2024-01-15.rds
Scripts: numbered for execution order (see directory structure above)
WORKFLOW 7: Web Scraping and Digital Data Collection
Use when primary data must be collected from websites, social media platforms, news archives, or public APIs. Always attempt API-first; fall back to HTML scraping only when no API exists.
Step 1: Ethical and Legal Framework
Before any scraping, review:
| Check | Requirement | Action if violated |
|---|
robots.txt | Check https://site.com/robots.txt; respect Disallow paths | Do not scrape disallowed paths |
| Terms of Service | Read ToS; many platforms explicitly prohibit scraping | Use official API or request data license |
| Rate limits | Impose delays ≥ 1–3 sec between requests | Use polite (R) or time.sleep() (Python) |
| CFAA | U.S. Computer Fraud and Abuse Act — public data generally protected (hiQ v. LinkedIn 2022) | Document that data was publicly accessible without authentication |
| GDPR / privacy | EU personal data requires legal basis for collection | Anonymize or aggregate; consult IRB |
| IRB | Public posts generally exempt (Category 4); semi-public or identifiable data may require expedited review | Document IRB determination before collection |
Polite scraping principles:
- Identify your bot with an academic user-agent:
"Research bot — [Name], [Institution] ([email])"
- Randomize delays:
Sys.sleep(runif(1, 1, 3)) / time.sleep(random.uniform(1, 3))
- Cache responses to avoid re-requesting the same pages
- Scrape during off-peak hours to minimize server load
Web Scraping: Legal and Ethical Checklist
Before scraping, verify:
Platform-specific guidance:
| Platform | Access Method | Rate Limit | Key Restriction |
|---|
| Twitter/X | X API v2 (Basic/Pro/Enterprise tiers; Academic Research track discontinued Jan 2025) | Varies by tier (Basic: 10K reads/mo free; Pro: 1M reads/mo $5K/mo) | Must not share raw tweet text; share IDs only; historical full-archive requires Pro+ |
| Reddit | Reddit API (register app) | 60 req/min (OAuth) | Respect subreddit rules; user consent not required for public posts |
| News sites | Web scraping + newspaper3k | Varies (check robots.txt) | Fair use for research; do not republish full text |
| Congressional Record | congress.gov API | No strict limit | Public domain; no restrictions |
Step 2: API-First Strategy
Always prefer an official API over HTML scraping. Prioritize by data type:
Social media:
| Platform | API / Access route | Package | Notes |
|---|
| Twitter / X | X API v2 (Basic: free 10K reads/mo; Pro: $5K/mo 1M reads; Enterprise: custom; Academic Research track discontinued Jan 2025) | academictwitteR (R, limited to existing tokens), tweepy (Py) | Pro+ required for full-archive search; Basic tier severely rate-limited; consider Bluesky AT Protocol as alternative |
| Reddit | Reddit API v2 + PRAW | RedditExtractoR (R), praw (Py) | Free; rate limits enforced; Pushshift archive restricted as of 2023 |
| Facebook / Instagram | Meta Content Library (replaces CrowdTangle) | contentid + Meta API | Application required; access for academic researchers |
| TikTok | TikTok Research API | REST + Python SDK | Application required; limited to public content |
| YouTube | YouTube Data API v3 | tuber (R), google-api-python-client (Py) | Free quota: 10,000 units/day |
News and text:
| Source | API / Access | Package | Notes |
|---|
| GDELT | BigQuery or direct download | Rgdelt (R), pandas direct | Free; global news events + tone + location + URLs |
| MediaCloud | mediacloud.org | mediacloud (Py) | Story-level coverage and framing; requires registration |
| NewsAPI | newsapi.org | REST | Free tier: 1 month history; developer tier: $449/mo |
| Internet Archive | CDX API + Wayback Machine | wayback (R), requests CDX | Full historical web; excellent for longitudinal designs |
Government and social science APIs:
library(blsAPI)
bls_data <- blsAPI(list(seriesid = "LNS14000000", startyear = "2010", endyear = "2023"))
library(WDI)
wdi <- WDI(country = c("US", "MX", "CA"), indicator = "NY.GDP.MKTP.CD",
start = 2000, end = 2022)
library(fredr)
fredr_set_key(Sys.getenv("FRED_API_KEY"))
unemp <- fredr(series_id = "UNRATE", observation_start = as.Date("2000-01-01"))
Step 3: Static HTML Scraping — R (rvest + polite)
Use for sites with HTML-rendered content (no JavaScript required):
library(rvest)
library(polite)
library(tidyverse)
session <- bow(
url = "https://example.com/articles",
user_agent = "Academic research bot (set your email in SCHOLAR_CROSSREF_EMAIL)",
delay = 2
)
page <- scrape(session)
titles <- page %>% html_nodes("h2.article-title") %>% html_text(trim = TRUE)
links <- page %>% html_nodes("a.article-link") %>% html_attr("href")
dates <- page %>% html_nodes("span.pub-date") %>% html_text(trim = TRUE)
df_page <- tibble(title = titles, url = links, date = dates)
base_url <- "https://example.com/articles?page="
results <- map_dfr(1:20, function(p) {
nod(session, paste0(base_url, p)) %>%
scrape() %>%
{tibble(
title = html_nodes(., "h2.article-title") %>% html_text(trim = TRUE),
url = html_nodes(., "a.article-link") %>% html_attr("href"),
date = html_nodes(., "span.pub-date") %>% html_text(trim = TRUE)
)}
}, .progress = TRUE)
saveRDS(results, "data/raw/scraped-articles-raw.rds")
write_csv(results, "data/raw/scraped-articles-raw.csv")
message("Scraped N = ", nrow(results), " articles")
Selector identification: In Chrome/Firefox, right-click an element → Inspect → right-click in DevTools → Copy → Copy selector. Prefer class-based selectors (.article-title) over position-based (div > p:nth-child(3)).
Step 4: Static HTML Scraping — Python (requests + BeautifulSoup)
import requests
from bs4 import BeautifulSoup
import pandas as pd
import time, random, json
from pathlib import Path
HEADERS = {
"User-Agent": "Academic research bot (set your email in SCHOLAR_CROSSREF_EMAIL)"
}
session = requests.Session()
session.headers.update(HEADERS)
def scrape_page(url: str, delay: tuple = (1.0, 3.0)) -> BeautifulSoup:
"""Politely fetch and parse a single page."""
time.sleep(random.uniform(*delay))
resp = session.get(url, timeout=15)
resp.raise_for_status()
return BeautifulSoup(resp.text, "html.parser")
def parse_article(soup: BeautifulSoup) -> dict:
return {
"title": soup.select_one("h2.article-title").get_text(strip=True),
"date": soup.select_one("span.pub-date").get_text(strip=True),
"body": " ".join(p.get_text(strip=True)
for p in soup.select("div.article-body p"))
}
records = []
for page_num in range(1, 21):
soup = scrape_page(f"https://example.com/articles?page={page_num}")
for card in soup.select("div.article-card"):
records.append({
"title": card.select_one("h2").get_text(strip=True),
"url": card.select_one("a")["href"],
"date": card.select_one("span.date").get_text(strip=True)
})
print(f"Page {page_num}: {len(records)} articles so far")
df = pd.DataFrame(records)
Path("data/raw").mkdir(parents=True, exist_ok=True)
df.to_csv("data/raw/scraped-articles-raw.csv", index=False)
print(f"Saved {len(df)} articles")
Step 5: Dynamic / JavaScript-Rendered Pages
For single-page applications (React, Vue) or pages that load content via XHR after initial HTML load:
R — chromote (recommended; headless Chrome):
library(chromote)
library(rvest)
b <- ChromoteSession$new()
b$Page$navigate("https://example.com/dynamic-page")
b$Page$loadEventFired()
Sys.sleep(2)
html <- b$Runtime$evaluate("document.documentElement.outerHTML")$result$value
page <- read_html(html)
b$close()
titles <- page %>% html_nodes("h2.article-title") %>% html_text(trim = TRUE)
Python — playwright (recommended over selenium):
from playwright.sync_api import sync_playwright
import time
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.set_extra_http_headers({
"User-Agent": "Academic research bot — [Your Name], [Your University]"
})
page.goto("https://example.com/dynamic-page")
page.wait_for_selector("div.article-list")
time.sleep(1)
while page.locator("button.load-more").count() > 0:
page.locator("button.load-more").click()
page.wait_for_load_state("networkidle")
time.sleep(random.uniform(1, 2))
html = page.content()
browser.close()
soup = BeautifulSoup(html, "html.parser")
Install: pip install playwright && playwright install chromium / install.packages("chromote")
Step 6: Social Media Data Collection
Twitter / X (X API v2)
Note (updated 2026): The Academic Research track was discontinued in January 2025. The code below still works with X API v2 Bearer tokens, but full-archive search now requires a Pro ($5K/mo) or Enterprise tier. The free Basic tier allows only 10K tweet reads/month. For new social media research projects, consider Bluesky (AT Protocol, free firehose access) or Meta Content Library as alternatives.
library(academictwitteR)
set_bearer()
tweets <- get_all_tweets(
query = '"redlining" lang:en -is:retweet',
start_tweets = "2020-01-01T00:00:00Z",
end_tweets = "2023-12-31T23:59:59Z",
n = 50000,
data_path = "data/raw/tweets/"
)
df_tweets <- bind_tweets(data_path = "data/raw/tweets/", output_format = "tidy")
import tweepy, json, os
client = tweepy.Client(bearer_token=os.environ["TWITTER_BEARER"], wait_on_rate_limit=True)
paginator = tweepy.Paginator(
client.search_all_tweets,
query = '"redlining" lang:en -is:retweet',
tweet_fields = ["created_at", "author_id", "text", "public_metrics", "geo"],
start_time = "2020-01-01T00:00:00Z",
end_time = "2023-12-31T23:59:59Z",
max_results = 500
)
tweets = [t for page in paginator for t in page.data or []]
with open("data/raw/tweets.jsonl", "w") as f:
for t in tweets:
f.write(json.dumps(t.data) + "\n")
print(f"Collected {len(tweets)} tweets")
Reddit (PRAW)
import praw, pandas as pd, os
reddit = praw.Reddit(
client_id = os.environ["REDDIT_CLIENT_ID"],
client_secret = os.environ["REDDIT_CLIENT_SECRET"],
user_agent = "Academic research — [Your Name], [Your University]"
)
subreddits = ["sociology", "urbanplanning", "firsttimehomebuyer"]
records = []
for sub_name in subreddits:
sub = reddit.subreddit(sub_name)
for post in sub.search("redlining", limit=500, time_filter="year"):
records.append({
"id": post.id, "subreddit": sub_name,
"title": post.title, "selftext": post.selftext,
"score": post.score, "num_comments": post.num_comments,
"created_utc": post.created_utc, "url": post.url
})
df = pd.DataFrame(records)
df.to_parquet("data/raw/reddit-posts.parquet", index=False)
Step 7: Large-Scale Text and News Sources
GDELT (global news events):
library(Rgdelt)
events <- gdelt_data(
start_date = "2020-01-01", end_date = "2023-12-31",
type = "events", country = "US"
)
Internet Archive Wayback Machine (longitudinal web):
import requests, pandas as pd
def wayback_cdx(url: str, from_date: str, to_date: str) -> pd.DataFrame:
"""Get all archived snapshots of a URL between two dates."""
cdx_url = (
f"http://web.archive.org/cdx/search/cdx?url={url}&output=json"
f"&from={from_date}&to={to_date}&fl=timestamp,statuscode,original"
f"&filter=statuscode:200&collapse=timestamp:6"
)
resp = requests.get(cdx_url, timeout=30)
rows = resp.json()[1:]
return pd.DataFrame(rows, columns=["timestamp", "statuscode", "url"])
snaps = wayback_cdx("example.com/policy-page", "20200101", "20231231")
for _, row in snaps.iterrows():
archive_url = f"https://web.archive.org/web/{row.timestamp}/{row.url}"
Step 8: Storage, Provenance, and Pipeline
Always record metadata alongside scraped content:
df_raw <- df_raw %>%
mutate(
scraped_at = Sys.time(),
source_url = "https://example.com",
scraper_version = "v1.0"
)
saveRDS(df_raw, paste0("data/raw/scraped-", Sys.Date(), ".rds"))
import duckdb, pandas as pd
con = duckdb.connect("data/raw/scraped.duckdb")
con.execute("""
CREATE TABLE IF NOT EXISTS articles (
id VARCHAR PRIMARY KEY,
url VARCHAR,
title VARCHAR,
date DATE,
