data-finder

name	data-finder
description	Find and assess datasets for a research question.

Data Finder

Find and assess datasets for your research question. Explorer-style passes search across data source categories; an Explorer-Critic pass then stress-tests each candidate against the research design.

Input: $ARGUMENTS — a topic, or from spec to read the research question from quality_reports/.

---

Step 1: Read Research Context

1. Find the most recent quality_reports/project_spec_*.md or quality_reports/specs/*.md — extract:

   • Research question
   • Empirical strategy (DiD, RDD, IV, etc.)
   • Treatment variable (what varies)
   • Outcome variable (what we measure)
   • Controls needed
   • Time period of interest
   • Geography (national, state, county, individual)
   • Unit of observation (individual, household, firm, establishment)

2. Read references/domain-profile.md if it exists — extract the Common Datasets section (domain-specific datasets to check first).

3. If no research spec exists, extract the variables and strategy from $ARGUMENTS directly. If the request is vague, ask: "What are the treatment and outcome variables, and what empirical strategy did you have in mind?"

---

Step 2: Split the Dataset Search into Two Explorer Passes

Read agents/explorer.md once before starting the Explorer search passes.

Split the source categories into two search passes. By default, run them inline. If the user explicitly requested delegated or parallel dataset search, you may use two explorer subagents.

Explorer A — Institutional Data:

Subagent prompt: "You are an Explorer agent. Research question: [question].
Empirical strategy: [strategy].
Variables needed — Treatment: [X], Outcome: [Y], Controls: [list],
Time period: [period], Geography: [geo], Unit: [unit].
Domain datasets (check first): [list from domain-profile if available].

Your source categories to search:
1. Public microdata (CPS, ACS, NHIS, MEPS, SIPP, QWI)
2. Administrative data (Medicare/Medicaid, IRS, SSA, vital statistics, court records)
3. Survey panels (PSID, HRS, Add Health, NLSY97/79, BHPS/UKHLS)

For each dataset found, produce the full Explorer report format.
Follow the Explorer agent instructions."

Explorer B — Broader and Alternative Sources:

Subagent prompt: "You are an Explorer agent. Research question: [question].
Empirical strategy: [strategy].
Variables needed — Treatment: [X], Outcome: [Y], Controls: [list],
Time period: [period], Geography: [geo], Unit: [unit].
Domain datasets (check first): [list from domain-profile if available].

Your source categories to search:
1. International data (World Bank, OECD, Eurostat, IMF, IPUMS International)
2. Novel/alternative (satellite, web scraping, proprietary, RCT registries)
3. Any field-specific datasets not covered by Explorer A

For each dataset found, produce the full Explorer report format.
Follow the Explorer agent instructions."

---

Step 3: Run the Explorer-Critic Pass

Read agents/explorer-critic.md before running the critique pass.

After both Explorer passes complete, run the Explorer-Critic on the full combined dataset list. If the user explicitly requested delegated review, you may use a separate critic subagent.

Subagent prompt: "You are an Explorer-Critic agent. Research question: [question].
Empirical strategy: [strategy].
Variables needed — Treatment: [X], Outcome: [Y], Controls: [list],
Time period: [period], Geography: [geo], Unit: [unit].

Here is the combined dataset list from the Explorer agents:
[paste all Explorer findings]

Apply the 5-point critique to each dataset:
1. Measurement validity
2. Sample selection
3. External validity
4. Identification compatibility
5. Known issues

Produce adjusted feasibility grades and deal-breaker flags.
Follow the Explorer-Critic agent instructions."

---

Step 4: Produce Ranked Output

After the Explorer-Critic pass completes, compile the final ranked report:

1. Sort datasets by adjusted feasibility grade (A first, then B, then C, then D).
2. Within each grade, sort by identification compatibility score (highest first).
3. Separate out deal-breaker datasets into the rejection table.

---

Step 5: Save Report

Create quality_reports/ if it is missing before saving.

Save to quality_reports/data_exploration_[sanitized_topic].md:

# Data Exploration: [Topic]

**Date:** [YYYY-MM-DD]
**Research question:** [one sentence]
**Empirical strategy:** [method]
**Variables sought:** Treatment = [X], Outcome = [Y], Controls = [list]

---

## Top Candidates (Grade A–B)

### 1. [Dataset Name] — Grade: A/B

**Provider:** [Name] | **Access:** [Public/Restricted/etc.] | **URL:** [link]

**Coverage:** [time period] | [geography] | [unit of observation] | N ≈ [size]

**Key Variables:**
- Treatment proxy: [variable]
- Outcome: [variable]
- Controls available: [list]

**Explorer-Critic Assessment:**
- Measurement validity: [1-2 sentences]
- Sample selection: [1-2 sentences]
- External validity: [1-2 sentences]
- Identification compatibility: [focused on the proposed strategy]
- Known issues: [specific documented problems]

**Bottom line:** [1-2 sentences — viable and under what conditions]

---

[Repeat for all A and B grade datasets]

---

## Accessible With Effort (Grade C)

[Brief summaries — name, access path, main limitation, why C not B]

---

## Rejection Table

| Dataset | Reason for Rejection | Deal-breaker? |
|---------|---------------------|---------------|
| [Name] | [Explorer-Critic finding] | YES/NO |

---

## Recommended Path Forward

1. **Best dataset:** [Name] — [one sentence why]
2. **Fallback if [best] unavailable:** [Name] — [why it's second choice]
3. **Access path for [best]:** [download URL, application URL, IRB requirements, restricted-data steps]

### Ingest Recipe for [best]

A copy-pasteable load-and-clean block for the recommended dataset. Tailor it to whether the source is a flat file, an API/portal, or a non-tabular source such as a PDF or scraped HTML table.

**R:**
```r
# Download (or login + download — note in a comment if manual)
library(tidyverse)
df <- readr::read_csv("data/raw/[file].csv")  # or haven::read_dta, arrow::read_parquet

df_clean <- df %>%
  rename(...) %>%
  mutate(...) %>%
  filter(...)

arrow::write_parquet(df_clean, "data/processed/[name].parquet")

Python:

import pandas as pd
df = pd.read_csv("data/raw/[file].csv")  # or pd.read_stata, pd.read_parquet
df_clean = (
    df.rename(columns={...})
      .assign(...)
      .query("...")
)
df_clean.to_parquet("data/processed/[name].parquet")

If the source is a PDF, scraped HTML page, or government portal API, replace the load step with the appropriate extraction recipe: tabulizer/pdftools/rvest/tidycensus/fredr in R, or pdfplumber/camelot/pandas.read_html/census/fredapi in Python. The data-analysis skill's Phase 0.5 documents the full set; cross-reference it if extraction is non-trivial.

Known traps for [best]

• [E.g., FIPS codes read as integers — drop leading zeros]
• [E.g., Survey weights required — srvyr::as_survey_design()]
• [E.g., Top-coding on income — note threshold]

---

Next Steps

• data-analysis [dataset] — begin analysis with the recommended dataset. Phase 0.5 handles non-tabular sources; Phase 3.5 runs design-specific identification diagnostics.
• lit-review [topic] — check if papers in the literature use these datasets (helps validate choice)

---

## Important

- **Identification compatibility is the most important criterion.** A perfectly accessible dataset that can't support the proposed empirical strategy is useless. The Explorer-Critic's grade on this dimension should drive the recommendation.
- **Access level affects timeline.** An FSRDC dataset may take 1-2 years to access. A public download can start today. Make this tradeoff explicit.
- **Don't reject C-grade datasets outright.** A FSRDC dataset with perfect identification fit may be the right choice for a dissertation. Present the access path clearly.