| name | education-data-explorer |
| description | Mirror-first discovery for Urban Institute Education Data: maps a research question to the mirror dataset FILES that answer it — canonical path, entity grain, year coverage, key variables, and join keys — for CCD, IPEDS, CRDC, College Scorecard, SAIPE, EDFacts, FSA, MEPS, NHGIS, PSEO, NACUBO, NCCS, EADA, and Campus Safety. Discovery output is the mirror path (plus grain, years, variables, caveats), the currency planning and acquisition consume. A pure readable-reference skill for read-only discovery agents — no code, no live HTTP probes; all facts keyed to pinned mirror vintage 0.26.1. Load BEFORE education-data-query — routing (which file, what grain) here, retrieval (fetch code, mirror config) there. |
| metadata | {"audience":"research-planner","domain":"data-access"} |
Education Data Explorer
Discovers which Urban Institute Education Data mirror files answer a research question and returns them as acquisition-ready routing facts: the canonical mirror path, what one row represents (grain), year coverage, curated key variables, and join keys. This is the discovery half of the education-data workflow — it owns question→file routing judgment, grain semantics, variable/code meaning, and cross-source joins. It does not own fetch mechanics: retrieval code, mirror configuration, codebook-URL construction, and the curated path catalog live in education-data-query. Load this skill first (routing), then education-data-query (retrieval). Every inventory fact here — years, variables, coded values — is keyed to mirror vintage 0.26.1 (pinned); the discovery path is pure markdown reading, with no instruction to fetch, probe, or execute anything.
Why mirror-first (and why no API)
DAAF's education pipeline never touches Urban's live REST API. All acquisition is mirror-based: a revision-pinned HuggingFace parquet mirror (Portal vintage 0.26.1, revision 0ad00ce0e232c96b0642459e4e7326607a8d26aa), with an unpinned urban_csv bulk-CSV fallback that uses the identical canonical paths. Acquisition consumes canonical mirror paths of the form {source}/{filename} (extensionless — the mirror layer appends .parquet or .csv). So discovery's job is to emit those paths, not URL routes. Portal endpoint routes appear here only as small citation/maintenance annotations, never as the thing you hand downstream.
The live REST API moves quarterly (endpoints rename, variables drift); a prior version of this skill hand-documented that API and went badly stale. The mirror is a frozen, reproducible snapshot — so the facts below are stable for vintage 0.26.1 and can be read as reference rather than re-probed.
Standing caveat (vintage + fallback). All year coverage, variables, and coded values below are exact for the pinned HuggingFace mirror at vintage 0.26.1. The urban_csv fallback is unpinned/current-Portal, so if a fetch falls through to it, vintage guarantees no longer hold exactly. Cite Portal version 0.26.1 for data fetched from the pinned mirror.
Reference File Structure
| File | Purpose | When to Read |
|---|
references/schools-datasets.md | K-12 school-level mirror families (CCD, CRDC, EDFacts, MEPS, NHGIS) | Researching individual schools |
references/districts-datasets.md | District/LEA-level mirror families (CCD, SAIPE, EDFacts) | Researching school districts |
references/colleges-datasets.md | College-level mirror families (IPEDS, Scorecard, FSA, PSEO, EADA, NACUBO, NCCS, NHGIS, Campus Safety) | Researching higher education |
references/variable-dictionary-{source}.md | Complete per-source variable inventory (2,235 per-source variables, deduplicated from 2,994 endpoint-variable rows; verbatim labels/types/filter flags/coded values) | Confirming a real variable name or its coded values |
references/variable-codes.md | Curated, analytically load-bearing coded-value schemes with cross-source populated-subset notes | Interpreting or filtering categorical codes |
references/maintenance-live-api.md | Live-API catalog surfaces + regeneration mechanics — maintenance sessions only | Rebuilding the mirror/dictionaries at a new vintage |
The 14 variable-dictionary-{source}.md files are mechanically generated from the Portal varlist at vintage 0.26.1. The routing files curate 5-12 key variables per family and point to the full dictionary entry for the rest. Never invent a variable name — if it is not in the dictionary, it is not in the mirror.
Decision Trees
What data level (and file) do I need?
What entity is one row of my analysis?
├─ An individual K-12 school (ncessch) → schools level
│ └─ references/schools-datasets.md → pick a CCD/CRDC/EDFacts/MEPS/NHGIS family
├─ A school district / LEA (leaid) → school-districts level
│ └─ references/districts-datasets.md → pick a CCD/SAIPE/EDFacts family
└─ A college / university (unitid) → college-university level
└─ references/colleges-datasets.md → pick an IPEDS/Scorecard/FSA/PSEO/EADA/NACUBO/NCCS/csafety family
What topic → which mirror family?
Research topic?
├─ K-12 enrollment / demographics → ccd/schools_ccd_enrollment_{year} (or ccd/schools_ccd_lea_enrollment_{year} at district level)
├─ School/district directory & staffing → ccd/schools_ccd_directory | ccd/school-districts_lea_directory
├─ District finance → ccd/districts_ccd_finance
├─ District poverty → saipe/districts_saipe
├─ Civil rights / discipline / course access → crdc/schools_crdc_* (see schools-datasets.md — one file per topic)
├─ K-12 assessments / grad rates → edfacts/{schools,districts}_edfacts_assessments_{year} | _grad_rates_{year}
├─ School poverty measure → meps/schools_meps
├─ College directory / characteristics → ipeds/colleges_ipeds_directory | _institutional-characteristics
├─ College enrollment → ipeds/colleges_ipeds_fall-enrollment-race_{year} | _enrollment-fte | _headcount
├─ College completions (by CIP) → ipeds/colleges_ipeds_completions-2digcip_{year} | -6digcip_{year}
├─ College graduation rates → ipeds/colleges_ipeds_grad-rates | -200pct | -pell
├─ College finance / endowment → ipeds/colleges_ipeds_finance | nacubo/colleges_nacubo_endow | nccs/colleges_nccs_all
├─ Student aid / loans → ipeds/colleges_ipeds_sfa_* | fsa/colleges_fsa_grants | _loans | _campus_based_volume
├─ Post-college earnings → scorecard/colleges_scorecard_earnings | pseo/colleges_pseo_{year}
├─ Athletics equity → eada/colleges_eada_inst_characteristics
├─ Campus crime (hate crimes) → csafety/colleges_csafety_hate_crimes
└─ Census geography crosswalk → nhgis/{schools,colleges}_nhgis_geog_{census-year}
How do I confirm a variable name or code?
Need a variable?
├─ Curated key variables → the family entry in the *-datasets.md routing file
├─ Every variable in a family → references/variable-dictionary-{source}.md (§ that family)
└─ What a categorical code means → references/variable-codes.md
Quick Reference: Data Levels
| Level | Key Sources | Primary ID | ID Format |
|---|
| schools | CCD, CRDC, EDFacts, MEPS, NHGIS | ncessch | 12-char zero-padded string |
| school-districts | CCD, SAIPE, EDFacts | leaid | 7-char zero-padded string |
| college-university | IPEDS, Scorecard, FSA, PSEO, EADA, NACUBO, NCCS, NHGIS, Campus Safety | unitid | 6-digit integer |
Zero-padded IDs: ncessch (12) and leaid (7) are zero-padded numeric strings — leading zeros are load-bearing (FIPS 01-09 states). education-data-query handles the force-string/pad-and-assert read recipe; discovery just needs to name the join key.
Quick Reference: Join Keys
| Join across | Key | Example use |
|---|
| School sources (CCD ↔ CRDC ↔ EDFacts ↔ MEPS ↔ NHGIS) | ncessch (+ year) | Enrollment + discipline + poverty |
| District sources (CCD ↔ SAIPE ↔ EDFacts) | leaid (+ year) | Directory + poverty + outcomes |
| College sources (IPEDS ↔ Scorecard ↔ FSA ↔ PSEO ↔ EADA ↔ NACUBO ↔ NCCS ↔ csafety) | unitid (+ year) | Characteristics + earnings + aid |
| School → district rollup | leaid present on school files | Aggregate schools to their LEA |
| Any level → census geography | ncessch/unitid via NHGIS crosswalk | Link to tract/CBSA/region |
Match on year when joining — coverage windows differ across families (see below), so align years before merging.
Year Coverage Overview (vintage 0.26.1)
Verified ranges from the endpoint-file bridge and route reconciliation. Non-contiguous families are flagged; see the routing files for exact year lists.
| Family (representative) | Coverage | Note |
|---|
| CCD directory / enrollment | 1986-2024 | Annual |
| CCD district finance | 1991, 1994-2020 | Non-contiguous early years |
| CRDC (most topics) | 2011, 2013, 2015, 2017, 2020, 2021 | Biennial; chronic-absenteeism reaches 2022 |
| EDFacts assessments | 2009-2018, 2020 | 2019 absent (COVID testing waivers) |
| EDFacts grad rates | 2010-2019 | — |
| IPEDS directory / inst-characteristics | 1980, 1984-2024 | — |
| IPEDS finance | 1979, 1983-2017 | Mirror ends 2017 (API reaches further back, differently packaged) |
| IPEDS grad-rates (150%) | 1996-2023 | Wholesale-revalued in v0.26.1 — re-run any pre-2026q3 analysis |
| IPEDS completions (2-/6-dig CIP) | 1991-2023 / 1983-2023 | Yearly shards |
| Scorecard earnings | 2003-2014, 2018 | Sparse; other Scorecard families 1996-2020 |
| SAIPE | 1995, 1997, 1999-2024 | Poverty pct on a 0-1 scale despite _pct |
| FSA | 1999-2021 | 90/10 from 2014; campus-based from 2001 |
| MEPS | 2009-2022 | 2023 empty on 2026-08-06 probe |
| NHGIS (schools / colleges) | 1986-2023 / 1980-2023 | Crosswalk file-years; census vintages 1990-2020 |
| PSEO | 2001-2021 | Yearly shards |
| NACUBO / NCCS / EADA / Campus Safety | 2012-2022 / 1993-2016 / 2002-2021 / 2005-2021 | — |
Grain Logic: What a Row Represents
A mirror file's grain is the set of columns that uniquely identify a row. Reading it correctly is the difference between a clean join and a silent fan-out.
- Base grain is the entity ID +
year (e.g., one row per ncessch × year in CCD directory).
- Disaggregation dimensions multiply rows. In the live API these appear as URL path segments (
.../enrollment/{year}/{grade}/race/sex/); in the mirror file the same dimensions are columns (grade, race, sex) and each combination is its own row. So CCD enrollment is one row per ncessch × year × grade × race × sex. To get a school total, filter to the 99/total code on each dimension rather than summing (summing double-counts the totals rows).
- CRDC files carry
disability and lep dimensions in addition to race/sex; a discipline file is one row per school × year × the disaggregation the file name advertises.
- Yearly vs Single files. "Yearly" families have one file per year with
{year} in the path (fetch each year, concatenate). "Single" families pack all years into one file (filter on year locally). The routing files mark each family Single or Yearly.
- Coded totals. Categorical dimensions use
99 = Total and negative sentinels for missing (-1/-2/-3/-99) — see variable-codes.md. A "total" row and its component rows coexist in the same file.
What Discovery Should Emit (the acquisition currency)
Discovery output feeds planning, then acquisition. For each dataset a research question needs, emit:
- Canonical mirror path(s) —
{source}/{filename}, extensionless, with {year} placeholder where the family is Yearly. This is the primary currency.
- Grain — what one row represents, including disaggregation dimensions.
- Years — the specific years needed, within the family's coverage window.
- Key variables — the real column names (from the routing file / dictionary) the analysis will use.
- Join keys — the ID(s) linking this file to the others in the plan.
- Caveats — load-bearing warnings (suppression, revaluation, scale, coverage gaps).
Portal endpoint routes are optional citation annotations only — never the hand-off artifact. Do not instruct downstream to probe, fetch, or call an API; naming the path and years is sufficient for education-data-query to construct the fetch.
Cross-Reference to Related Skills
| Skill | Purpose | When |
|---|
education-data-query | Fetch code, mirror config, codebook URLs, path catalog | After identifying paths/years/variables here |
education-data-context | Portal-wide interpretation (missing codes, grade encoding, licensing, joins) | Before analyzing retrieved data |
education-data-source-{ccd,ipeds,crdc,scorecard,saipe,edfacts,fsa,meps,nhgis,nccs,nacubo,eada,campus-safety,pseo} | Deep source context: methodology, limitations, historical changes | When you need collection methodology or source-specific caveats beyond routing |
Load a education-data-source-* skill when a question needs collection methodology, known limitations, or historical coding changes for a specific source. Each source skill carries a skill-last-updated provenance date — flag stale ones to the orchestrator.
Worked Example: Planning a Discovery Output
Question: "Relationship between school poverty and AP course offerings in California high schools?"
- Level / grain: schools — one row per
ncessch × year.
- Files:
crdc/schools_crdc_apib_enroll (Single; 2011-2021) — AP/IB enrollment. Key vars: enrl_ap, enrl_ap_math, enrl_ap_science, enrl_ib, enrl_gifted_talented. Grain adds race/sex/disability/lep columns — filter to totals for a school-level measure.meps/schools_meps (Single; 2009-2022) — modeled poverty. Key var: meps_poverty_pct.
- Join:
ncessch + year; overlap window 2011-2021.
- Filter:
fips == 6 (California), high schools via CCD school_level == 3 (join ccd/schools_ccd_directory).
- Emit: the three paths, the year window, the key variables, the
ncessch join key, and the CRDC disaggregation-totals caveat. Then load education-data-query to fetch.
Common Pitfalls
- Emitting a route instead of a path. Downstream needs
crdc/schools_crdc_discipline_k12_{year}, not /schools/crdc/discipline/{year}/....
- Summing disaggregated rows. Total rows (
99) coexist with component rows — filter, don't sum.
- Assuming a variable exists. If it is not in
variable-dictionary-{source}.md, it is not in the mirror.
- Ignoring the coverage window. Align
year across families before joining; several have gaps (EDFacts 2019, IPEDS finance ends 2017).
- Trusting fallback vintage. Facts are exact for the pinned mirror only; the
urban_csv fallback is current-Portal.
