Discovers education data from Urban Institute Portal: endpoints, variables, year coverage, join keys (CCD, IPEDS, CRDC, Scorecard, SAIPE). Use to map questions to data. Load before education-data-query — discovery here, download there.
Discovers education data from Urban Institute Portal: endpoints, variables, year coverage, join keys (CCD, IPEDS, CRDC, Scorecard, SAIPE). Use to map questions to data. Load before education-data-query — discovery here, download there.
Discovers available education data from the Urban Institute Education Data Portal: endpoints, variables, year coverage, and join keys for schools, districts, and colleges (CCD, IPEDS, CRDC, Scorecard, SAIPE, and more). Use during discovery and scoping phases when identifying what data exists, mapping research questions to endpoints, or resolving variable name discrepancies between documentation and actual field names. Load before education-data-query — this skill covers discovery; education-data-query handles the download.
Discover available education data from the Urban Institute Education Data Portal for research planning and query design.
What is the Education Data Portal?
Comprehensive education data from Urban Institute - free and publicly available
Three data levels: schools, school-districts, college-university
Multiple data sources: CCD, IPEDS, CRDC, College Scorecard, EDFacts, SAIPE, FSA, MEPS, PSEO, etc.
Coverage: 1980-2023 depending on source
Access: Mirror downloads (parquet/CSV) via education-data-query skill
Curation layer, not a direct pass-through: The EDP standardizes data from original federal sources with lowercase variable names, integer-encoded categoricals, and uniform missing value codes (, , )
-1
-2
-3
Coverage varies by source: Some sources are fully mirrored; others are partially mirrored with only a subset of variables or datasets available. See individual education-data-source-* skills for source-specific coverage details
Skill Provenance Note: Each *-data-source-* skill includes
provenance.skill_last_updated in its frontmatter. When exploring data
sources during Stage 2, note the provenance dates of any skills you
reference — if more than a few months old, flag this in your findings
so the orchestrator can consider re-verifying with data-ingest.
Note: This workflow uses mirror-based file downloads, not paginated API calls. See education-data-query skill for fetch patterns and datasets-reference.md for file paths.
Reference File Structure
File
Purpose
When to Read
schools-endpoints.md
All school-level endpoints and variables
Researching K-12 schools
districts-endpoints.md
All district-level endpoints and variables
Researching school districts
colleges-endpoints.md
All college-level endpoints and variables
Researching higher education
variable-codes.md
Code values for states, grades, race, etc.
Interpreting or filtering data
metadata-api.md
Programmatic endpoint/variable discovery
Dynamic exploration
Decision Trees
What data level do I need?
What entity am I researching?
├─ Individual K-12 schools → schools level
│ └─ See ./references/schools-endpoints.md
├─ School districts / LEAs → school-districts level
│ └─ See ./references/districts-endpoints.md
├─ Colleges / Universities → college-university level
│ └─ See ./references/colleges-endpoints.md
└─ Not sure
├─ Need school-specific data (discipline, AP, demographics) → schools
├─ Need aggregate district data (finance, poverty) → school-districts
└─ Need postsecondary data (enrollment, aid, outcomes) → college-university
What topic am I researching?
Research topic?
├─ Enrollment / Demographics
│ ├─ K-12 public schools → CCD enrollment endpoints
│ ├─ Civil rights indicators → CRDC enrollment
│ └─ Colleges → IPEDS enrollment
├─ School Finance
│ ├─ District revenue/expenditure → CCD finance
│ └─ College finance → IPEDS finance
├─ Student Outcomes
│ ├─ K-12 assessments → EDFacts
│ ├─ Graduation rates (K-12) → EDFacts
│ ├─ College completion → IPEDS completions
│ └─ Post-college earnings → College Scorecard
├─ Student Aid / Loans
│ ├─ College financial aid → IPEDS aid
│ ├─ Federal loans/grants → FSA
│ └─ Debt/repayment → College Scorecard
├─ Discipline / Civil Rights
│ └─ K-12 discipline, harassment, restraint → CRDC
├─ Poverty Estimates
│ └─ District-level → SAIPE
└─ Directory / Location
├─ K-12 schools → CCD directory
├─ Districts → CCD directory
└─ Colleges → IPEDS directory
How do I find specific variables?
Finding variables?
├─ Know the endpoint → Check reference file for variable list
├─ Know the topic → Use topic index below
├─ Need to search programmatically → See ./references/metadata-api.md
└─ Need code definitions → See ./references/variable-codes.md
Quick Reference: Data Levels
Level
Key Sources
Primary ID
ID Format
schools
CCD, CRDC, EDFacts, MEPS, NHGIS
ncessch
12-char string
school-districts
CCD, SAIPE, EDFacts
leaid
7-char string
college-university
IPEDS, Scorecard, FSA, PSEO, EADA
unitid
6-digit integer
Quick Reference: Data Sources
Source
Level
Description
Years
CCD
Schools, Districts
Public K-12 directory, enrollment, finance
1986-2023
CRDC
Schools
Civil rights indicators, discipline, AP courses
2011-2021
EDFacts
Schools, Districts
Assessments, graduation rates
2009-2020
IPEDS
Colleges
Enrollment, completions, finance, institutional data
Next step: Load education-data-query skill to construct the actual API calls
Common Pitfalls
Year coverage varies: Always check years available for each endpoint
Different ID formats: ncessch (12-char), leaid (7-char), unitid (6-digit)
Disaggregation in URL: Grade, race, sex are often URL path components, not query params
Missing data codes: -1, -2, -3 have specific meanings (see variable-codes.md)
Pre-Query Validation
CRITICAL: Variable Name Discrepancies
The Education Data Portal API often uses different variable names than documentation suggests. Always fetch a sample first:
# Test query to verify actual column names
response = requests.get(
"https://educationdata.urban.org/api/v1/college-university/ipeds/directory/2023/"
)
data = response.json()
print("Actual columns:", list(data['results'][0].keys()))
Known discrepancies:
Documented
Actual API Field
Endpoint
inst_level
institution_level
IPEDS Directory
applicants_total
number_applied
IPEDS Admissions
admissions_total
number_admitted
IPEDS Admissions
grad_rate_150pct
completion_rate_150pct
IPEDS Graduation Rates
school_poverty
meps_poverty_pct
MEPS
population_5_17_poverty
est_population_5_17_poverty
SAIPE
See the relevant education-data-source-* skill for comprehensive variable mappings per source.
Metadata API Limitations
The metadata API has undocumented limitations:
?section=schoolsworks to filter by data level
?source=ipeds does NOT work - filter client-side instead
Response field names differ: source is actually class_name, source_name is actually label
Data Source Details
Quick summaries below. For comprehensive documentation including methodology, variable definitions, data quality issues, and historical changes, load the corresponding education-data-source-* skill.
CCD (Common Core of Data)
Coverage: All public elementary and secondary schools and districts in the U.S.
Topic
Schools
Districts
Directory
Yes
Yes
Enrollment
Yes (by grade, race, sex)
Yes (by grade, race, sex)
Finance
No
Yes (revenue, expenditure)
Key Variables:
ncessch: 12-character NCES school ID
leaid: 7-character NCES district ID
enrollment: Total enrollment count
free_or_reduced_price_lunch: FRPL-eligible students (poverty proxy)
Deep dive: Load education-data-source-ccd for survey components, data collection process, variable coding, state variations, and historical changes (e.g., 2006 locale code revision, 2010 race category changes).
CRDC (Civil Rights Data Collection)
Coverage: Biennial survey of public schools (2011, 2013, 2015, 2017, 2020, 2021)
Topics:
Discipline (suspensions, expulsions, arrests)
Chronic absenteeism
Harassment and bullying
Restraint and seclusion
Advanced courses (AP, IB, gifted)
Course offerings
Teacher qualifications
Retention
COVID impacts (2020 only)
Key Feature: Disaggregation by race, sex, disability, and LEP status
Deep dive: Load education-data-source-crdc for civil rights legal context (Title VI, IX, Section 504), collection methodology, underreporting issues, and year-to-year changes.
EDFacts
Coverage: State assessment and accountability data
Topics:
Assessment proficiency rates (reading, math)
Graduation rates (4-year adjusted cohort)
Key Feature: Data available by special populations (disability, economically disadvantaged, LEP, homeless, migrant, foster care)
CRITICAL: State assessment scores CANNOT be compared across states (different tests, cut scores).
Deep dive: Load education-data-source-edfacts for ESSA/NCLB accountability context, why cross-state comparison is invalid, ACGR methodology, and subgroup reporting rules.
IPEDS (Integrated Postsecondary Education Data System)
Coverage: All Title IV-eligible postsecondary institutions
institution_level: 1=Less-than-2-year, 2=2-year, 4=4-year (no code 3)
hbcu: Historically Black college indicator
Deep dive: Load education-data-source-ipeds for critical graduation rate limitations (first-time full-time only), GASB vs FASB finance accounting, survey components, and identifier changes.
College Scorecard
Coverage: Title IV institutions with outcome data
Topics:
Post-college earnings (6 and 10 years after entry)
Student debt and repayment
Default rates
Completion rates by income level
Key Feature: Links education to labor market outcomes
CRITICAL: Only covers Title IV aid recipients (selection bias toward lower-income students).
Deep dive: Load education-data-source-scorecard for earnings methodology (IRS data), population coverage limitations, suppression rules, and field-of-study data.
SAIPE (Small Area Income and Poverty Estimates)
Coverage: Census Bureau poverty estimates for school districts
Key Variables:
population_5_17_poverty: Children 5-17 in poverty
population_5_17_poverty_pct: Percent in poverty
median_household_income: District median income
Deep dive: Load education-data-source-saipe for model-based estimation methodology, confidence intervals (not available at district level), and comparison to other poverty measures.
FSA (Federal Student Aid)
Coverage: Title IV institutions receiving federal aid
Topics:
Pell grants
Direct loans (subsidized, unsubsidized, PLUS)
Campus-based aid (Perkins, work-study)
Financial responsibility scores
90/10 revenue (for-profit institutions)
Deep dive: Load education-data-source-fsa for Title IV program details, financial responsibility composite scores, and 90/10 rule compliance.
Additional Data Sources
Source
Coverage
Deep Dive Skill
MEPS
School-level poverty estimates (superior to FRPL for cross-state comparison)
education-data-source-meps
NHGIS
Census geography crosswalks for schools
education-data-source-nhgis
NACUBO
College endowment data
education-data-source-nacubo
NCCS
Nonprofit data for private colleges (Form 990)
education-data-source-nccs
EADA
College athletics equity data
education-data-source-eada
Campus Safety
Campus crime statistics (Clery Act)
education-data-source-campus-safety
PSEO
Post-graduation employment outcomes
education-data-source-pseo
Joining Data Across Sources
School-Level Joins
Join school data across sources using ncessch:
Source 1
Source 2
Join Key
Use Case
CCD
CRDC
ncessch
Enrollment + discipline
CCD
EDFacts
ncessch
Directory + assessments
CCD
MEPS
ncessch
Enrollment + poverty
CRDC
MEPS
ncessch
AP courses + poverty
Note: Match on year when joining (years may not align perfectly)
District-Level Joins
Join district data using leaid:
Source 1
Source 2
Join Key
Use Case
CCD Directory
CCD Finance
leaid
Characteristics + spending
CCD
SAIPE
leaid
Enrollment + poverty
CCD
EDFacts
leaid
Enrollment + outcomes
College-Level Joins
Join college data using unitid:
Source 1
Source 2
Join Key
Use Case
IPEDS Directory
IPEDS Finance
unitid
Characteristics + finance
IPEDS
Scorecard
unitid
Enrollment + earnings
IPEDS
FSA
unitid
Enrollment + aid data
Disaggregation Patterns
URL Path Disaggregation
Some disaggregations are part of the URL path:
/schools/ccd/enrollment/{year}/{grade}/ # By grade
/schools/ccd/enrollment/{year}/{grade}/race/ # By grade and race
/schools/ccd/enrollment/{year}/{grade}/race/sex/ # By grade, race, and sex
Query Parameter Disaggregation
Other filters are query parameters:
?fips=6 # California only
?charter=1 # Charter schools only
?school_level=3 # High schools only
?urban_centric_locale=11 # Large cities only
