| name | education-data-context |
| description | Interpretation guidance for Urban Institute Education Data Portal datasets. The Portal is a curation layer over federal data: lowercase variable names, integer-encoded categoricals, standardized missing codes (-1 missing, -2 not applicable, -3 suppressed). Covers year definitions, grade encoding (grade=-1 is Pre-K, not missing), suppression rates, ODC-By licensing, and cross-source join identifiers. Load before analyzing any Portal data. Routes to source-specific deep-dive skills for individual datasets. |
| metadata | {"audience":"any-agent","domain":"data-documentation"} |
Education Data Context
Data origin, caveats, and interpretation guidance for Urban Institute Education Data Portal datasets. Use when interpreting Portal coded values (-1/-2/-3 missing/not-applicable/suppressed), understanding year definitions (fall vs. academic year), applying correct grade encoding (grade=-1 means Pre-K, not missing), assessing suppression rates, citing data under ODC-By license, or reviewing any Portal data before analysis. Also covers joining identifiers across CCD, IPEDS, CRDC, and other sources, and routes to source-specific deep-dive skills.
This skill provides critical context for interpreting data from the Urban Institute Education Data Portal. Education data has source-specific limitations that can significantly affect analysis validity.
Why Data Context Matters
- Source-specific limitations: Each data source (CCD, IPEDS, CRDC, etc.) has unique constraints
- Missing values have meaning: Codes like -1, -2, -3 indicate specific conditions, not random missingness
- Definitions change over time: Variable definitions, categories, and coding schemes evolve
- State comparisons require caution: State-level data often cannot be directly compared
- Citation is required: The ODC Attribution License mandates proper citation
- Skill provenance matters: Each
*-data-source-* skill includes provenance.skill_last_updated in its frontmatter. If this date is more than a few months old, treat the skill's claims about coded values, suppression patterns, and data quality with caution ā data sources evolve and skill documentation may have drifted. Consider re-running data-ingest to re-verify.
Data Provenance: The Education Data Portal
All education data currently accessible through this system is obtained from the Urban Institute Education Data Portal (EDP), not directly from original source agencies (NCES, Census Bureau, Department of Education, etc.). The EDP is a curation and standardization layer that:
- Renames variables to lowercase (e.g.,
enrollment not MEMBER)
- Re-encodes categoricals as integers (e.g.,
1 not "Regular school")
- Standardizes missing values using codes
-1 (missing), -2 (not applicable), -3 (suppressed)
- May subset each source's full variable catalog ā not all variables from the original source are necessarily available through the Portal
Each education-data-source-* skill documents what is available through the Portal for that source, including any known gaps relative to the original data collection. When a skill also documents variables or components only available from the original source directly, this is clearly noted.
Note: This provenance applies specifically to the current education data source skills. Future data source skills may access data from other providers with different characteristics.
Reference File Structure
Quick Context (This Skill)
| File | Content | When to Read |
|---|
./references/ccd-context.md | K-12 schools/districts caveats | After pulling CCD data |
./references/ipeds-context.md | College/university caveats | After pulling IPEDS data |
./references/crdc-context.md | Civil rights data caveats | After pulling CRDC data |
./references/scorecard-context.md | College Scorecard caveats | After pulling Scorecard data |
./references/edfacts-context.md | Assessment/graduation caveats | After pulling EDFacts data |
./references/data-relationships.md | Joining tables, identifiers | When merging datasets |
Deep-Dive Source Skills (Comprehensive Documentation)
These skills document both EDP-available data and original source context. Each skill notes when content applies only to the original source (not available through the Portal).
For comprehensive understanding beyond the quick context files above, load the dedicated data source skill:
| Data Source | Deep-Dive Skill | Key Deep Topics |
|---|
| CCD | education-data-source-ccd | Survey components, EDFacts submission, state variations, historical changes |
| CRDC | education-data-source-crdc | Civil rights legal context, underreporting issues, year-to-year evolution |
| EDFacts | education-data-source-edfacts | ESSA/NCLB context, why states aren't comparable, ACGR methodology |
| IPEDS | education-data-source-ipeds | All 12+ surveys, graduation rate population limits, GASB vs FASB |
| Scorecard | education-data-source-scorecard | IRS earnings methodology, Title IV selection bias, suppression rules |
| SAIPE | education-data-source-saipe | Model-based estimation, no district confidence intervals |
| FSA | education-data-source-fsa | Title IV programs, financial responsibility scores, 90/10 rule |
| MEPS | education-data-source-meps | Superior to FRPL for cross-state poverty comparison |
| NHGIS | education-data-source-nhgis | Census geography links, boundary changes over time |
| NACUBO | education-data-source-nacubo | Endowment study methodology, voluntary participation bias |
| NCCS | education-data-source-nccs | Form 990 data, NTEE codes, private college relevance |
| EADA | education-data-source-eada | Title IX context, not same as compliance data |
| Campus Safety | education-data-source-campus-safety | Clery Act, underreporting, geography definitions |
| PSEO | education-data-source-pseo | LEHD methodology, experimental status, state coverage |
When to load deep-dive skills:
- Need to understand data collection methodology in detail
- Analyzing historical trends and need to know about definition changes
- Encountering data quality issues that require deeper investigation
- Writing documentation or reports that require precise methodology descriptions
Decision Trees
What data source did I pull from?
What endpoint did you use?
āā schools/ccd/* ā Read ./references/ccd-context.md
ā āā Need more depth? ā Load education-data-source-ccd skill
āā school-districts/* ā Read ./references/ccd-context.md
ā āā Need more depth? ā Load education-data-source-ccd skill
āā schools/crdc/* ā Read ./references/crdc-context.md
ā āā Need more depth? ā Load education-data-source-crdc skill
āā schools/edfacts/* ā Read ./references/edfacts-context.md
ā āā Need more depth? ā Load education-data-source-edfacts skill
āā schools/meps/* ā Load education-data-source-meps skill
āā college-university/ipeds/* ā Read ./references/ipeds-context.md
ā āā Need more depth? ā Load education-data-source-ipeds skill
āā college-university/scorecard/* ā Read ./references/scorecard-context.md
ā āā Need more depth? ā Load education-data-source-scorecard skill
āā college-university/fsa/* ā Load education-data-source-fsa skill
āā college-university/nacubo/* ā Load education-data-source-nacubo skill
āā college-university/eada/* ā Load education-data-source-eada skill
āā college-university/pseo/* ā Load education-data-source-pseo skill
āā school-districts/saipe/* ā Load education-data-source-saipe skill
āā Multiple sources ā Read ./references/data-relationships.md first
How do I interpret missing values?
What value do you see?
āā In a CATEGORICAL column (grade, race, sex)?
ā āā These use integer encoding, NOT coded missing values!
ā āā grade = -1 means Pre-K (NOT missing!)
ā āā race = 1-7 (NOT WH, BL, HI strings)
ā āā sex = 1-2 (NOT M, F strings)
āā In a NUMERIC column (enrollment, FTE, counts)?
ā āā -1 ā Missing/not reported (treat as NULL)
ā āā -2 ā Not applicable (exclude from that variable's analysis)
ā āā -3 ā Suppressed for privacy (cannot recover)
āā null/blank?
ā āā Source matters:
ā āā CCD, CRDC, EDFacts ā Should use -1/-2/-3 codes
ā āā Scorecard, MEPS, NACUBO ā Use native nulls
āā Ranges (e.g., "10-20") ā EDFacts suppression bounds
āā Unsure ā Check source-specific reference file
What are the limitations?
What type of analysis are you doing?
āā Cross-state comparison
ā āā K-12 assessments ā INVALID (states not comparable)
ā āā K-12 other metrics ā Check state reporting consistency
ā āā College data ā Generally valid (federal definitions)
āā Time series
ā āā Check for definition changes
ā āā Check for ID changes (schools/districts merge/split)
ā āā Check COVID-19 impact (2020-2021)
āā Subgroup analysis
ā āā Check suppression rates
ā āā Smaller groups = more suppression
ā āā Cannot impute suppressed values accurately
āā Graduate outcomes
āā IPEDS ā First-time full-time only
āā Scorecard ā Title IV recipients only
Universal Data Caveats
Portal Integer Encoding System
CRITICAL: The Education Data Portal uses integer codes, not string labels, for categorical variables. This applies to all sources.
Demographic Variable Encodings
| Variable | Integer Values | NOT Strings |
|---|
| Race | 1-7, 99 (total) | Not WH, BL, HI, AS, etc. |
| Sex | 1 (Male), 2 (Female), 3 (Another gender, IPEDS 2022+), 4 (Unknown gender, IPEDS 2022+), 9 (Unknown), 99 (Total) | Not M, F |
| Grade | -1 to 13, 99 (total) | Not PK, KG, 01, etc. |
Race codes:
| Value | Meaning |
|---|
| 1 | White |
| 2 | Black |
| 3 | Hispanic |
| 4 | Asian |
| 5 | American Indian/Alaska Native |
| 6 | Native Hawaiian/Pacific Islander |
| 7 | Two or more races |
| 8 | Nonresident alien (postsecondary only) |
| 9 | Unknown |
| 99 | Total (all races) |
Grade codes:
| Value | Meaning |
|---|
| -1 | Pre-K (SEMANTIC TRAP: NOT missing data!) |
| 0 | Kindergarten |
| 1-12 | Grades 1-12 |
| 13 | Ungraded |
| 99 | Total (all grades) |
SEMANTIC TRAP - Grade -1:
In CCD enrollment data, grade = -1 means Pre-Kindergarten, NOT missing data. This is a common source of errors. Missing data in enrollment uses the separate coded value system (-1/-2/-3) only for numeric fields like enrollment counts, not for the grade categorical variable.
df = df.filter(pl.col("grade") >= 0)
pre_k = df.filter(pl.col("grade") == -1)
k_12 = df.filter(pl.col("grade").is_between(0, 12))
total = df.filter(pl.col("grade") == 99)
Variable Names Are Lowercase
Portal variable names are lowercase, not the uppercase names from original NCES documentation:
enrollment not MEMBER or ENROLLMENTgrade not GRADEfips not FIPS or STATE
Rate and Proportion Normalization
The Portal normalizes certain rate and proportion variables to a 0-1 scale, while the original IPEDS surveys report them as 0-100 percentages. This is a Portal transformation, not an IPEDS source issue.
Known affected variables:
| Variable | Source Survey | Portal Scale | Original IPEDS Scale |
|---|
completion_rate_150pct | GRS (Graduation Rates) | 0-1 | 0-100 |
retention_rate | EF (Fall Enrollment / Retention) | 0-1 | 0-100 |
Guidance:
- Always check the actual range of rate variables after fetching -- if
max <= 1.0, the variable is on a 0-1 scale and may need rescaling to 0-100 for interpretability
- Do not assume all rate variables across all datasets are normalized -- this finding is specific to the IPEDS variables listed above
- Quality checks testing
value > 100 will not catch invalid data on 0-1 scaled variables; adjust thresholds accordingly (e.g., test value > 1.0 instead)
Missing Value Codes
| Code | Meaning | How to Handle |
|---|
| -1 | Missing/not reported | Treat as NULL; document missingness rate |
| -2 | Not applicable | Exclude from analysis of that variable |
| -3 | Suppressed (privacy) | Cannot be recovered; affects small-cell analyses |
| null/blank | Genuinely missing | Treat as NULL |
IMPORTANT: Coded values (-1/-2/-3) apply to numeric measure columns (enrollment counts, FTE, etc.), NOT to categorical identifier columns like grade, race, or sex. Those use the integer encoding system above.
Missing Data Handling Varies by Source:
| Source | Missing Data Pattern |
|---|
| CCD, CRDC, EDFacts | Use -1/-2/-3 coded values for numeric fields |
| Scorecard, MEPS, NACUBO | Use native null values |
| IPEDS | Mix of both (check specific variables) |
Important: Filter coded values BEFORE calculating statistics:
df["enrollment"].mean()
df.filter(pl.col("enrollment") >= 0)["enrollment"].mean()
Year Definitions
year refers to the FALL of the academic yearyear=2020 means the 2020-21 school year- Graduation rates use cohort entry year (cohort started 4-6 years prior)
- Finance data may use fiscal year (varies by institution)
| Data Type | Year Interpretation |
|---|
| Fall enrollment | Fall of indicated year |
| Academic year totals | Full year starting fall of indicated year |
| Graduation rates | Cohort entry year (outcomes measured later) |
| Completions | Degrees awarded during indicated academic year |
Suppression
Data is suppressed to protect student privacy:
- Small cell sizes: Typically fewer than 5-10 students
- Affects disaggregated data: Race, disability, gender breakdowns
- More suppression in smaller schools: Rural areas most affected
- Cannot be imputed accurately: Do not attempt to recover
- Complementary suppression: Other cells may be suppressed to prevent calculation
State Reporting Variation
State education agencies interpret federal definitions differently:
- Dropout definitions vary (CCD covers grades 7-12, CPS covers 10-12)
- Average daily attendance calculated differently by state law
- Discipline categories interpreted inconsistently
- Missing data tends to cluster by state
Data Quality Checklist
Before analyzing any Education Data Portal data:
Quick Coverage Check
df.group_by("fips").agg([
pl.col("variable").filter(pl.col("variable") == -1).count().alias("missing"),
pl.col("variable").filter(pl.col("variable") == -3).count().alias("suppressed"),
pl.col("variable").count().alias("total")
])
Citation Requirements
Full Citation Format
Use for publications, reports, and formal documents:
[Dataset name(s)], Education Data Portal (Version X.X.X),
Urban Institute, accessed [Month DD, YYYY],
https://educationdata.urban.org/documentation/,
made available under the ODC Attribution License.
Example:
Common Core of Data (CCD) School Directory, Education Data Portal
(Version 0.20.0), Urban Institute, accessed January 15, 2026,
https://educationdata.urban.org/documentation/,
made available under the ODC Attribution License.
Short Citation Format
Use for visualizations, dashboards, and space-constrained contexts:
Source: [Dataset name(s)], Education Data Portal v.X.X.X,
Urban Institute, ODC-By License.
Example:
Source: CCD School Directory, Education Data Portal v.0.20.0,
Urban Institute, ODC-By License.
License Terms
License: Open Data Commons Attribution License (ODC-By) v1.0
Key requirements:
- Must attribute the Urban Institute as data source
- Must indicate if data was modified
- May use for any purpose including commercial
- May redistribute with attribution
Notification
Email educationdata@urban.org with any published work using the data. This helps the Urban Institute track usage and improve the portal.
Quick Reference: Source-Specific Caveats
| Source | Key Limitation | Critical For | Quick Reference | Deep Dive |
|---|
| CCD | Public schools only; state reporting varies | K-12 enrollment, demographics | ./references/ccd-context.md | education-data-source-ccd |
| IPEDS | First-time full-time students only for grad rates | College graduation analysis | ./references/ipeds-context.md | education-data-source-ipeds |
| CRDC | Biennial; self-reported; underreporting | Equity/discipline analysis | ./references/crdc-context.md | education-data-source-crdc |
| Scorecard | Title IV recipients only; earnings suppressed | Earnings/outcomes analysis | ./references/scorecard-context.md | education-data-source-scorecard |
| EDFacts | State assessments NOT comparable across states | Achievement analysis | ./references/edfacts-context.md | education-data-source-edfacts |
| SAIPE | Model-based estimates; no district CIs | District poverty | ā | education-data-source-saipe |
| FSA | Federal aid only; timing varies | Student aid analysis | ā | education-data-source-fsa |
| MEPS | Model estimates; 100% FPL only | School poverty (cross-state) | ā | education-data-source-meps |
| NHGIS | Boundary changes over time | Geography linking | ā | education-data-source-nhgis |
| EADA | Self-reported; NOT Title IX compliance | Athletics equity | ā | education-data-source-eada |
| Campus Safety | Underreporting; comparability issues | Campus crime | ā | education-data-source-campus-safety |
| PSEO | Experimental; partial state coverage | Employment outcomes | ā | education-data-source-pseo |
What Each Source Covers
| Source | Universe | Update Frequency |
|---|
| CCD | All public schools and districts | Annual |
| IPEDS | All Title IV postsecondary institutions | Annual |
| CRDC | Sample/universe of public schools | Biennial |
| Scorecard | Title IV aid recipients | Annual |
| EDFacts | Public schools with state assessments | Annual |
Data Lag Reference
Data availability lags behind the current year. As of January 2026:
| Source | Survey Component | Typical Lag | Latest Available |
|---|
| IPEDS | Directory | ~1 year | 2023 |
| IPEDS | Admissions-Enrollment | ~2 years | 2022 |
| IPEDS | Fall Enrollment | ~2-3 years | 2021 |
| IPEDS | Finance | ~2-3 years | Varies |
| CCD | Directory/Enrollment | ~1-2 years | 2022 |
| CCD | Finance | ~2-3 years | 2020 |
| CRDC | All (biennial) | ~1-2 years | 2021 |
| EDFacts | Assessments | ~1-2 years | 2020 |
| EDFacts | Graduation Rates | ~1-2 years | 2020 |
| SAIPE | Poverty estimates | ~18 months | 2023 |
| Scorecard | Earnings/outcomes | ~2-3 years | 2020 |
| MEPS | School poverty | ~2-3 years | 2019 |
Always verify year availability before building pipelines. Use mirror discovery endpoints (see mirrors.yaml) or filter downloaded data to confirm which years are present. See education-data-query skill for mirror-based fetch patterns.
Common Analysis Mistakes
DO NOT:
-
Compare state assessment scores across states (EDFacts)
- Each state has different tests and cut scores
-
Use IPEDS graduation rates to represent all students
- Only tracks first-time, full-time students
-
Assume Scorecard earnings represent all graduates
- Only covers Title IV aid recipients
-
Calculate statistics without filtering coded values
- -1, -2, -3 are not zeros; they corrupt calculations
-
Compare 2020-2021 data to prior years without noting COVID
- Testing waivers, discipline changes, enrollment shifts
-
Merge data across years assuming stable identifiers
- Schools and districts merge, split, and change IDs
-
Assume Portal rate variables are on a 0-100 percentage scale
- Some IPEDS rate variables (e.g.,
completion_rate_150pct, retention_rate) are normalized to 0-1 proportions in the Portal, even though the original IPEDS surveys use 0-100. Always check the actual range after fetching. See "Rate and Proportion Normalization" above.
DO:
- Check suppression rates before disaggregating
- Use within-state comparisons for assessment data
- Document all data limitations in your analysis
- Verify identifier stability for longitudinal analyses
- Cite the data source properly
Cross-References
- Variable definitions: Load
education-data-explorer skill to understand what variables measure
- Query assistance: Load
education-data-query skill to re-fetch data with different parameters
- Joining data: Read
./references/data-relationships.md for identifier mappings
- Deep source context: Load the appropriate
education-data-source-* skill for comprehensive methodology, historical changes, and detailed variable definitions
- Source-specific gotchas: Load the relevant
education-data-source-* skill for variable name mappings, data lags, and endpoint-specific behaviors
Topic Index
| Topic | Location |
|---|
| Bureau of Indian Education schools | ./references/ccd-context.md |
| Charter school coverage | ./references/ccd-context.md |
| Chronic absenteeism | ./references/crdc-context.md |
| Citation format | This file: Citation Requirements |
| COVID-19 data impact | ./references/crdc-context.md |
| Discipline data | ./references/crdc-context.md |
| Dropout definitions | ./references/ccd-context.md |
| Earnings data limitations | ./references/scorecard-context.md |
| Finance data (colleges) | ./references/ipeds-context.md |
| GASB vs FASB accounting | ./references/ipeds-context.md |
| Graduation rate caveats | ./references/ipeds-context.md |
| Identifier relationships | ./references/data-relationships.md |
| Joining tables | ./references/data-relationships.md |
| LEAID format | ./references/data-relationships.md |
| Locale codes | ./references/ccd-context.md |
| Missing value codes | This file: Universal Data Caveats |
| NCESSCH format | ./references/data-relationships.md |
| Net price calculation | ./references/ipeds-context.md |
| ODC-By License | This file: Citation Requirements |
| OPEID vs UNITID | ./references/data-relationships.md |
| Private schools | ./references/ccd-context.md (not covered) |
| Proficiency data | ./references/edfacts-context.md |
| Race category changes | ./references/ccd-context.md |
| Sampling (CRDC) | ./references/crdc-context.md |
| State assessment comparability | ./references/edfacts-context.md |
| State FIPS codes | ./references/data-relationships.md |
| Student financial aid | ./references/ipeds-context.md |
| Suppression | This file: Universal Data Caveats |
| Title IV institutions | ./references/ipeds-context.md |
| Transfer students | ./references/ipeds-context.md |
| UNITID changes | ./references/ipeds-context.md |
| Year definitions | This file: Universal Data Caveats |
