education-data-source-edfacts

# EDFacts Data Source Reference EDFacts — federal K-12 outcome data from State Education Agencies, covering state assessment proficiency rates, ACGR graduation rates, and ESSA accountability indicators at school and district level (assessments 2009-2020, graduation rates 2010-2019). Use when analyzing within-state achievement trends, subgroup proficiency gaps, or adjusted cohort graduation rates. Complements CCD (school characteristics) with outcome data. State assessment scores CANNOT be compared across states; use NAEP for cross-state comparisons. EDFacts is the U.S. Department of Education's centralized data collection system for pre-K through grade 12 education data from State Education Agencies (SEAs). It provides state assessment proficiency rates, graduation rates, and accountability indicators — the authoritative federal source for state-level K-12 outcome data. > **CRITICAL: Value Encoding** > > The Urban Institute Education Data Portal converts NCES string codes (e.g., `ALL`, `CWD`, `LEP`) to **integer codes**. Always verify actual data values before filtering — do not rely on documentation labels alone. > > | Context | Subgroup "All" | English Learner | Sex "Male" | > |---------|----------------|-----------------|------------| > | **Portal integer** | `99` | `1` | `1` | > | NCES string | `ALL` | `LEP` | `M` | > > See `./references/variable-definitions.md` for complete encoding tables. ## What is EDFacts? - **Collector**: U.S. Department of Education, via State Education Agencies (SEAs) - **Coverage**: All public schools and districts in 50 states + DC - **Content**: State assessment proficiency rates, ACGR graduation rates, participation rates, accountability indicators - **Frequency**: Annual collection - **Available years**: Assessments 2009-10 to present; Graduation rates 2010-11 to present - **Primary identifiers**: `ncessch` (school ID, Int64), `leaid` (district ID, Int64), `fips` (state FIPS code, Int64) - **Key limitation**: State assessment scores CANNOT be compared across states (different tests, different cut scores) - **Available through**: Education Data Portal mirrors ## Reference File Structure | File | Purpose | When to Read | |------|---------|--------------| | `accountability-context.md` | ESSA, NCLB history, accountability systems | Understanding policy context | | `assessment-data.md` | Proficiency levels, test scores, limitations | Working with assessment data | | `graduation-rates.md` | ACGR methodology, cohort definitions | Analyzing graduation data | | `variable-definitions.md` | Key variables, suppression codes, special values | Interpreting specific variables | | `data-quality.md` | Known issues, state variations, COVID impacts | Data cleaning, limitations | | `subgroup-reporting.md` | Special populations, disaggregation | Analyzing by student groups | ## Decision Trees ### What type of analysis? ``` What EDFacts data do you need? ├─ Assessment/proficiency data │ ├─ Within-state trends → Valid analysis │ ├─ Cross-state comparison → INVALID - use NAEP instead │ └─ Subgroup gaps → See ./references/subgroup-reporting.md ├─ Graduation rates (ACGR) │ ├─ Understand methodology → See ./references/graduation-rates.md │ ├─ Extended rates (5-year, 6-year) → See ./references/graduation-rates.md │ └─ Subgroup rates → See ./references/subgroup-reporting.md ├─ Understanding variables │ ├─ Missing/suppressed values → See ./references/variable-definitions.md │ ├─ Range vs. exact values → See ./references/variable-definitions.md │ └─ Subgroup codes → See ./references/subgroup-reporting.md └─ Data quality concerns ├─ COVID-19 impacts (2019-20) → See ./references/data-quality.md ├─ State reporting changes → See ./references/data-quality.md └─ Suppression rates → See ./references/data-quality.md ``` ### Is my comparison valid? ``` What are you comparing? ├─ Same state, different years │ ├─ Same assessment system? → Valid │ └─ Different tests? → Break in time series ├─ Schools within same state → Valid ├─ Districts within same state → Valid ├─ Subgroups within same school → Valid (check suppression) ├─ Different states │ ├─ Proficiency rates → INVALID │ ├─ Graduation rates (ACGR) → More comparable │ └─ Use NAEP instead → Valid └─ National ranking by proficiency → INVALID ``` ## Quick Reference: EDFacts Data Elements ### Assessment Data | Data Element | Description | Available Years | |--------------|-------------|-----------------| | Proficiency rates | % meeting state standards in reading/math | 2009-10 to present | | Participation rates | % of students assessed | 2012-13 to present | | Achievement levels | Below Basic, Basic, Proficient, Advanced | Varies by state | | Grade levels | Grades 3-8, high school (varies) | 2009-10 to present | ### Graduation Data | Data Element | Description | Available Years | |--------------|-------------|-----------------| | 4-year ACGR | Adjusted Cohort Graduation Rate | 2010-11 to present | | 5-year ACGR | Extended graduation rate | 2011-12 to present | | 6-year ACGR | Further extended rate | 2012-13 to present | | Diploma types | Regular diploma only in ACGR | All years | ### Key Identifiers > **Portal Data Types:** All identifiers are **Int64** in the Portal parquet files. The NCES source format (zero-padded strings) is shown for reference only. When joining with other Portal datasets, join on the integer columns directly. | ID | Portal Type | NCES Source Format | Level | Example (Int64) | |----|-------------|-------------------|-------|-----------------| | `ncessch` | Int64 | 12-char zero-padded | School | `10000500870` | | `ncessch_num` | Int64 | Same as ncessch | School | `10000500870` | | `leaid` | Int64 | 7-char zero-padded | District/LEA | `100005` | | `leaid_num` | Int64 | Same as leaid | District/LEA | `100005` | | `fips` | Int64 | 2-digit | State | `1` (Alabama) | ### Data Levels | Level | Identifier | Dataset Path Pattern | |-------|------------|---------------------| | School | `ncessch` (Int64) | `edfacts/schools_edfacts_*` | | District/LEA | `leaid` (Int64) | `edfacts/districts_edfacts_*` | | State | `fips` (Int64) | Aggregate from lower levels | ### Subgroups Reported > **Note:** Not all subgroup columns are present in every dataset. Grad rates data does NOT have `sex`, `migrant`, or `military_connected` columns. | Subgroup | NCES Code | Portal Integer | Column | Available In | |----------|-----------|----------------|--------|--------------| | All students | `ALL` | `99` | race, sex, lep, disability | Assessments, Grad Rates | | Economically disadvantaged | `ECODIS` | `1` | econ_disadvantaged | Assessments, Grad Rates | | Students with disabilities | `CWD` | `1` | disability | Assessments, Grad Rates | | English learners | `LEP` | `1` | lep | Assessments, Grad Rates | | Homeless | `HOM` | `1` | homeless | Assessments, Grad Rates | | Foster care | `FCS` | `1` | foster_care | Assessments, Grad Rates | | Migrant | `MIG` | `1` | migrant | Assessments only | | Military connected | `MIL` | `1` | military_connected | Assessments only | | Race/ethnicity | Multiple | `1-7, 99` | race | Assessments, Grad Rates | | Sex | `M/F` | `1, 2, 99` | sex | Assessments only | **EDFacts Filter Column Pattern:** - Special population columns (lep, disability, homeless, etc.) use `1` = subgroup, `99` = total - Race column uses integer codes (1=White, 2=Black, etc.) - Sex column uses `1` = Male, `2` = Female, `99` = Total (assessments only) ### Grade Codes (grade_edfacts) | Code | Grade Level | |------|-------------| | `3`-`8` | Grades 3-8 (individual) | | `9` | Grades 9-12 combined | | `99` | Total (all grades) | ### Race Codes > **Empirically verified** from 2018 school assessment data. Only these values appear in the `race` column: | Code | Category | |------|----------| | `1` | White | | `2` | Black | | `3` | Hispanic | | `4` | Asian | | `5` | American Indian/Alaska Native | | `7` | Two or More Races | | `99` | Total | > **Note:** Code `6` (Native Hawaiian/Pacific Islander) is NOT observed in the data. Codes `8` (Nonresident alien), `9` (Unknown), `20` (Other), `-1`, `-2`, `-3` are also not observed in the race column. These codes may exist in other Portal sources but are absent from EDFacts. ### Sex Codes | Code | Category | |------|----------| | `1` | Male | | `2` | Female | | `9` | Unknown | | `99` | Total | ### Disability Codes > **Empirically verified** from 2018 school assessment and 2019 grad rate data. Only `1` and `99` are observed in the `disability` column. The expanded codes (0-4) documented in other Portal sources are NOT present in EDFacts datasets. | Code | Category | |------|----------| | `1` | Students with disabilities (IDEA-eligible) | | `99` | Total (all students) | ### LEP Codes | Code | Category | |------|----------| | `1` | Students who are limited English proficient | | `99` | All students (total) | ### Special Population Columns For `homeless`, `migrant`, `econ_disadvantaged`, `foster_care`, `military_connected`: | Code | Category | |------|----------| | `1` | Yes (in subgroup) | | `99` | Total (all students) | ### Missing Data Codes | Code | Meaning | When Used | |------|---------|-----------| | `-1` | Missing/not applicable | Data not reported | | `-2` | Not reported | Item doesn't apply to this entity | | `-3` | Suppressed for privacy | Data suppressed for small N-size | | `-9` | Rounds to zero | Value rounds to zero | | Range values | Exact value suppressed | Range provided instead of exact value | | `_midpt` suffix | Calculated midpoint of suppressed range | Use for analysis when exact values are suppressed | **Always use `_midpt` variables for analysis when exact values are suppressed.** ## Data Access All EDFacts data is fetched via the **Education Data Portal mirror system**. There is no API access. **Key references:** - **`mirrors.yaml`** -- Mirror definitions, URL templates, read strategies - **`datasets-reference.md`** -- Canonical dataset paths (one path works for all mirrors) - **`fetch-patterns.md`** -- `fetch_from_mirrors()` and `fetch_yearly_from_mirrors()` patterns > **Truth Hierarchy:** When interpreting variable values, apply this priority: > 1. **Actual data file** (what you observe in the parquet/CSV) — this IS the truth > 2. **Live codebook** (.xls in mirror) — authoritative documentation, may lag > 3. **This skill documentation** — convenient summary, may drift from codebook > > If this documentation contradicts the codebook, trust the codebook. If the codebook contradicts observed data, trust the data and investigate. ### Key Datasets | Dataset | Path | Type | Columns | |---------|------|------|---------| | School Assessments | `edfacts/schools_edfacts_assessments_{year}` | Yearly (2009-2018, 2020) | 26 cols | | School Grad Rates | `edfacts/schools_edfacts_grad_rates_{year}` | Yearly (2010-2019) | 18 cols | | District Assessments | `edfacts/districts_edfacts_assessments_{year}` | Yearly (2009-2018, 2020) | 23 cols |