education-data-source-ipeds

# IPEDS Data Source Reference IPEDS (Integrated Postsecondary Education Data System) — the primary federal data system for ~6,500 U.S. postsecondary institutions, comprising 12+ annual survey components: enrollment, completions, graduation rates, finance, financial aid, admissions, human resources, and institutional characteristics (1980-present, varies by component). Use when analyzing postsecondary enrollment, degree completions by CIP code, institutional finances, or admissions data. Graduation rates track first-time full-time students only (150% cohort). Cross-sector finance comparisons require care due to GASB vs. FASB accounting. Comprehensive guide to understanding and using IPEDS data correctly. IPEDS is the most widely used source for postsecondary education data but has significant complexities — including sector-specific accounting standards, cohort-limited graduation rates, and integer-encoded categorical variables — that users must understand. > **CRITICAL: Value Encoding** > > This document describes **Education Data Portal** integer encodings, which differ from NCES raw file string codes. The Portal converts categorical variables to integers for consistency across sources. > > | Context | Race White | Race Black | Sex Male | Sector Public 4-yr | > |---------|------------|------------|----------|-------------------| > | **Portal (integers)** | `1` | `2` | `1` | `1` | > | NCES raw files | `EFFY_WHITE` | `EFFY_BKAA` | `M` | varies | > > **Always verify codes against Portal codebooks** (available alongside each dataset in the Portal mirrors). ## What is IPEDS? IPEDS (Integrated Postsecondary Education Data System) is a system of 12+ interrelated survey components: - **Administered by**: National Center for Education Statistics (NCES) - **Coverage**: ~6,500 Title IV-participating postsecondary institutions - **Frequency**: Annual collection in three periods (Fall, Winter, Spring) - **Mandate**: Required for Title IV federal student aid participation - **Available years**: 1980-present (varies by component) - **Primary identifier**: UNITID (6-digit institution ID) - **Available through**: Education Data Portal mirrors (32 datasets covering most survey components; some variables not mirrored — see Data Access section) ## Reference File Structure | File | Purpose | When to Read | |------|---------|--------------| | `survey-components.md` | All 12+ IPEDS surveys with collection periods | Understanding data structure | | `graduation-rates.md` | **CRITICAL** GRS limitations and who is tracked | Any graduation rate analysis | | `enrollment-data.md` | Fall vs 12-month, FTE calculations | Enrollment comparisons | | `finance-data.md` | GASB vs FASB accounting standards | Cross-sector finance analysis | | `financial-aid.md` | Net price, aid types, populations | Aid and cost analysis | | `institution-identifiers.md` | UNITID, OPEID, mergers, closures | Data linking and longitudinal work | | `completions-data.md` | Degrees awarded, CIP codes | Completions and outcomes | | `data-quality.md` | Known issues, sector comparisons | Quality assurance | ## Decision Trees ### What data am I working with? ``` Working with IPEDS data? ├─ Graduation rates → ./references/graduation-rates.md (READ FIRST!) ├─ Enrollment counts → ./references/enrollment-data.md ├─ Finance/revenue/expenses → ./references/finance-data.md ├─ Financial aid/net price → ./references/financial-aid.md ├─ Degrees/completions → ./references/completions-data.md ├─ Institutional info → ./references/survey-components.md (IC section) ├─ Human resources/salaries → ./references/survey-components.md (HR section) └─ Linking to other data → ./references/institution-identifiers.md ``` ### Is my analysis valid? ``` Cross-sector comparison? ├─ Comparing grad rates across sectors │ └─ CAUTION: Different populations → ./references/graduation-rates.md ├─ Comparing finances across sectors │ └─ CAUTION: GASB vs FASB → ./references/finance-data.md ├─ Comparing net price across sectors │ └─ CAUTION: Aid populations differ → ./references/financial-aid.md └─ Time series analysis └─ Check for institutional changes → ./references/institution-identifiers.md ``` ### Finding specific variables? ``` Need variable definitions? ├─ Survey component overview → ./references/survey-components.md ├─ Graduation cohort definitions → ./references/graduation-rates.md ├─ Enrollment level/status → ./references/enrollment-data.md ├─ Revenue/expense categories → ./references/finance-data.md ├─ Aid types and populations → ./references/financial-aid.md └─ CIP codes for programs → ./references/completions-data.md ``` ## Quick Reference: Survey Components | Component | Abbrev | Collection | Key Content | |-----------|--------|------------|-------------| | Institutional Characteristics | IC | Fall | Directory, tuition, mission | | 12-Month Enrollment | E12 | Fall | Unduplicated headcount, FTE | | Completions | C | Fall | Degrees by CIP, demographics | | Cost | CST | Fall/Winter | Cost of attendance, net price | | Admissions | ADM | Winter | Applications, admits, enrollees | | Student Financial Aid | SFA | Winter | Aid counts and amounts | | Graduation Rates | GR | Winter | 150% completion rates | | Graduation Rates 200% | GR200 | Winter | 200% completion rates | | Outcome Measures | OM | Winter | Part-time and transfer outcomes | | Fall Enrollment | EF | Spring | Point-in-time enrollment | | Finance | F | Spring | Revenue, expenses, assets | | Human Resources | HR | Spring | Employees, salaries | | Academic Libraries | AL | Spring | Library resources (biennial) | ### Key Identifiers | ID | Format | Level | Example | Notes | |----|--------|-------|---------|-------| | `unitid` | 6-digit integer | Institution | `100654` | Unique, persistent across years; changes on merger | | `opeid` | 8-digit string | Institution (Title IV) | `00100200` | Links to FSA/NSLDS; shared across branches | ### Institution Type Codes | Variable | Values | Meaning | |----------|--------|---------| | `inst_control` | 1 | Public | | | 2 | Private nonprofit | | | 3 | Private for-profit | | | -1 | Missing/not reported | | `institution_level` | 1 | Less than 2-year | | | 2 | 2-year (at least 2 but less than 4) | | | **4** | 4-year or above | | | -1 | Missing/not reported | | `sector` | 0 | Administrative unit | | | 1 | Public, 4-year or above | | | 2 | Private not-for-profit, 4-year or above | | | 3 | Private for-profit, 4-year or above | | | 4 | Public, 2-year | | | 5 | Private not-for-profit, 2-year | | | 6 | Private for-profit, 2-year | | | 7 | Public, less-than 2-year | | | 8 | Private not-for-profit, less-than 2-year | | | 9 | Private for-profit, less-than 2-year | | | -1 | Sector unknown (not active) | | `hbcu` | 1 | Historically Black College/University | | | 0 | Not HBCU | | | -1 | Missing/not reported | | `tribal_college` | 1 | Tribal College | | | 0 | Not Tribal College | | | -1 | Missing/not reported | | `degree_granting` | 1 | Degree-granting | | | 0 | Non-degree-granting | **Note:** There is **no code 3** for `institution_level`. The Portal uses codes 1, 2, 4 (not 1, 2, 3). ### `inst_size` Categories | Code | Meaning | |------|---------| | 1 | Under 1,000 | | 2 | 1,000 - 4,999 | | 3 | 5,000 - 9,999 | | 4 | 10,000 - 19,999 | | 5 | 20,000 and above | **Note:** `inst_size` is a category code (1-5), not an actual enrollment count. ### Race/Ethnicity Codes (Portal Integer Encoding) | Code | Category | Notes | |------|----------|-------| | `1` | White | Single race, non-Hispanic | | `2` | Black | Single race, non-Hispanic | | `3` | Hispanic | Any race | | `4` | Asian | Single race, non-Hispanic | | `5` | American Indian/Alaska Native | Single race, non-Hispanic | | `6` | Native Hawaiian/Pacific Islander | Single race, non-Hispanic | | `7` | Two or more races | Multiple races selected, non-Hispanic | | `8` | Nonresident alien | International students | | `9` | Unknown | Race/ethnicity unknown | | `20` | Other | Other race/ethnicity | | `99` | Total | All races combined | | `-1` | Missing/not reported | | | `-2` | Not applicable | | | `-3` | Suppressed | Privacy protection | > **Historical note:** Prior to 2010, Asian included Pacific Islanders (code 6 did not exist), and "Two or more races" (code 7) was not collected. ### Sex Codes (Portal Integer Encoding) | Code | Category | |------|----------| | `1` | Male | | `2` | Female | | `3` | Nonbinary/Another gender | | `4` | Unknown/Prefer not to say | | `9` | Unknown | | `99` | Total | | `-1` | Missing/not reported | | `-2` | Not applicable | | `-3` | Suppressed | > **Note:** Codes 3 and 4 are recent additions for non-binary gender reporting. Historical data may only have codes 1, 2, and 99. The exact meaning of codes 3 vs 4 may vary by endpoint — check the specific codebook. ### Missing Data Codes | Code | Meaning | When Used | |------|---------|-----------| | `-1` | Missing/not reported | Data not submitted by institution | | `-2` | Not applicable | Item doesn't apply to this institution type | | `-3` | Suppressed | Data suppressed for privacy | | `null` | Not available | Field not collected for this survey year | ### Year Field Meanings | Data Type | Year Field Meaning | |-----------|--------------------| | Institutional characteristics | As of fall of indicated year | | Fall enrollment | As of fall census date | | 12-month enrollment | July 1 to June 30 academic year | | Completions | Awarded during academic year | | Graduation rates | **Cohort entered** in indicated year | | Finance | Fiscal year ending in indicated year | | Student financial aid | For indicated academic year | ## Data Access Datasets for IPEDS are available via the mirror system. See `datasets-reference.md` for canonical paths, `mirrors.yaml` for mirror configuration, and `fetch-patterns.md` for fetch code patterns. **Key datasets:** | Dataset | Type | Path | Codebook | |---------|------|------|----------| | Directory | Single | `ipeds/colleges_ipeds_directory` | `ipeds/codebook_colleges_ipeds_directory` | | Admissions | Single | `ipeds/colleges_ipeds_admissions-enrollment` | `ipeds/codebook_colleges_ipeds_admissions-enrollment` | | Enrollment FTE | Single | `ipeds/colleges_ipeds_enrollment-fte` | `ipeds/codebook_colleges_ipeds_enrollment-fte` | | Graduation Rates | Single | `ipeds/colleges_ipeds_grad-rates` | `ipeds/codebook_colleges_ipeds_grad-rates` | | Finance | Single | `ipeds/colleges_ipeds_finance` | `ipeds/codebook_colleges_ipeds_finance` | 32 IPEDS datasets exist in the mirror (5 shown above). See `datasets-reference.md` for the complete list with all paths and codebook paths. > **Known Portal gaps:** > - **Distance education enrollment variables** (`efdeexc`, `efdesom`, `efdenom`) are not in Portal mirror datasets. Use the NCES IPEDS Data Center for these. > - **Open-admissions policy variable** (`OPENADMP`) is not in Portal mirror datasets. Note: `open_public` is NOT the same thing — see Common Pitfalls below. > - **Finance data** may have a year lag relative to NCES releases (last verified through 2017 in some datasets). > > For data not available through Portal mirrors, access NCES directly at https://nces.ed.gov/ipeds/. Codebooks are `.xls` files co-located with data in all mirrors. Use `get_codebook_url()` from `fetch-patterns.md` to construct download URLs: ```python url = get_codebook_url("ipeds/codebook_colleges_ipeds_directory") ``` > **Truth Hierarchy:** When interpreting variable values, apply this priority: > 1. **Actual data file** (what you observe in the parquet/CSV) — this IS the truth > 2. **Live codebook** (.xls in mirror) — authoritative documentation, may lag > 3. **This skill documentation** — convenient summary, may drift from codebook > > If this documentation contradicts the codebook, trust the codebook. If the codebook contradicts observed data, trust the data and investigate. ### Filtering ```python import polars as pl # Admissions totals: filter to sex=99 for institution-level totals # WRONG - includes duplicates (~26K rows with multiple sex values per institution) df = pl.read_parquet("data/raw/admissions.parquet") # CORRECT - one row per institution-year (~8K rows)