Use this Skill for social stratification: intergenerational income mobility (Chetty rank-rank), occupational prestige (ISEI), EGP class schema, and transition matrices.
Use this Skill for social stratification: intergenerational income mobility (Chetty rank-rank), occupational prestige (ISEI), EGP class schema, and transition matrices.
β is the IGE. High β (close to 1) = low mobility (child income closely tracks parent income).
The US IGE is approximately 0.45; Nordic countries are closer to 0.15-0.25.
Rank-Rank Slope: Convert income to percentile ranks within cohort, then regress:
rank_child = α + ρ × rank_parent + ε
ρ (rank-rank slope) is less sensitive to outliers than IGE and better suited to censored or
top-coded income data. Chetty et al. (2014) found ρ ≈ 0.341 for the United States.
Upward Mobility: P(child in Q5 | parent in Q1) — fraction of children born in the bottom
income quintile who reach the top quintile as adults. This measure (sometimes called "Chetty
mobility") varies enormously across geographic areas and demographic groups.
ISEI (Hauser & Warren 1997): An occupation scoring system derived from the regression of
income and education on the Standard Occupational Classification. Scores range from ~16 (lowest
manual) to ~90 (physicians, judges). Assigned from 4-digit ISCO or national occupation codes.
EGP Schema (Erikson, Goldthorpe & Portocarero 1979): Categorical class schema:
Transition matrix and social fluidity: The (origin × destination) mobility table shows
probabilities of ending in each class given each origin class. The odds ratio:
OR(a,b;c,d) = (n_ac × n_bd) / (n_ad × n_bc)
where a,b are origin classes and c,d are destination classes. OR = 1 means equal relative
odds (perfect fluidity). Deviations from 1 indicate barriers to mobility.
PSID (Panel Study of Income Dynamics) data: https://psidonline.isr.umich.edu/
Register for free access. Download cross-year individual/family files.
export PSID_PATH="/data/psid_crossyear.csv"
Core Workflow
import os
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors
from scipy import stats
import warnings
warnings.filterwarnings("ignore")
# ---------------------------------------------------------------------------# 1. IGE and Rank-Rank Slope# ---------------------------------------------------------------------------defintergenerational_elasticity(
parent_income: pd.Series,
child_income: pd.Series,
) -> dict:
"""
Estimate intergenerational income elasticity (log-log OLS).
Parameters
----------
parent_income, child_income : pd.Series
Income series (positive values; zeros will be dropped).
Returns
-------
dict with ige (beta), r_squared, n, intercept.
"""
mask = (parent_income > 0) & (child_income > 0)
lp = np.log(parent_income[mask])
lc = np.log(child_income[mask])
slope, intercept, r, p, se = stats.linregress(lp, lc)
return {
"ige": round(slope, 4),
"intercept": round(intercept, 4),
"r_squared": round(r ** 2, 4),
"se": round(se, 4),
"p_value": round(p, 4),
"n": int(mask.sum()),
}
defrank_rank_slope(
parent_income: pd.Series,
child_income: pd.Series,
bootstrap_n: int = 500,
seed: int = 42,
) -> dict:
"""
Estimate rank-rank slope with bootstrap confidence interval (Chetty method).
Parameters
----------
parent_income, child_income : pd.Series
bootstrap_n : int
Number of bootstrap replications for CI.
seed : int
Returns
-------
dict with rr_slope, intercept, ci_lower, ci_upper, se, n.
"""
mask = parent_income.notna() & child_income.notna()
pr = parent_income[mask].rank(pct=True)
cr = child_income[mask].rank(pct=True)
slope, intercept, r, p, se = stats.linregress(pr, cr)
# Bootstrap CI
rng = np.random.default_rng(seed)
boot_slopes = []
n = len(pr)
pr_arr, cr_arr = pr.values, cr.values
for _ inrange(bootstrap_n):
idx = rng.integers(0, n, n)
s, *_ = stats.linregress(pr_arr[idx], cr_arr[idx])
boot_slopes.append(s)
ci_lo, ci_hi = np.percentile(boot_slopes, [2.5, 97.5])
return {
"rr_slope": round(slope, 5),
"intercept": round(intercept, 5),
"ci_lower": round(ci_lo, 5),
"ci_upper": round(ci_hi, 5),
"se": round(np.std(boot_slopes), 5),
"n": int(mask.sum()),
}
defupward_mobility_rate(
parent_income: pd.Series,
child_income: pd.Series,
n_quintiles: int = 5,
from_quintile: int = 1,
to_quintile: int = 5,
) -> dict:
"""
Compute upward mobility: P(child in top quintile | parent in bottom quintile).
Returns
-------
dict with p_upward, n_origin, n_both.
"""
mask = parent_income.notna() & child_income.notna()
pq = pd.qcut(parent_income[mask], n_quintiles, labels=False) + 1
cq = pd.qcut(child_income[mask], n_quintiles, labels=False) + 1
origin_mask = pq == from_quintile
n_origin = origin_mask.sum()
n_both = ((pq == from_quintile) & (cq == to_quintile)).sum()
return {
"p_upward": round(n_both / n_origin, 5) if n_origin > 0else np.nan,
"n_origin": int(n_origin),
"n_destination": int(n_both),
"from_quintile": from_quintile,
"to_quintile": to_quintile,
}
# ---------------------------------------------------------------------------# 2. ISEI Occupational Prestige# ---------------------------------------------------------------------------# Simplified ISEI lookup by ISCO-08 major group (real application uses 4-digit codes)
ISEI_ISCO_MAJOR = {
1: 68, # Managers2: 74, # Professionals3: 56, # Technicians and associate professionals4: 40, # Clerical support workers5: 32, # Services and sales workers6: 23, # Skilled agricultural workers7: 34, # Craft and related trades8: 31, # Plant and machine operators9: 20, # Elementary occupations0: 47, # Armed forces
}
defassign_isei(occupation_codes: pd.Series, lookup: dict | None = None) -> pd.Series:
"""
Assign ISEI scores from occupation codes.
Parameters
----------
occupation_codes : pd.Series
ISCO major group codes (1-digit integers for simplified lookup).
lookup : dict, optional
Custom {occupation_code: isei_score} mapping.
Returns
-------
pd.Series of ISEI scores.
"""
lkp = lookup or ISEI_ISCO_MAJOR
return occupation_codes.map(lkp)
# EGP schema: mapping from ISCO major group to EGP class (simplified)
EGP_ISCO_MAJOR = {
1: "I", # Managers → Higher service2: "I", # Professionals → Higher service3: "II", # Technicians → Lower service4: "IIIa", # Clerical → Routine non-manual5: "IIIb", # Service workers → Routine non-manual6: "IVc", # Agricultural self-employed7: "V_VI", # Craft → Skilled manual8: "V_VI", # Operators → Skilled manual9: "VIIa", # Elementary → Unskilled manual
}
EGP_ORDER = ["I", "II", "IIIa", "IIIb", "IVa", "IVb", "IVc", "V_VI", "VIIa", "VIIb"]
defassign_egp(occupation_codes: pd.Series, lookup: dict | None = None) -> pd.Series:
"""Assign EGP class labels from occupation codes."""
lkp = lookup or EGP_ISCO_MAJOR
return occupation_codes.map(lkp)
# ---------------------------------------------------------------------------# 3. Intergenerational Transition Matrix# ---------------------------------------------------------------------------defmobility_transition_matrix(
origin_class: pd.Series,
destination_class: pd.Series,
class_order: list[str] | None = None,
normalize: str = "origin",
) -> pd.DataFrame:
"""
Compute an intergenerational class transition matrix.
Parameters
----------
origin_class : pd.Series
Parent's (or respondent's father's) class.
destination_class : pd.Series
Respondent's current class.
class_order : list of str, optional
Ordered list of class labels for the matrix.
normalize : str
'origin' (row percentages), 'destination' (column), or 'all'.
Returns
-------
pd.DataFrame — transition proportions (percentages).
"""
classes = class_order orsorted(set(origin_class.dropna()) | set(destination_class.dropna()))
tab = pd.crosstab(
origin_class, destination_class,
values=np.ones(len(origin_class)), aggfunc="sum",
normalize=normalize,
).reindex(index=classes, columns=classes, fill_value=0)
return (tab * 100).round(1)
defcompute_odds_ratio(
table: pd.DataFrame,
class_a: str,
class_b: str,
class_c: str,
class_d: str,
) -> float:
"""
Compute odds ratio for social fluidity from a mobility table.
OR = (n_ac × n_bd) / (n_ad × n_bc)
"""
n_ac = table.loc[class_a, class_c]
n_bd = table.loc[class_b, class_d]
n_ad = table.loc[class_a, class_d]
n_bc = table.loc[class_b, class_c]
if n_ad == 0or n_bc == 0:
return np.nan
return (n_ac * n_bd) / (n_ad * n_bc)
Advanced Usage
Shapley Decomposition of Income Variance
import numpy as np
import pandas as pd
import statsmodels.api as sm
defshapley_r2_decomposition(
outcome: pd.Series,
predictors: dict[str, pd.Series],
) -> pd.DataFrame:
"""
Shapley decomposition of R² across predictor groups.
For each predictor group, compute the average marginal contribution to R²
across all possible orderings (approximated by sequential addition and deletion).
Parameters
----------
outcome : pd.Series
predictors : dict {group_name: pd.Series}
Returns
-------
pd.DataFrame with group, shapley_r2, pct_contribution.
"""from itertools import combinations
groups = list(predictors.keys())
df_all = pd.concat([outcome] + list(predictors.values()), axis=1).dropna()
y = df_all.iloc[:, 0]
defr2_for_subset(group_subset):
ifnot group_subset:
return0.0
X = sm.add_constant(df_all[[g for g in group_subset]])
model = sm.OLS(y, X).fit()
return model.rsquared
n = len(groups)
shapley = {g: 0.0for g in groups}
weight = 1.0 / n
for size inrange(n):
for subset in combinations(groups, size):
base = r2_for_subset(list(subset))
for g in groups:
if g notin subset:
marginal = r2_for_subset(list(subset) + [g]) - base
shapley[g] += marginal / (n * len(list(combinations(
[x for x in groups if x != g], size))))
total = sum(shapley.values())
rows = [{"group": g, "shapley_r2": round(v, 5),
"pct_contribution": round(v / total * 100, 2) if total > 0else np.nan}
for g, v in shapley.items()]
return pd.DataFrame(rows).sort_values("shapley_r2", ascending=False)
Troubleshooting
Problem
Cause
Solution
IGE > 1 or < 0
Outliers in income distribution
Winsorize at 1st/99th percentile before log transform
Rank-rank slope unstable
Small sample size
Bootstrap CI is wide with N < 500; increase sample or report CI
ISEI mapping fails
Non-standard occupation codes
Convert national codes to ISCO-08 first using crosswalk table
Transition matrix rows don't sum to 100
Missing values in one class
Check for NaN after assign_egp(); assign "Unknown" category
Odds ratio is NaN
Zero cell in the table
Collapse rare classes; add 0.5 Laplace smoothing
External Resources
Chetty, R. et al. (2014). Where is the land of opportunity? QJE, 129(4), 1553-1623.
Erikson, R. & Goldthorpe, J.H. (1992). The Constant Flux. Oxford University Press.