| name | regression-modeling |
| description | Modeling relationships between variables using regression. Covers simple linear regression, multiple regression, polynomial regression, logistic regression, model fitting (least squares, maximum likelihood), residual analysis, model diagnostics, R-squared, adjusted R-squared, multicollinearity, variable selection, and the Box-Jenkins dictum that all models are wrong but some are useful. Use when predicting outcomes, quantifying relationships, building predictive models, or diagnosing model fit. |
| type | skill |
| category | statistics |
| status | stable |
| origin | tibsfox |
| modified | false |
| first_seen | "2026-04-11T00:00:00.000Z" |
| first_path | examples/skills/statistics/regression-modeling/SKILL.md |
| superseded_by | null |
Regression Modeling
Regression models quantify the relationship between a response variable and one or more explanatory variables. The goal may be prediction ("what will Y be when X = 10?"), explanation ("how does Y change when X increases by one unit?"), or both. This skill covers the core regression toolkit from simple linear regression through logistic regression, with emphasis on the diagnostics that separate a useful model from a misleading one.
Agent affinity: box (model building, diagnostics, "all models are wrong"), pearson (correlation, regression theory), efron (computational model fitting)
Concept IDs: stat-descriptive-statistics, stat-hypothesis-testing
Simple Linear Regression
The model
Y = beta_0 + beta_1 * X + epsilon, where epsilon ~ N(0, sigma^2).
- beta_0: Y-intercept. The predicted value of Y when X = 0.
- beta_1: Slope. The change in predicted Y for a one-unit increase in X.
- epsilon: Error term. Captures everything the model does not explain.
Least squares estimation
The least squares estimates minimize the sum of squared residuals:
b_1 = sum((x_i - x-bar)(y_i - y-bar)) / sum((x_i - x-bar)^2)
b_0 = y-bar - b_1 * x-bar
The fitted line y-hat = b_0 + b_1 * x passes through the point (x-bar, y-bar).
Interpretation
- b_1 = 2.3: "For each one-unit increase in X, Y increases by 2.3 units on average."
- b_0 = 15.7: "When X = 0, the predicted Y is 15.7." (Only meaningful if X = 0 is within the data range.)
- Extrapolation warning: The model is only trustworthy within the range of observed X values. Extrapolating beyond that range assumes the linear relationship continues, which may be false.
R-Squared and Model Fit
R-squared (coefficient of determination)
R^2 = 1 - SS_residual / SS_total = SS_regression / SS_total.
R^2 is the proportion of variance in Y explained by the model. Range: 0 to 1. An R^2 of 0.72 means the model explains 72% of the variability in Y.
Adjusted R-squared
R^2_adj = 1 - (SS_residual / (n-k-1)) / (SS_total / (n-1)).
Adjusted R^2 penalizes for adding predictors. It can decrease when a useless predictor is added. Use adjusted R^2, not R^2, for comparing models with different numbers of predictors.
Cautions about R-squared
- A high R^2 does not mean the model is correct. A quadratic relationship fit with a line can have moderate R^2 but systematically wrong predictions.
- A low R^2 does not mean the model is useless. In social science, R^2 = 0.10 with a clear causal mechanism is scientifically important.
