Statistics › Section N · Counting, Bayesian inference, and multiple regression › Topic N.5

Reading regression output and multiple regression

Research papers, business reports and software summaries all speak one language: the regression table. This final topic teaches that language, then extends it - several predictors at once, each holding the others constant - the main tool observational studies use to adjust for other variables.

The regression table

Regression software prints much the same table everywhere: each coefficient's estimate with its standard error, its t statistic and its p-value, plus the model's \(r^2\) and residual standard error at the foot. Nothing in it is new - the slope test is the correlation-significance topic's \(t = b_1/SE_{b_1}\) on \(n-2\) df, the \(r^2\) and \(s_e\) are the residual topic's summaries - but the table is where research is actually read. Learning to move down its columns - estimate, then uncertainty, then verdict - is the last mechanical skill this course teaches, and the one a reader of research uses weekly.

Multiple regression

Real outcomes answer to several variables at once: income depends on education AND experience; a house's price depends on size AND location; a patient's recovery depends on treatment AND age. Multiple regression fits them all in one line:

\hat{y} = b_0 + b_1x_1 + b_2x_2 + \cdots

and each coefficient is read the same way as a simple slope - the predicted change in \(y\) per unit of ITS variable, HOLDING THE OTHERS CONSTANT. With two predictors the mathematics fits a plane through the data instead of a line; the interpretation stays one-dimensional, one coefficient at a time.

Holding others constant - the payoff

The phrase is everything. "Each extra year of education is associated with about $3,200 more income, holding experience constant" - the education coefficient from an income regression that also includes experience. Without the adjustment, education's coefficient would also soak up any effect of experience that happens to travel with education; WITH it, the two paths separate, as closely as observational data allow. This is how regression adjusts for confounders - the lurking-variable problem from the correlation topic, given a working tool. It is still association, not experiment: unmeasured confounders (ability, family background) remain, and only randomisation clears them. But "holding constant" is the observational sciences' best approximation, and it matters enough that the phrase must never be dropped from the interpretation.

Worked example: reading a table, line by line

A salary study of \(n = 200\) workers regresses income (thousands of dollars) on years of education and years of experience. The table: education \(b_1 = 3.2\), \(SE = 0.8\), \(t = 4.0\), \(p < 0.001\); experience \(b_2 = 1.5\), \(SE = 0.7\), \(t \approx 2.14\), \(p \approx 0.033\); intercept \(b_0 = 18\), \(p \approx 0.06\). Each t test has \(200 - 3 = 197\) df, one lost per estimated coefficient. Reading: 1. Education: about $3,200 per year, convincing evidence it is nonzero (\(t = 4.0\)). 2. Experience: about $1,500 per year, significant at the 5% level but not by much (\(p \approx 0.033\)). 3. Intercept: $18,000 for zero education and zero experience - technically the model's floor, practically a formality. 4. Check: the 95% interval for education runs \(3.2 \pm 1.97 \times 0.8 \approx (1.6, 4.8)\) thousand - the plausible per-year value, sized.

Worked example: what the model does NOT say

The same table tempts three over-readings, worth rehearsing. 1. "Education CAUSES $3,200 per year" - no: the coefficient is an association among workers, and workers with more education differ in unmeasured ways; only randomised evidence could make the causal claim. 2. "Dropping out to zero education means $18,000" - no: the intercept sits far from the data (few workers have no education and no experience), and the model's line is not evidence there. 3. "Experience barely matters, since its p-value is close to 0.05" - no: 0.033 is below 0.05, so experience IS significant at that level; what remains uncertain is its size, since its interval, \(1.5 \pm 1.97 \times 0.7\), runs from about $100 to $2,900 per year. The table states; the reader interprets, carefully.

Categorical predictors: indicator variables

Regression speaks numbers, but categories can enter through indicator variables (dummy variables): a predictor equal to 1 for members of a group and 0 for everyone else. A two-category variable needs one indicator; its coefficient then reads as the average difference between the groups, adjusted for every other predictor in the model - "the wage gap between the categories, holding education and experience constant". A k-category variable needs \(k-1\) indicators, with the leftover group serving as the baseline that the others' coefficients measure against. This one trick lets sex, region, treatment arm, season - any category - sit in the same table as the quantitative variables.

Multicollinearity: when predictors travel together

When two predictors are highly correlated with each other, their individual coefficients become unstable: the model cannot tell which of the two deserves the credit, so the estimates wobble from sample to sample and the standard errors inflate. This is multicollinearity, and it is common - house size and number of bedrooms, a person's height and weight, a test's subscales. The predictions stay fine; the interpretation suffers, because "holding bedrooms constant" barely varies in the data and the size coefficient answers a nearly-hollow question. The checks: look at the predictor correlations, and watch for coefficients that flip signs when a near-duplicate enters or leaves the model.

Overfitting: too many predictors

Every added predictor raises \(r^2\) - or at worst leaves it - so \(r^2\) alone rewards stuffing the model with variables, and a model chased that way will fit the SAMPLE's noise rather than the world's signal. This is overfitting: a model that gleams on the data it was built on and stumbles on new data. The guards: adjusted \(r^2\), which penalises extra predictors; and testing on data the model never saw, the gold standard. The link to the effect-size topic's multiple-testing warning is direct: trying predictor after predictor until something looks significant is multiple testing in another form, and the same honesty disciplines apply.

Common mistakes to avoid

First, quoting a multiple-regression coefficient without "holding the others constant" - the phrase is not decoration; it is the meaning. Second, claiming causation from observational coefficients: adjustment is approximation, not experiment. Third, skipping the LINE checks because the output looks polished - the table's t-tests inherit every condition of the correlation-significance topic. Fourth, adding predictors to raise \(r^2\): overfitting with extra steps. Fifth, trusting coefficients whose predictors are near-duplicates - multicollinearity makes them unstable. Sixth, reading the intercept's p-value as a test of the model: the intercept is a floor, rarely a claim.

Key takeaways

The regression table reads coefficient by coefficient: estimate, SE, \(t\), p - the slope test of the significance topic, tabulated, with \(r^2\) and \(s_e\) at the foot. Multiple regression extends the line to \(\hat{y} = b_0 + b_1x_1 + b_2x_2 + \cdots\), each coefficient meaning the predicted change per unit of its own variable, holding the others constant - the working tool of confounder adjustment, still association, never experiment. The salary table: education $3,200 per year (\(t = 4.0\), interval about 1.6 to 4.8 thousand), experience significant but imprecise, intercept a formality. Categories enter as indicators against a baseline. Watch multicollinearity (unstable coefficients) and overfitting (high \(r^2\), poor new-data predictions), and interpret every coefficient with the phrase that makes it true: holding others constant.

Flashcards

Flashcard

Regression table

Estimate, SE, t, p per coefficient; r² and s_e below

The research paper's language.

Test your understanding

Question 1 of 5

A regression of income on education and experience gives an education coefficient of 3.2 (thousand dollars per year). This means:

Question 2 of 5

In the salary table, education has b = 3.2 with SE = 0.8. The 95% interval is roughly:

Question 3 of 5

A categorical predictor with 3 categories needs how many indicator variables?

Question 4 of 5

House size and bedroom count are highly correlated predictors in a price model. The likely symptom:

Question 5 of 5

Why can a rise in r² alone not justify adding another predictor?

← Previous topic