Multiple

The goal of multiple regression is to develop a linear model that can be used to explain variation in or to predict values of the Y variable. Multiple regression refers to a model in which a dependent variable, Y, is a linear function of a set of independent variables, X₁, X₂, ..., X_P. The population model is written as:
$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \cdots + \beta_p X_p + \varepsilon$

The Y variable is sometimes called the response variable, while the X's are sometimes referred to as the predictor variables. The term ε is a random variable that is uncorrelated with the independent variables.

Suppose we have a random sample of N observations from the population: $\{Y_i,X_{1i},X_{2i}, \dots , X_{pi}\}$ , i = 1,...,N. The sample regression line is written:

$Y_i = b_0 + b_1 X_{i1} + b_2 X_{2i} + \dots + b_p X_{pi} + e_i$

We will use the sample to estimate the parameters of the population regression model. In order to do so, we assume that:

The data $\{Y_i,X_{1i},X_{2i}, \dots , X_{pi}\}$ is a random sample from the population
The random variable ε has a mean of zero

In order to conduct inference in the multiple regression model, we often make an additional assumption:

The random variable ε has a normal distribution with a variance, σ², that is constant

Least squares estimation

As in simple regression, the parameters of the regression model are estimated by finding values of $b_0, b_1, \dots, b_p$ that minimize the sum of squared errors:
$SSE = \sum_{i=1}^{N} e_i^2 = \sum_{i=1}^{N} (Y_i - b_0 - b_1 X_{1i} - \dots - b_p X_{pi})^2$ .
We can write the set of X variables in matrix form, $\bold{X} = [ \bold{1} X_1 \dots X_p]$ where 1 is an Nx1 vector of ones. The matrix X is N x (p+1). we can write the vector of coefficients as $\bold{\beta} = [\beta_0 \text{ } \beta_1 \dots \beta_p]'$ . Then the least squares estimates of the coefficients are given by:

$\bold{\hat{\beta}} = [\hat{\beta_0} \text{ } \hat{\beta_1} \dots \hat{\beta_p}]' = [\bold{X}'\bold{X}]^{-1}\bold{X}'\bold{Y}$ .

The estimate $\hat{\beta}$ is unbiased, which means that, on average, the estimate will equal the population value. More formally, $E(\hat{\beta}) = \beta$ .

If we assume that the random variable ε has a normal distribution, then the estimate has a normal distribution with a mean of β and variance equal to ....

ANOVA

The analysis of variance (ANOVA) table summarizes several quantities associated with the regression equation:

Source	DF	SS	MS	F
Regression	p	SSR	MSR	MSR/MSE
Error	N-p-1	SSE	MSE
Total	N-1	SST

SST = Sum of Squares Total = $\sum_{i=1}^{N} (Y_i - \bar{Y})^2$
SSE = Sum of Squares Error = $\sum_{i=1}^{N} (Y_i - \hat{Y_i})^2$
SSR = Sum of Squares Regression = $\sum_{i=1}^{N} (\hat{Y_i} - \bar{Y})^2$

With some algebra, we could show that SST = SSE + SSR.

The degrees of freedom for SST are labeled DFT, and equals N-1. The degrees of freedom for SSE are labeled DFE and equals N - p - 1. The degrees of freedom for SSR are labeled DFR and equal p. So DFT = DFE + DFR.

MSR is the mean squared regrssion, and equals SSR/DFR. MSE is mean squared error, and equals SSE/DFE.

The F-statistic, F = MSR/MSE, is relevant for testing the null hypothesis that all of the population slope coefficients are equal to zero against the alternative that at least one of the population slope coefficients is not equal to zero:

$H_0: \beta_1 = \beta_2 = \dots = \beta_p = 0 \\ H_A: \text{at least one } \beta_i \text{ does not equal zero}$

If we reject the null hypothesis, we conclude that there is at least one significant variable; in other words, we have enough evidence to conclude that at least one X variable explains variation in the Y variation. If we fail to reject the null hypothesis, we conclude that we don't have enough evidence to show that any of the X variables are significant.

The R² of the regression, R² = SSR/SST, measures the percentage of the variation in Y that is explained by the X variables. Note that R² = 1 - SSE/SST.

Interpretation

The regression coefficient for independent variable X_pi measure the effect of a one unit change in X_pi on Y_i on average, holding the values of the other independent variables constant. For example, suppose we estimate a regression of the price of a new home (Price) on the square footage of the house (SqFt) and the number of rooms (Number):

Price = $20,000 + $150 SqFt + $1500 (Number)

Holding the number of rooms in a house constant, an increase in the square footage of the house by one foot is associated with a $150 increase in the price of a house, on average. Holding the square footage of a house constant, an increase in the number of rooms increases the price of a house by $1500 on average.

Significance of individual coefficients

Testing the significance of the coefficients proceeds along the same lines as in Simple Regression. The hypothesis we wish to test is:
$H_0: \beta_j = 0 \\ H_A: \beta_j \ne 0$ . The test statistic is:
$\frac{\hat{\beta_j}}{SE_{\hat{\beta_j}}}$
This statistic has a t-distribution with N-p-1 degrees of freedom.
If we reject the null hypothesis, we conclude that the variable X_j is significant, given all of the other variables in the model. If we fail to reject the null hypothesis, then we conclude that there is not enough evidence to show that X_j is significant, given all of the other variables in the model. The outcome of a test always depends on the other variables in the model. When we say a variable is significant, we mean that variation in that variable helps to explain variation in the Y variable.

Model building

The best models are built on theoretical knowledge about the situation being studied. If we wish to explain the price of houses, for example, we would want to know what variables affect the price of houses (e.g., number of bathrooms, location, square footage, etc.).