Simple

In simple linear regression, we study the relationship between a dependent variable, Y, and an independent variable, X. We call this simple regression because there is only one independent variable. We assume that there is a linear relationship between the two variables, and that variation in X can explain variation in Y (but not the reverse - variation in Y cannot explain variation in X). The population regression model is written as:

The point of the regression model is to either explain variation in the Y variable or to develop a model for predicting values of Y.

The assumptions of the linear regression model are

1. the error term e has mean zero
2. the error term has a normal distribution with a constant standard deviation σ

The parameters of the model are the intercept, β₀, the slope β₁, and the standard deviation, σ.

Estimation

Our sample consists of pairs of observations (x_i,y_i), i = 1,...,N. We use the data to estimate the parameters of the regression line, β₀ and β₁. The estimates are denoted by b₀ and b₁. Our goal is to find the sample regression line that is closest to the population regression line.

The estimates of the parameters of the population regression line can be calculated from the following formulas:

where r is the correlation coefficient, s_x is the sample standard deviation of x and s_y is the sample standard deviation of y. These formulas are the ordinary least squares (OLS) estimates; these estimates minimize the sum of squared errors of the regression:

We can estimate the standard deviation of the error term with the following formula:

Under assumption 2, the regression estimates, b₀ and b₁, have normal distributions with means of ß₀ and ß₁ respectively. We can form confidence intervals for each using the t distribution with N-2 degrees of freedom:

where t* is a critical value from the t-distribution and SE represents the standard error of the respective coefficients:

The predicted value of y_i for a particular value of x_i is then given by .

Inference

We can test the whether a linear relationship exists between X and Y by testing the following hypothesis:

The relevant test statistic is

which has a t-distribution with N-2 degrees of freedom.

Minitab commands

The command for simple and multiple regression is found under Stat>Regression>Regression. The response is the dependent variable, Y; the predictor is the independent variable, X.