Simple

In simple linear regression, the goal is to estimate the linear relationship between a dependent variable, Y, and an independent variable, X. We call this simple regression because there is only one independent variable. We assume that there is a linear relationship between the two variables, and that differences (variation) in X can explain differences (variation) in Y (but not the reverse - variation in Y cannot explain variation in X). More specifically, we assume that a change in X can explain a change in Y. The population regression model is written as:
$Y = \beta_0 + \beta_1 X + \varepsilon$

The essential assumption of the linear regression model is:

the error term $\varepsilon$ has mean zero

For purposes of testing, we often assume that:

the error term has a normal distribution with a constant standard deviation σ

The parameters of the model are the intercept, β₀, the slope β₁, and the standard deviation, σ.

Estimation

Our random sample consists of pairs $(X_i, Y_i), i = 1, \dots,N.$ . We use the data to estimate the parameters of the regression line, β₀ and β₁. The estimates are denoted by b₀ and b₁. Our goal is to find the sample regression line, that is "closest" to the population regression line (so that the vertical distance between the population and sample regression lines is minimized).

The estimates of the parameters of the population regression line can be calculated from the following formulas:

$b_1 = r \frac{s_y}{s_x} \\ b_0 = \bar{y} - b_1\bar{x}$

where r is the correlation coefficient, s_x is the sample standard deviation of x and s_y is the sample standard deviation of y. These formulas are the ordinary least squares (OLS) estimates; these estimates minimize the sum of squared errors of the regression:

$SSE = \sum_{i=1}^{N} (y_i - \hat{y}_i)^2$

We can estimate the standard deviation of the error term with the following formula:
$s = \sqrt{\frac{1}{N-2}\sum_{i=1}^{N}e_i^2}$

Under assumption 2, the regression estimates, b₀ and b₁, have normal distributions with means of β₀ and β₁ respectively. We can form confidence intervals for the slope coefficient using the t distribution with N-2 degrees of freedom:

$[b_1 - t^* SE_{b_1}, b_1 + t^* SE_{b_1}]$

where t* is a critical value from the t-distribution and SE represents the standard error of the slope coefficients:
$SE_{b_1} = \frac{s}{\sqrt{\sum(x_i - \bar{x})^2}}$

The predicted value of y_i for a particular value of x_i is denoted by $\hat{y}$ . Since the prediction uses estimates of the parameters, it has a sampling distribution and, hence, a standard error. The standard error of prediction for an individual response when x = x* is:

IR $SE_{\hat{y}}=s \sqrt{1 + \frac{1}{N} + \frac{(x^* - \bar{x})^2}{\sum{(x_i - \bar{x})^2}}}$

The standard error for predicting the mean (or average) response, $\hat{\mu}$ , for a particular value of x, x*, is given by:

ave $SE_{\hat{\mu}} = s\sqrt{\frac{1}{N} + \frac{(x^* - \bar{x})^2}{\sum(x_i - \bar{x})^2}}$

Inference

We can test the whether a linear relationship exists between X and Y by testing the following hypothesis:
$H_0: \beta_1 = 0 \\ H_A: \beta_1 \ne 0$
The relevant test statistic is
$t = \frac{b_1}{SE_{\beta_1}}$
which has a t-distribution with N-2 degrees of freedom.

Prediction

As above, to predict a value of y given a particular value of x, denoted by x*, we substitute the value of x into the estimated regression equation:

$\hat{y} = b_0 + b_1x^*$ .

This is also the mean or average response of y for a particular value of x, x*.

The average response is different from the predicted response. Suppose we have a sample of students for whom we know college GPA and high school GPA. We could use simple linear regression to predict college GPA from high school GPA. If we then wish to predict the college GPA of a particular student who had a high school GPA of 3.25, we would substitute 3.25 into the estimated regression equation. The appropriate standard error for that prediction is the standard error for an individual response.

If we wish to predict the average college GPA for all students whose high school GPAs were 3.25, we would again substitute 3.25 into the estimated regression equation. However, the appropriate standard error in this case is the standard error for the average.

Simple

Estimation

Inference

Prediction

Labels

Children (1)