FS2 1.1 - Least squares regression lines

FS2 1.1 - Least squares regression lines

A regression line is chosen by a precise measure of its errors, rather than by eye. You will explain least squares, calculate the regression of a response on an explanatory variable and recover a model when the data have been coded.

What makes a line the best fit?

Can the best-fitting straight line miss every data point?

Yes. For the three points (0,1),(1,2),(2,2)(0,1),(1,2),(2,2), the least squares line is y=7/6+x/2y=7/6+x/2, and none of the points lies on it. “Best” refers to the combined prediction errors, not the number of points hit exactly.

You already know

From A-level Mathematics, a scatter diagram shows the relationship between two variables, and a regression line can estimate a response within a suitable model. Interpolation is generally more defensible than extrapolation; an association does not establish causation. Here we calculate the line's coefficients and explain the least squares criterion.

Choose the response before fitting. In regression of yy on xx, values of xx are used to estimate yy. The errors are measured in the yy-direction.

Fitted value, residual and least squares

For a proposed line y^=a+bx\widehat y=a+bx, the fitted value at observation ii is y^i=a+bxi\widehat y_i=a+bx_i. Its residual is

ei=yi−y^ie_i=y_i-\widehat y_i

The least squares regression line of yy on xx minimises ∑ei2\sum e_i^2, the sum of squared vertical residuals, over choices of aa and bb.

A positive residual means the point is above the line; a negative one means it is below. Squaring prevents opposite signs from cancelling. Least squares does not minimise perpendicular distances to the line.

For the opening points, the fitted values are 7/6,5/3,13/67/6,5/3,13/6. The residuals are

−16,13,−16-\frac16,\quad \frac13,\quad -\frac16

Their squares sum to 1/61/6. In contrast, the line y=1+xy=1+x passes through two points but has residuals 0,0,−10,0,-1, with squared sum 11. Hitting more points does not necessarily give a smaller total squared error.

[DIAGRAM: asset_name: The best line need not pass through a point; asset_slug: 9fm0-fs2-118-vertical-residuals; caption: Least squares minimises squared vertical residuals, rather than the number of missed points; description: A scatter plot shows the three exact points zero comma one, one comma two and two comma two. The line y equals seven sixths plus x over two passes between them. Pale clay vertical segments join each observed point to its fitted value, with residual labels minus one sixth, plus one third and minus one sixth positioned clear of the points and line. The sum of squared residuals is stated as one sixth]
Least squares minimises squared vertical residuals, rather than the number of missed points

The formulas in the next part identify this minimum directly. No trial-and-error search through possible lines is needed.

Key idea

Least squares balances squared vertical prediction errors across the entire dataset; it need not pass through any individual observation.

Check the criterion itself.

Calculating the coefficients from summaries

The line depends on how the two variables vary together, measured relative to the variation in the explanatory variable.

First find the sample means and the centred sums. The expression (∑xi)2(\sum x_i)^2 means square the total; it is different from ∑xi2\sum x_i^2, the sum of individual squares.

Centred sums for regression

For nn paired observations,

x‾=∑xin,y‾=∑yin\overline x=\frac{\sum x_i}{n},\qquad \overline y=\frac{\sum y_i}{n} Sxx=∑(xi−x‾)2=∑xi2−(∑xi)2nS_{xx}=\sum(x_i-\overline x)^2 =\sum x_i^2-\frac{(\sum x_i)^2}{n} Sxy=∑(xi−x‾)(yi−y‾)=∑xiyi−(∑xi)(∑yi)nS_{xy}=\sum(x_i-\overline x)(y_i-\overline y) =\sum x_iy_i-\frac{(\sum x_i)(\sum y_i)}{n}

Use the same paired observations in every sum. In the formula booklet.

The denominator for the slope is SxxS_{xx}, because the response is being fitted on xx. Naming different variables does not change this response-versus-explanatory distinction.

Least squares line of y on x

If Sxx>0S_{xx}>0, the least squares line is

y^=a+bx,b=SxySxx,a=y‾−bx‾\widehat y=a+bx,\qquad b=\frac{S_{xy}}{S_{xx}},\qquad a=\overline y-b\overline x

It therefore passes through the mean point (x‾,y‾)(\overline x,\overline y). Pearson may write the fitted equation as y=a+bxy=a+bx. In the formula booklet.

When all xx-values are equal, Sxx=0S_{xx}=0, so these observations do not determine a unique slope. If Sxy=0S_{xy}=0 while Sxx>0S_{xx}>0, the fitted slope is zero; that alone does not mean all observations lie on the line.

Worked example 1: a positive fitted slope

Question. Eight paired observations have

∑x=80,∑x2=840,∑y=160,∑xy=1624\sum x=80,\quad \sum x^2=840,\quad \sum y=160,\quad \sum xy=1624

Find the regression line of yy on xx.

Plan. Calculate the means and centred sums, then the slope and intercept.

x‾=808=10,y‾=1608=20Sxx=840−8028=40explanatory variationSxy=1624−80(160)8=24centred cross-productb=2440=0.6,a=20−0.6(10)=14\begin{aligned} \overline x&=\frac{80}{8}=10,\qquad \overline y=\frac{160}{8}=20\\ S_{xx}&=840-\frac{80^2}{8}=40 &&\text{explanatory variation}\\ S_{xy}&=1624-\frac{80(160)}8=24 &&\text{centred cross-product}\\ b&=\frac{24}{40}=0.6,\qquad a=20-0.6(10)=14 \end{aligned}

Why this step? The slope divides the centred cross-product by the explanatory sum of squares, not by the number of observations.

Answer. y=14+0.6xy=14+0.6x.

Check. Substituting the mean x=10x=10 gives the mean response 2020.

Continue this lesson.

You’re previewing NovaLearn without an account. Create a free account to read the rest.

Create free account