THE LEAST SQUARES METHOD

The Least Squares Method
By Elena Llaudet, co-author of Data Analysis for Social Science: A Friendly and Practical Introduction (Princeton University Press, 2022)

 

To find the line of best fit, we use the least squares method, which chooses the line that minimizes the sum of the squared errors. Let's explore this in detail.

Take a look at the scatter plot on the right, which shows the relationship between two variables, X and Y. In theory, we could draw an infinite number of lines on this scatter plot, but some lines will do a better job than others at summarizing the relationship between X and Y. Intuitively, we know that the line of best fit should be as close to the data (that is, the dots in the scatter plot) as possible. Mathematically, this means that the line of best fit should have the smallest errors possible, where the errors are defined as the vertical distances between the dots and the line.

 

 

Let's consider the line shown in the scatter plot on the right (Line #1). Does this line do a good job in summarizing the relationship between X and Y? Not really. Among other things, Line #1 has a negative slope, while X and Y clearly have a positive linear relationship. (Higher values of X are often associated with higher values of Y and lower values of X are often associated with lower values of Y.)

One way to explore how well a line fits the data is to consider the errors associated with the line, that is, the vertical distances between the dots and the line. In this case, as we can see in the animated GIF, the errors (shown as vertical dashed red lines) are rather large because Line #1 is pretty far away from the data. We can conclude, then, that this line does not summarize the relationship between X and Y well because it fits the data poorly.

 

 

Let's see what happens when we consider a different line (Line #2). Does the new line shown in the scatter plot on the right fit the data better? Yes, it does. Line #2 is closer to the data than Line #1. As a result, the errors associated with Line #2 are much smaller. (The vertical distances between the dots and the line, shown as dashed red lines in the animated GIF, are much shorter.)

Is Line #2 the line of best fit? No, it is not. We can fit a line even closer to the data, that is, with even smaller errors, which will do a better job at summarizing the relationship X and Y.

 

 

See, for example, the new line shown on the scatter plot on the right (Line #3). As we can see in the animated GIF, the errors associated with this line are the smallest among the three lines we have considered because Line #3 is the one closest to the data. In fact, Line #3 is the line with the smallest errors among all possible lines. It is, therefore, the line of best fit; the line that best summarizes the relationship between X and Y.

Mathematically, the least squares method finds the line that minimizes the sum of the squared errors (also known as residuals). Why do we want to minimize the sum of the *squared* errors rather than minimize the sum of the errors? Because we want to avoid having positive errors cancel out negative errors. This method is called *least squares* because it *minimizes* the sum of the *squared* errors, hence, it finds *the least squares.*

 

 

OVERVIEW: In the animated GIF to the right, we can see the original scatter plot and the three lines we have considered with their associated errors. Here we can clearly see that the errors become smaller as we go from Line #1 to Line #2 and as we go from Line #2 to Line #3, improving the fit of the line to the data with every change. 

(Looking for more animated GIFs and/or interactive graphs to help explain statistical concepts? Check my website.)