LessonThis is a lesson restated in new words. Any figure in it is an example from the source, not this desk's record.
Ordinary least squares draws the line that minimizes the sum of squared gaps between observed points and the line. With one input, the line is an intercept plus a slope times the input. Stack a column of ones next to the input, call that matrix X, and stack the intercept and the slope into a vector b. The minimizer is b = (X transpose X) inverse, times X transpose y. The lesson reaches that formula from a scatter of company sentiment against the next day's stock price, and it closes by showing how a line fit on a handful of points can miss later points.
Why it matters
A picture of a positive cloud and a line through it is the goal. The matrix form is what makes the goal computable, and what generalizes when a second input appears. The sentiment example is the lesson's story for why someone would bother. If new sentiment readings arrive, the fitted line produces a fitted price. The lesson wants the line both as a prediction device and as a description of the cloud already in hand. It also wants the warning that a line fit to a small cloud moves when more of the cloud arrives. Lasso and ridge are named as a later fix and are not derived.
The idea in plain words
The scatter puts sentiment on the horizontal axis and the next price on the vertical axis. The cloud slopes up. A line through it has the school form y = m x + b. In the lesson's letters the price is alpha plus beta times sentiment, plus an error. The error may be modeled as normal, or it may have some other distribution. The fitted values ignore the error: y-hat equals alpha plus beta times sentiment.
For n observations, y-hat is an n by 1 vector. Sentiment is an n by 1 vector. Alpha and beta are single numbers. The compact version builds X as n by 2. The first column is ones, so the intercept is applied once per row. The second column is the sentiment readings. b is the 2 by 1 vector of alpha and beta. Their product X b is n by 1, and the first row of that product is alpha plus beta times the first sentiment, which is the first fitted price. Every other row repeats the pattern.
Differentiate the squared-error product with respect to b and set the derivative to zero. Terms without b drop. After the factors of two cancel, the normal equation says X transpose X times b equals X transpose y. Multiply by the inverse of X transpose X. Then b equals that inverse times X transpose y. Shapes settle the order: X transpose y is 2 by 1, and so is b.
How to use it
Form y as the column of prices. Form X with a column of ones and a column of sentiment. Compute X transpose X, invert it, and multiply by X transpose y. The first entry of the result is the intercept alpha. The second is the slope beta. Fitted prices are X times that vector. Squared error for any other intercept and slope is at least as large, which is what the derivative condition was for.
The lesson then sketches five points, a line through them, and later points that keep the upward slope but pull the line. A price read from the first line can sit too high. Regularization, lasso and ridge, is named as a later repair and is not derived.
What usually goes wrong
Dropping the column of ones forces the line through the origin. That is a different model. The lesson's alpha lives in that column.
Transposing in the wrong place produces a vector whose length is n instead of 2, or a row instead of a column. Check shapes at each multiply. X is n by 2, y is n by 1, and b is 2 by 1.
Minimizing the raw sum of errors, without squares, lets positive and negative gaps cancel. The lesson squares first so that a point below the line counts the same, in the criterion, as a point the same distance above it.
The five-point sketch is the overfitting warning: a line through a small sample can overstate a later price. Normality is optional. The squared-error minimizer is the same formula either way.
Questions people ask
What goes in the two columns of X?
A column of ones for the intercept, and a column of the sentiment readings for the slope.
What is the formula for the coefficients?
b equals (X transpose X) inverse times X transpose y. The entries of b are the intercept and the slope.
What do the five points illustrate?
A line fit to a small sample can sit away from a line fit after more points arrive, and a price read from the first line can be too high. Regularization is named as a later topic.