LessonThis is a lesson restated in new words. Any figure in it is an example from the source, not this desk's record.
Two features are linearly independent when the only weights that combine them into the zero vector are zeros. If some other weights also work, one feature is a remix of the other, and a model that uses both is paying for a copy. In the plane, the unit arrows along the axes are independent and can build every point. A pair that lies on one line cannot.
Why it matters
A vector here is a magnitude and a direction, not a pin stuck at one place. The arrow from the origin to (1, 1) is the same vector as any parallel arrow of that length drawn somewhere else. The length, also called the norm, is the square root of the sum of squared coordinates. For (1, 1) that is the square root of 2.
Addition is coordinate by coordinate. Scalar multiplication stretches every coordinate by the same number. The dot product multiplies matching coordinates and adds those products. A dot product of zero means the arrows are orthogonal: the angle between them is 90 degrees. Orthogonal vectors are linearly independent. The converse is not required for the definition below, but the right angle is the picture that makes independence obvious.
The idea in plain words
Call the axis arrows i = (1, 0) and j = (0, 1). Each has length 1, and their dot product is 0, so they are an orthonormal basis of the plane: orthogonal, and normal in the sense of unit length. Any point (a, b) is a times i plus b times j. Those two arrows span the plane. Span means every point is some weighted sum of them.
The formal test uses weights. Vectors m and n are linearly independent when q1 times m plus q2 times n equals the zero vector only if both weights are zero. For i and j, a nonzero weight on either arrow leaves a nonzero coordinate, so the test passes.
Dependence is the opposite. Take m = (1, 2) and n = (2, 4). The second is twice the first. The weights negative 2 and positive 1 give zero: negative 2 times (1, 2) plus 1 times (2, 4) is (0, 0). A nontrivial solution means dependence. On a plot, the two arrows sit on one line. Every combination of them stays on that line. The point (1, 0) is not on it, so no weights recover (1, 0). The second arrow added no new direction. Its information is redundant.
The same algebra is why a third feature can be useless. If q1 x + q2 y + q3 z = 0 and q3 is not zero, you can solve for z as a weighted sum of x and y. Collecting z does not add a direction the first two features lacked. In a credit-style example from the lecture, three series aimed at an interest rate collapse in exactly this way when one is a blend of the others. You do not need the third series.
How to use it
Before you add a feature, ask the weight question. Could a few of the columns, with some coefficients, rebuild another column? If yes, drop or replace the rebuilt one. In two dimensions you can see it: parallel arrows are dependent, and arrows that are not parallel are independent.
What usually goes wrong
Treating “more columns” as “more information” is the modeling failure. The third series in the credit sketch is real data and still adds nothing once it is a linear blend of the first two. The cost is collection, a wider matrix, and coefficients that trade off against each other because the fit cannot tell the copies apart.
Questions people ask
Does a dot product of zero prove independence?
Yes, for the vectors whose dot product you computed. A right angle implies independence. Independence does not require a right angle. (1, 0) and (1, 1) are independent and not orthogonal.
What is a basis in the plane?
A basis is a small set of vectors that can rebuild every vector in that space as a weighted sum. i = (1, 0) and j = (0, 1) do that for the plane, and they are orthonormal because they are perpendicular and each has length 1.
Why drop a feature that fails the weight test?
Because it is a weighted sum of features you already have. The lecture’s three-input sketch shows the third input solved exactly from the first two, so the extra measurements do not enlarge the set of directions the model can use.