LessonThis is a lesson restated in new words. Any figure in it is an example from the source, not this desk's record.
Training a model means moving parameters until inputs map to the outputs you wanted, with less error. On a bowl-shaped error surface, walking downhill reaches the bottom. On a surface with many valleys, that walk can stop in a valley that is only locally low. Stochastic gradient descent adds noise to each step. Simulated annealing sometimes accepts a worse step on purpose, then does so less often as a temperature cools. The lecture uses both ideas to show why a fit gets stuck. It does not claim that any of these procedures will make money.
Why it matters
Language and image models are trained by reducing error across too many parameters to plot. A two-input surface is only a picture: height is error, and the floor axes are parameters. The update has to work from local information. A single bowl is the easy case. Many losses have several valleys, so different starts finish at different heights.
The idea in plain words
Gradient descent starts at a random parameter vector and computes the gradient, the vector of partial derivatives. That vector points toward the steepest increase of error, so the update subtracts a step in that direction, scaled by a learning rate. A small rate, the lecture's picture uses 0.1, is a short step along the local tangent. A huge rate can leap past the bottom and land high on the far side, because a straight tangent is a poor guide far from where it was drawn. On a convex bowl, repeated short steps settle at the minimum without anyone looking at a plot.
When several basins exist, the same rule stays inside whichever basin holds the start. The lecture's picture ends in a side valley while a deeper valley sits elsewhere. Random restarts can hunt for it, but without a hint about where to start, the hunt need not finish in useful time.
Stochastic gradient descent keeps the subtraction and adds noise to the gradient. Step length is still set by the learning rate. A little noise jitters and still falls into the nearby basin. A larger rate wanders across more of the surface before it settles. Two runs from the same start need not match. On a ridge, a noiseless gradient might step into the worse valley. Noise gives a positive chance of stepping toward the better one. You do not choose the direction of a given noisy step. You only choose how large the noise may be.
Simulated annealing does not follow the gradient. It proposes a random neighbor and looks at the change in error. A better neighbor is accepted for sure. A worse neighbor is accepted with probability given by an exponential in the error increase divided by a temperature. High temperature makes uphill moves common. As the temperature falls, those moves become rare and the search settles. A lecture run of 100 iterations from the same start reached the lowest point on that picture, after plain gradient descent and a low-noise stochastic run had not. Dropping the temperature immediately made the search behave like a cautious gradient method. Keeping it high, or cooling slowly, visited other basins, including ones that were not the deepest.
How to use it
If the error really is one bowl, modest gradient steps are enough and extra noise mostly adds jitter. If the optimizer stops while the fit is still poor, the lecture's diagnosis is a local basin. Then enlarge the noise, or add a rule that still accepts some bad steps early.
The lecture reports an annealing-style schedule with stochastic steps, and with optimizers such as Adam, while training variational autoencoders on arbitrage-free volatility surfaces. Arbitrage-free means the quotes do not contradict one another. In that job the mix is what left basins where the generated surfaces were useless. That is one project, not a claim that local minima vanish in general.
What usually goes wrong
The learning rate is a hyperparameter, a knob you choose. Too small, and a nonconvex run never leaves its valley. Too large, and even a convex run oscillates. Copying 0.1 from a convex demo onto a different loss is not a derivation.
Annealing fails quietly. Aggressive cooling accepts the first decent basin and wastes the temperature. Very slow cooling spends the budget wandering and can still stop in a mediocre place, as one high-temperature trial did. A teaching surface in the lecture also dipped below zero. Real error is usually a squared or absolute distance and stays non-negative. The basins are the point, not the sign of the height.
Questions people ask
Is stochastic gradient descent just a random start?
No. Both methods can start at random. The stochastic version randomizes the step, so the path can move against the local slope.
Does annealing need derivatives?
Not in the form written here. It compares function values at the current point and a proposed neighbor, which still works when a gradient is unavailable.
Why bring up language models?
Only as motivation. A prompt goes in and a response comes out, and that map was trained by reducing error. The lecture stops at the optimizer. It does not cover feedback from human raters, or supervised versus unsupervised learning.