The geometry of the normal equations
In this article, I show that the normal equations define the orthogonal projection of a vector onto a linear subspace.
Let be a vector space and a linear subspace. Let be a matrix whose column vectors are a basis for , i.e. . Each vector in can be expressed as a linear combination of the column vectors in :
The orthogonal projection
Given a vector , the orthogonal projection of onto is the vector in such that the residual is orthogonal to every vector in :
Solving the equations
A vector is orthogonal to every vector in if and only if , hence we are looking to solve the following equation:
Since , we can find a vector such that:
So the equation to be solved is:
Which is the matrix form of the normal equations.