Normal Equation

The normal equation is \(A^TAx=A^Tb\) with \(A\in R^{m\times n}\) and \(m\gg n\).

Proof.

1. \(rank(A)=rank(A^TA)\)
2. \(col(A^TA)\subset row(A)\)

• From 1 and 2, we can see \(col(A^TA)=row(A)\).

3. Obviously \(A^Tb \in row(A)=col(A^TA)\), so the normal equation always has solution.

• If \(rank(A^TA)=n\), then the solution is unique.
• If \(rank(A^TA)<n\), there are infinitely many solutions. We can find the solution with minimal euclidean norm using SVD theory.