最小二乘法求解线性方程组公式推导

N个未知数的M行（M>N）方程组如下。其中x,y是已知量，k是未知量：

$${\left\{\begin{matrix} k_{1}x_{1,1}+k_{2}x_{1,2}+ \cdots +k_{N}x_{1,N}=y_{1}\\ k_{1}x_{2,1}+k_{2}x_{2,2}+ \cdots +k_{N}x_{2,N}=y_{2}\\ \vdots \\ k_{1}x_{M,1}+k_{2}x_{M,2}+ \cdots +k_{N}x_{M,N}=y_{M} \end{matrix}\right.}$$

一般有两种最小二乘法，分别是加权的和不加权的。我们逐个介绍它们的解法。

一、不加权的最小二乘法

构造误差函数：

$${ \begin{equation} e=\sum_{m=1}^{M}\left ( k_{1}x_{m,1}+k_{2}x_{m,2}+ \cdots +k_{N}x_{m,N}-y_{m} \right )^{2} \end{equation} }$$

为求e的最小值，我们对${k_{n}}$求导，并使导数为零得：

$${\left\{\begin{matrix} \frac{\partial e}{\partial k_{1}}=2 \sum_{m=1}^{M} x_{m,1} \left ( k_{1}x_{m,1}+k_{2}x_{m,2}+ \cdots +k_{N}x_{m,N}-y_{m} \right )=0 \\ \frac{\partial e}{\partial k_{2}}=2 \sum_{m=1}^{M} x_{m,2} \left ( k_{1}x_{m,1}+k_{2}x_{m,2}+ \cdots +k_{N}x_{m,N}-y_{m} \right )=0 \\ \vdots \\ \frac{\partial e}{\partial k_{N}}=2 \sum_{m=1}^{M} x_{m,N} \left ( k_{1}x_{m,1}+k_{2}x_{m,2}+ \cdots +k_{N}x_{m,N}-y_{m} \right )=0 \end{matrix}\right.}$$

整理得：

$${\left\{\begin{matrix} \sum_{m=1}^{M} x_{m,1}x_{m,1}k_{1}+\sum_{m=1}^{M} x_{m,1}x_{m,2}k_{2}+ \cdots +\sum_{m=1}^{M} x_{m,1}x_{m,N}k_{N}=\sum_{m=1}^{M} x_{m,1}y_{m} \\ \sum_{m=1}^{M} x_{m,2}x_{m,1}k_{1}+\sum_{m=1}^{M} x_{m,2}x_{m,2}k_{2}+ \cdots +\sum_{m=1}^{M} x_{m,2}x_{m,N}k_{N}=\sum_{m=1}^{M} x_{m,2}y_{m}\\ \vdots \\ \sum_{m=1}^{M} x_{m,N}x_{m,1}k_{1}+\sum_{m=1}^{M} x_{m,N}x_{m,2}k_{2}+ \cdots +\sum_{m=1}^{M} x_{m,N}x_{m,N}k_{N}=\sum_{m=1}^{M} x_{m,N}y_{m} \end{matrix}\right. }$$

改写成矩阵形式如下：

$${ \begin{pmatrix} \sum_{m=1}^{M} x_{m,1}x_{m,1} & \sum_{m=1}^{M} x_{m,1}x_{m,2} & \cdots & \sum_{m=1}^{M} x_{m,1}x_{m,N} \\ \sum_{m=1}^{M} x_{m,2}x_{m,1} & \sum_{m=1}^{M} x_{m,2}x_{m,2} & \cdots & \sum_{m=1}^{M} x_{m,2}x_{m,N} \\ \vdots & \vdots & \ddots & \vdots \\ \sum_{m=1}^{M} x_{m,N}x_{m,1} & \sum_{m=1}^{M} x_{m,N}x_{m,2} & \cdots & \sum_{m=1}^{M} x_{m,N}x_{m,N} \end{pmatrix} \begin{pmatrix} k_{1}\\ k_{2}\\ \vdots \\ k_{N} \end{pmatrix}= \begin{pmatrix} \sum_{m=1}^{M} x_{m,1}y_{m} \\ \sum_{m=1}^{M} x_{m,2}y_{m} \\ \vdots \\ \sum_{m=1}^{M} x_{m,N}y_{m} \end{pmatrix} }$$

结果就是：

$${ \begin{pmatrix} k_{1}\\ k_{2}\\ \vdots \\ k_{N} \end{pmatrix}= \begin{pmatrix} \sum_{m=1}^{M} x_{m,1}x_{m,1} & \sum_{m=1}^{M} x_{m,1}x_{m,2} & \cdots & \sum_{m=1}^{M} x_{m,1}x_{m,N} \\ \sum_{m=1}^{M} x_{m,2}x_{m,1} & \sum_{m=1}^{M} x_{m,2}x_{m,2} & \cdots & \sum_{m=1}^{M} x_{m,2}x_{m,N} \\ \vdots & \vdots & \ddots & \vdots \\ \sum_{m=1}^{M} x_{m,N}x_{m,1} & \sum_{m=1}^{M} x_{m,N}x_{m,2} & \cdots & \sum_{m=1}^{M} x_{m,N}x_{m,N} \end{pmatrix} ^{-1} \begin{pmatrix} \sum_{m=1}^{M} x_{m,1}y_{m} \\ \sum_{m=1}^{M} x_{m,2}y_{m} \\ \vdots \\ \sum_{m=1}^{M} x_{m,N}y_{m} \end{pmatrix} }$$

从上式可以看出来，其实它就是公式：

$${\mathbf{K}=\left ( \mathbf{X^{T}}\mathbf{X} \right )^{-1}\mathbf{X^{T}}\mathbf{Y} }$$

二、加权的最小二乘法

其实就是(1)式的每一个累加项加上权重${ \omega_{m} }$，其误差函数如下：

$${ \begin{equation} e=\sum_{m=1}^{M} \omega_{m} \left ( k_{1}x_{m,1}+k_{2}x_{m,2}+ \cdots +k_{N}x_{m,N}-y_{m} \right )^{2} \end{equation} }$$

解法同(1)式都是求偏导数。这里不再描述过程了，直接给出结果：

$${ \begin{pmatrix} k_{1}\\ k_{2}\\ \vdots \\ k_{N} \end{pmatrix}= \begin{pmatrix} \sum_{m=1}^{M} \omega_{m}x_{m,1}x_{m,1} & \sum_{m=1}^{M} \omega_{m}x_{m,1}x_{m,2} & \cdots & \sum_{m=1}^{M} \omega_{m}x_{m,1}x_{m,N} \\ \sum_{m=1}^{M} \omega_{m}x_{m,2}x_{m,1} & \sum_{m=1}^{M} \omega_{m}x_{m,2}x_{m,2} & \cdots & \sum_{m=1}^{M} \omega_{m}x_{m,2}x_{m,N} \\ \vdots & \vdots & \ddots & \vdots \\ \sum_{m=1}^{M} \omega_{m}x_{m,N}x_{m,1} & \sum_{m=1}^{M} \omega_{m}x_{m,N}x_{m,2} & \cdots & \sum_{m=1}^{M} \omega_{m}x_{m,N}x_{m,N} \end{pmatrix} ^{-1} \begin{pmatrix} \sum_{m=1}^{M} \omega_{m}x_{m,1}y_{m} \\ \sum_{m=1}^{M} \omega_{m}x_{m,2}y_{m} \\ \vdots \\ \sum_{m=1}^{M} \omega_{m}x_{m,N}y_{m} \end{pmatrix} }$$