平方损失函数为例的BP的关键公式推导

看了刘建平老师的博客https://www.cnblogs.com/pinard/p/6422831.html对如下其中两个公式进行详细推导
损失函数为（大写字母为矩阵，小写字母字母加粗为列向量，其中 $W^L$ 的维度为 $M_L*M_{L-1}$ ,即第 $L$ 层神经元个数乘以第 $L-1$ 层神经元个数）：

J (W, b, x, y) = \frac{1}{2} {‖ a^{L} - y ‖}_{2}^{2} = \frac{1}{2} {‖ σ (W^{L} a^{L - 1} + b^{L}) - y ‖}_{2}^{2}

$J(W, \mathbf{b}, \mathbf {x}, \mathbf{y})=\frac{1}{2}\left\|\mathbf{a}^{L}-\mathbf{y}\right\|_{2}^{2}=\frac{1}{2}\left\|\sigma\left(W^{L} \mathbf {a}^{L-1}+\mathbf {b}^{L}\right)-\mathbf {y}\right\|_{2}^{2}$

推导的两个公式如下：

\frac{\partial J (W, b, x, y)}{\partial W^{L}} = [(a^{L} - y) ⊙ σ^{'} (z^{L})] {(a^{L - 1})}^{T}

$\frac{\partial J(W, \mathbf{b}, \mathbf {x}, \mathbf{y})}{\partial W^{L}}=\left[\left(\mathbf {a}^{L}-\mathbf {y}\right) \odot \sigma^{\prime}\left(\mathbf {z}^{L}\right)\right]\left(\mathbf {a}^{L-1}\right)^{T}$

\frac{\partial J (W, b, x, y)}{\partial b^{L}} = (a^{L} - y) ⊙ σ^{'} (z^{L})

$\frac{\partial J(W, \mathbf{b}, \mathbf {x}, \mathbf{y})}{\partial \mathbf {b}^{L}}=\left(\mathbf {a}^{L}-\mathbf {y}\right) \odot \sigma^{\prime}\left(\mathbf{z}^{L}\right)$

用到两个链式求导法则如下（都来源于刘建平老师博客，链接在文章末尾）
当标量对n个向量进行链式求导，即 $\mathbf{y}_{1} \rightarrow \mathbf{y}_{2} \rightarrow \ldots \rightarrow \mathbf{y}_{\mathbf{n}} \rightarrow z_{1}$ ，链式求导法则如下：

\begin{matrix} (1) & \frac{\partial z}{\partial y_{1}} = {(\frac{\partial y_{n}}{\partial y_{n - 1}} \frac{\partial y_{n - 1}}{\partial y_{n - 2}} \dots \frac{\partial y_{2}}{\partial y_{1}})}^{T} \frac{\partial z}{\partial y_{n}} \end{matrix}

$\begin{equation}\frac{\partial z}{\partial \mathbf{y}_{\mathbf{1}}}=\left(\frac{\partial \mathbf{y}_{\mathbf{n}}}{\partial \mathbf{y}_{\mathbf{n}-\mathbf{1}}} \frac{\partial \mathbf{y}_{\mathbf{n}-\mathbf{1}}}{\partial \mathbf{y}_{\mathbf{n}-\mathbf{2}}} \ldots \frac{\partial \mathbf{y}_{\mathbf{2}}}{\partial \mathbf{y}_{\mathbf{1}}}\right)^{T} \frac{\partial z}{\partial \mathbf{y}_{\mathbf{n}}}\tag {1}\end{equation}$

当 $z=f(\mathbf {y}),\mathbf {y}=X\mathbf {a}+\mathbf {b}$ 即 $X\rightarrow \mathbf{y}\rightarrow z$ 其中 $X$ 为矩阵， $\mathbf {y}$ 为向量，链式求导结果如下：

\begin{matrix} (2) & \frac{\partial z}{\partial X} = \frac{\partial z}{\partial y} a^{T} \end{matrix}

$\begin{equation}\frac{\partial z}{\partial X}=\frac {\partial z}{\partial {\mathbf{y}}}a^T\tag {2}\end{equation}$

先推导第一个公式，考虑如下复合结构(注意最后所求的 $J$ 是标量)

W^{L} \to z^{L} \to u^{L} \to J

$W^L\rightarrow \mathbf{z}^L\rightarrow \mathbf{u}^L\rightarrow J$

其中 $J=\frac{1}{2}\Vert \mathbf{u}^L \Vert_2^2$

u^{L} = a^{L} - y = σ (z^{L}) - y

$\mathbf{u}^L=\mathbf{a}^L-\mathbf{y}=\sigma (\mathbf{z}^L)-\mathbf{y}$

z^{L} = W^{L} a^{L - 1} + b^{L}

$\mathbf{z}^L=W^L\mathbf{a}^{L-1}+\mathbf{b}^L$

由公式 $(2)$ 可得

\frac{\partial J}{\partial W^{L}} = \frac{\partial J}{\partial z^{L}} (a^{L - 1})^{T}

$\frac{\partial J}{\partial W^{L}}=\frac{\partial J}{\partial \mathbf {z}^{L}}(\mathbf a^{L-1})^T$

又有公式 $(1)$ 可得

\frac{\partial J}{\partial z^{L}} = (\frac{\partial u^{L}}{\partial z^{L}})^{T} \frac{\partial J}{\partial u^{L}}

$\frac{\partial J}{\partial \mathbf {z}^{L}}=(\frac {\partial \mathbf{u}^L}{\partial \mathbf{z}^L})^T\frac {\partial J}{\partial \mathbf{u}^L}$

其中后半部分比较简单

\frac{\partial J}{\partial u^{L}} = u^{L} = a^{L} - y

$\frac {\partial J}{\partial \mathbf{u}^L}=\mathbf {u}^L=\mathbf{a}^L-\mathbf{y}$

前半部分向量对向量求导，布局为雅克比矩阵形式，结果如下：

\frac{\partial u^{L}}{\partial z^{L}} = \frac{\partial (σ (z^{L}) - y)}{\partial z^{L}} = (\begin{array}{lllc} \frac{\partial σ (z_{1}^{L})}{\partial z_{1}^{L}} & 0 & \dots & 0 \\ 0 & \frac{\partial σ (z_{2}^{L})}{\partial z_{2}^{L}} & \dots & 0 \\ ⋮ & ⋮ & ⋱ & ⋮ \\ 0 & 0 & \dots & \frac{\partial σ (z_{M_{L}}^{L})}{\partial z_{M_{L}}^{L}} \end{array}) = diag (σ^{'} (z^{L}))

$\frac{\partial \mathbf{u}^{L}}{\partial \mathbf{z}^{L}}=\frac{\partial\left(\sigma\left(\mathbf{z}^{L}\right)-\mathbf{y}\right)}{\partial \mathbf{z}^{L}}=\left(\begin{array}{lllc} \frac{\partial \sigma\left(z_{1}^{L}\right)}{\partial z_{1}^{L}} & 0 & \cdots & 0 \\ 0 & \frac{\partial \sigma\left(z_{2}^{L}\right)}{\partial z_{2}^{L}} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \frac{\partial \sigma\left(z_{M_{L}}^{L}\right)}{\partial z_{M_{L}}^{L}} \end{array}\right)=\operatorname{diag}\left(\sigma^{\prime}\left(\mathbf{z}^{L}\right)\right)$

得到的矩阵为对称矩阵，带入到上式，结果为：

\frac{\partial J}{\partial z^{L}} = diag (σ^{'} (z^{L})) (a^{L} - y) = σ^{'} (z^{L}) ⊙ (a^{L} - y) = (a^{L} - y) ⊙ σ^{'} (z^{L})

$\frac{\partial J}{\partial \mathbf {z}^{L}}=\operatorname{diag}\left(\sigma^{\prime}\left(\mathbf{z}^{L}\right)\right)(\mathbf{a}^L-\mathbf{y})=\sigma^{\prime}\left(\mathbf{z}^{L}\right)\odot (\mathbf{a}^L-\mathbf{y})=(\mathbf{a}^L-\mathbf{y})\odot\sigma^{\prime}\left(\mathbf{z}^{L}\right)$

故

\frac{\partial J}{\partial W^{L}} = [(a^{L} - y) ⊙ σ^{'} (z^{L})] {(a^{L - 1})}^{T}

$\frac{\partial J}{\partial W^{L}}=\left[\left(\mathbf {a}^{L}-\mathbf {y}\right) \odot \sigma^{\prime}\left(\mathbf {z}^{L}\right)\right]\left(\mathbf {a}^{L-1}\right)^{T}$

第二个式子推导就很简单，由公式 $(1)$ 可得