反向传播算法及其梯度扩散
最近开始认真学习了下反向传播算法和梯度传递的问题，其本质是导数的链式法则的应用，由此可以分析出为什么sigmoid激活函数不适合用于做多层网络的激活函数，可以考虑联系我的另一篇关于激活函数的文章。如有谬误，请联系指正。转载请注明出处。 联系方式： e-mail : FesianXu@163.comQQ : 973926198github : https://github.com/FesianXu完整代码开源： click me || 其中的sigmoid_base_BP.py
要知道什么是反向传播算法，我们还是要从神经网络开始将起。神经网络如图所示。
图中所示的是一个具有一个输入层（input layer），一个隐藏层（hidden layer），一个输出层（output layer）的三层神经网络。我们都知道，在神经网络的训练过程中，其权值是通过梯度的变化大小来更新的，我们这里先进行符号规定，以便于接下来的分析。w j k l w^l_{jk} w j k l l-1 层的第k个神经元与l 层的第j个神经元的连接权值
b j l b^l_{j} b j l l 层的第j个神经元的偏置。
z j l z^l_{j} z j l l 层的第j神经元的输入。
a j l a^l_{j} a j l l 层的第j神经元的激活输出。
用公式表达即是：(1.1) z j l = ∑ k ( w j k l ∗ a k l − 1 + b j l ) z^l_{j} = \sum_{k} (w^l_{jk}*a^{l-1}_{k}+b^l_j) \tag{1.1} z j l = k ∑ ( w j k l ∗ a k l − 1 + b j l ) ( 1 . 1 )
(1.2) a j l = σ ( z j l ) = σ ( ∑ k ( w j k l ∗ a k l − 1 + b j l ) ) a^l_j = \sigma(z^l_j) = \sigma(\sum_k (w^l_{jk}*a^{l-1}_k+b^l_j) ) \tag{1.2} a j l = σ ( z j l ) = σ ( k ∑ ( w j k l ∗ a k l − 1 + b j l ) ) ( 1 . 2 )
其中的σ ( x ) \sigma(x) σ ( x ) sigmoid，ReLU，tanh 等。C = 1 2 n ∗ ∑ x ∣ ∣ y − a L ∣ ∣ 2 C = \frac{1}{2n}*\sum_x||y-a^L||^2 C = 2 n 1 ∗ ∑ x ∣ ∣ y − a L ∣ ∣ 2
我们知道，进行权值更新的时候，我们采用的是梯度下降法更新(Gradient Descent)， 公式如下：(1.3) w j k l : = w j k l − η ∗ ∂ C ∂ w j k l
w^l_{jk} := w^l_{jk}-\eta*\frac{\partial{C}}{\partial{w^l_{jk}}}
\tag{1.3}
w j k l : = w j k l − η ∗ ∂ w j k l ∂ C ( 1 . 3 ) ∂ C ∂ w j k l \frac{\partial{C}}{\partial{w^l_{jk}}} ∂ w j k l ∂ C
δ L = ∇ a C ⨀ σ ′ ( z L ) \delta^L = \nabla_a{C}\bigodot \sigma^\prime(z^L) δ L = ∇ a C ⨀ σ ′ ( z L ) 其中我们要注意的是，对于某层的误差δ l \delta^l δ l δ l = ∂ C ∂ z l \delta^l = \frac{\partial{C}}{\partial{z^l}} δ l = ∂ z l ∂ C δ j l = ∂ C ∂ z j l \delta^l_j = \frac{\partial{C}}{\partial{z^l_j}} δ j l = ∂ z j l ∂ C
推导：(2.1) δ j L = ∂ C ∂ z j L = ∂ C ∂ a j L ∗ ∂ a j L ∂ z j L = ∂ C ∂ a j L ∗ σ ′ ( z j L )
\delta^L_j = \frac{\partial{C}}{\partial{z^L_j}} = \frac{\partial{C}}{\partial{a^L_j}}*\frac{\partial{a^L_j}}{\partial{z^L_j}} = \frac{\partial{C}}{\partial{a^L_j}}*\sigma^\prime(z^L_j)
\tag{2.1}
δ j L = ∂ z j L ∂ C = ∂ a j L ∂ C ∗ ∂ z j L ∂ a j L = ∂ a j L ∂ C ∗ σ ′ ( z j L ) ( 2 . 1 )
所以当扩展到对于所有的第L层的神经元时，我们为了方便用一个矩阵去代表我们的结果：(2.2) δ L = ( δ 1 L δ 2 L )
\delta^L = \left(\begin{array}{c}
\delta^L_1 \\
\delta^L_2
\end{array}\right)
\tag{2.2}
δ L = ( δ 1 L δ 2 L ) ( 2 . 2 )
(2.3) δ l = ( δ 1 l , δ 2 l , ⋯  , δ n l ) T
\delta^l = (\delta^l_1, \delta^l_2, \cdots,\delta^l_n)^T
\tag{2.3}
δ l = ( δ 1 l , δ 2 l , ⋯ , δ n l ) T ( 2 . 3 )
(2.4) σ ′ ( z l ) = ( σ ′ ( z 1 l ) , σ ′ ( z 2 l ) , ⋯  , σ ′ ( z n l ) )
\sigma^\prime(z^l) = (\sigma^\prime(z^l_1),\sigma^\prime(z^l_2),\cdots,\sigma^\prime(z^l_n))
\tag{2.4}
σ ′ ( z l ) = ( σ ′ ( z 1 l ) , σ ′ ( z 2 l ) , ⋯ , σ ′ ( z n l ) ) ( 2 . 4 ) (2.5) δ L = ∇ a C ⨀ σ ′ ( z L )
\delta^L = \nabla_a{C}\bigodot \sigma^\prime(z^L)
\tag{2.5}
δ L = ∇ a C ⨀ σ ′ ( z L ) ( 2 . 5 ) ∇ ( ⋅ ) \nabla(·) ∇ ( ⋅ ) 梯度 ，具体为∇ x f ( x ) \nabla_xf(x) ∇ x f ( x ) f ( x ) f(x) f ( x ) x x x (2.6) ∇ x f ( x ) = ( ∂ f ( x ) ∂ x 1 , ∂ f ( x ) ∂ x 2 , ⋯  , ∂ f ( x ) ∂ x n ) , x ∈ R n
\nabla_xf(x) = (\frac{\partial{f(x)}}{\partial{x_1}}, \frac{\partial{f(x)}}{\partial{x_2}}, \cdots, \frac{\partial{f(x)}}{\partial{x_n}}), x \in R^n
\tag{2.6}
∇ x f ( x ) = ( ∂ x 1 ∂ f ( x ) , ∂ x 2 ∂ f ( x ) , ⋯ , ∂ x n ∂ f ( x ) ) , x ∈ R n ( 2 . 6 ) ⨀ \bigodot ⨀ 点乘 ，表示了两个操作数的各个元素的分别相乘，前提是两个操作数的维度统一。
δ j l = ∑ k ( δ k l + 1 ∗ w k j l + 1 ∗ σ ′ ( z j l ) ) \delta^l_j = \sum_k(\delta^{l+1}_k*w^{l+1}_{kj}*\sigma^\prime(z^l_j)) δ j l = ∑ k ( δ k l + 1 ∗ w k j l + 1 ∗ σ ′ ( z j l ) ) (3.1) δ j l = ∂ C ∂ z j l = ∑ k ∂ C ∂ z k l + 1 ∗ ∂ z k l + 1 ∂ a j l ∗ ∂ a j l ∂ z j l = ∑ k δ k l + 1 ∗ ∂ ( w k j l + 1 ∗ a j l + b k l + 1 ) ∂ a j l ∗ σ ′ ( z j l ) = ∑ k ( δ k l + 1 ∗ w k j l + 1 ∗ σ ′ ( z j l ) )
\delta^l_j =
\frac{\partial{C}}{\partial{z^l_j}} =
\sum_k \frac{\partial C}{ \partial z^{l+1}_k} * \frac{ \partial{z^{l+1}_k} }{ \partial{a^l_j} } *
\frac{ \partial{a^l_j} }{ \partial{z^l_j} } \\
= \sum_k \delta^{l+1}_k * \frac{ \partial({w^{l+1}_{kj}} * a^l_j+b^{l+1}_k )}{ \partial{a^l_j} } * \sigma^\prime(z^l_j)
= \sum_k(\delta^{l+1}_k*w^{l+1}_{kj}*\sigma^\prime(z^l_j))
\tag{3.1}
δ j l = ∂ z j l ∂ C = k ∑ ∂ z k l + 1 ∂ C ∗ ∂ a j l ∂ z k l + 1 ∗ ∂ z j l ∂ a j l = k ∑ δ k l + 1 ∗ ∂ a j l ∂ ( w k j l + 1 ∗ a j l + b k l + 1 ) ∗ σ ′ ( z j l ) = k ∑ ( δ k l + 1 ∗ w k j l + 1 ∗ σ ′ ( z j l ) ) ( 3 . 1 )
同样的，为了表示方便，也多用矩阵形式表示，有：
(3.2) δ 1 l = δ 1 l + 1 ∗ w 11 l + 1 ∗ σ ′ ( z 1 l ) + δ 2 l + 1 ∗ w 21 l + 1 ∗ σ ′ ( z 1 l ) \delta^l_1 = \delta^{l+1}_1*w^{l+1}_{11}*\sigma^\prime(z^l_1)+
\delta^{l+1}_2*w^{l+1}_{21}*\sigma^\prime(z^l_1)
\tag{3.2}
δ 1 l = δ 1 l + 1 ∗ w 1 1 l + 1 ∗ σ ′ ( z 1 l ) + δ 2 l + 1 ∗ w 2 1 l + 1 ∗ σ ′ ( z 1 l ) ( 3 . 2 )
(3.3) δ 2 l = δ 1 l + 1 ∗ w 12 l + 1 ∗ σ ′ ( z 2 l ) + δ 2 l + 1 ∗ w 22 l + 1 ∗ σ ′ ( z 2 l )
\delta^l_2 = \delta^{l+1}_1*w^{l+1}_{12}*\sigma^\prime(z^l_2)+
\delta^{l+1}_2*w^{l+1}_{22}*\sigma^\prime(z^l_2)
\tag{3.3}
δ 2 l = δ 1 l + 1 ∗ w 1 2 l + 1 ∗ σ ′ ( z 2 l ) + δ 2 l + 1 ∗ w 2 2 l + 1 ∗ σ ′ ( z 2 l ) ( 3 . 3 )
(3.4) δ 3 l = δ 1 l + 1 ∗ w 13 l + 1 ∗ σ ′ ( z 3 l ) + δ 2 l + 1 ∗ w 23 l + 1 ∗ σ ′ ( z 3 l )
\delta^l_3 = \delta^{l+1}_1*w^{l+1}_{13}*\sigma^\prime(z^l_3)+
\delta^{l+1}_2*w^{l+1}_{23}*\sigma^\prime(z^l_3)
\tag{3.4}
δ 3 l = δ 1 l + 1 ∗ w 1 3 l + 1 ∗ σ ′ ( z 3 l ) + δ 2 l + 1 ∗ w 2 3 l + 1 ∗ σ ′ ( z 3 l ) ( 3 . 4 )
(3.5) δ 4 l = δ 1 l + 1 ∗ w 14 l + 1 ∗ σ ′ ( z 4 l ) + δ 2 l + 1 ∗ w 24 l + 1 ∗ σ ′ ( z 4 l )
\delta^l_4 = \delta^{l+1}_1*w^{l+1}_{14}*\sigma^\prime(z^l_4)+
\delta^{l+1}_2*w^{l+1}_{24}*\sigma^\prime(z^l_4)
\tag{3.5}
δ 4 l = δ 1 l + 1 ∗ w 1 4 l + 1 ∗ σ ′ ( z 4 l ) + δ 2 l + 1 ∗ w 2 4 l + 1 ∗ σ ′ ( z 4 l ) ( 3 . 5 )
(3.6) W l + 1 = ( w 11 l + 1 w 12 l + 1 w 13 l + 1 w 14 l + 1 w 21 l + 1 w 22 l + 1 w 23 l + 1 w 24 l + 1 )
W^{l+1} = \left(\begin{array}{ccc}
w^{l+1}_{11} & w^{l+1}_{12} & w^{l+1}_{13} & w^{l+1}_{14}\\
w^{l+1}_{21} & w^{l+1}_{22} & w^{l+1}_{23} & w^{l+1}_{24}
\end{array}\right)
\tag{3.6}
W l + 1 = ( w 1 1 l + 1 w 2 1 l + 1 w 1 2 l + 1 w 2 2 l + 1 w 1 3 l + 1 w 2 3 l + 1 w 1 4 l + 1 w 2 4 l + 1 ) ( 3 . 6 )
所以，有：(3.7) δ l = ( ( W l + 1 ) T ∗ δ l + 1 ) ⨀ σ ′ ( z l )
\delta^l = ((W^{l+1})^T * \delta^{l+1}) \bigodot \sigma^\prime(z^l)
\tag{3.7}
δ l = ( ( W l + 1 ) T ∗ δ l + 1 ) ⨀ σ ′ ( z l ) ( 3 . 7 )
∂ C ∂ w j k l = a k l − 1 ∗ δ j l \frac{\partial{C}}{\partial{w^l_{jk}}} = a^{l-1}_k*\delta^l_j ∂ w j k l ∂ C = a k l − 1 ∗ δ j l (4.1) ∂ C ∂ w j k l = ∂ C ∂ z j l ∗ ∂ z j l ∂ w j k l = a k l − 1 ∗ δ j l = a k l − 1 ∗ ∑ k ( δ k l + 1 ∗ w k j l + 1 ∗ σ ′ ( z j l ) )
\frac{\partial{C}}{\partial{w^l_{jk}}} = \frac{\partial{C}}{\partial{z^l_{j}}} *
\frac{\partial{z^l_j}}{\partial{w^l_{jk}}} =
a^{l-1}_k*\delta^l_j=a^{l-1}_k *
\sum_k(\delta^{l+1}_k*w^{l+1}_{kj}*\sigma^\prime(z^l_j))
\tag{4.1}
∂ w j k l ∂ C = ∂ z j l ∂ C ∗ ∂ w j k l ∂ z j l = a k l − 1 ∗ δ j l = a k l − 1 ∗ k ∑ ( δ k l + 1 ∗ w k j l + 1 ∗ σ ′ ( z j l ) ) ( 4 . 1 )
由此可以看出，∂ C ∂ w j k l \frac{\partial{C}}{\partial{w^l_{jk}}} ∂ w j k l ∂ C σ ′ ( z j l ) \sigma^\prime(z^l_j) σ ′ ( z j l ) ∂ C ∂ w j k l \frac{\partial{C}}{\partial{w^l_{jk}}} ∂ w j k l ∂ C 梯度消散 的问题(The problem of gradient vanishing)。ReLU (Rectified Linear Units)函数，R e L U = m a x ( 0 , x ) ReLU = max(0, x) R e L U = m a x ( 0 , x ) x > 0 x > 0 x > 0
∂ C ∂ b j l = δ j l \frac{\partial{C}}{\partial{b^l_j}} = \delta^l_j ∂ b j l ∂ C = δ j l (5.1) ∂ C ∂ b j l = ∂ C ∂ z j l ∗ ∂ z j l ∂ b j l = δ j l ∗ 1 = δ j l
\frac{\partial{C}}{\partial{b^l_j}} =
\frac{\partial{C}}{\partial{z^l_j}} *
\frac{\partial{z^l_j}}{\partial{b^l_j}} = \delta^l_j*1 = \delta^l_j
\tag{5.1}
∂ b j l ∂ C = ∂ z j l ∂ C ∗ ∂ b j l ∂ z j l = δ j l ∗ 1 = δ j l ( 5 . 1 )
(6.1) w l : = w l − η ∗ δ l ∗ ( a l − 1 ) T
w^l := w^l-\eta*\delta^l*(a^{l-1})^T
\tag{6.1}
w l : = w l − η ∗ δ l ∗ ( a l − 1 ) T ( 6 . 1 )
(6.2) b l : = b l − η ∗ δ l
b^l := b^l-\eta*\delta^l
\tag{6.2}
b l : = b l − η ∗ δ l ( 6 . 2 )
我们在这篇文章中推导了BP算法的公式，我们接下来将在下一篇文章《基于numpy和python的反向传播算法的实现与分析》 中利用numpy和python实现BP算法，有兴趣的朋友请移步。
《基于numpy和python的反向传播算法的实现与分析》
