softmax求导的计算

本文参考于(https://zhuanlan.zhihu.com/p/105758059）
大家可以参考上述链接，更加详细。

softmax之前的输入为
$z = [z_1,z_2,...,z_n]$
经过softmax之后，
$a_i = \frac{e^{z_i}}{\sum_{k=1}^{n}e^{z_k}}$
可得a向量 $a = [\frac{e^{z_1}}{\sum_{k=1}^{n}e^{z_k}},\frac{e^{z_2}}{\sum_{k=1}^{n}e^{z_k}},...,\frac{e^{z_n}}{\sum_{k=1}^{n}e^{z_k}}]$
目标向量为
y = [0,0,0,...,1,..0]，假设 $y_j=1$ 其余均为0
损失函数为交叉熵损失
$L = -\sum_{i=1}^{n}y_i*lna_i$ ,又其他均为0，故可以简写成 $L = -y_j*lna_j = -lna_j$

目标是标量L对向量z求导， $\frac{\partial L}{\partial Z} = \frac{\partial L}{\partial a}*\frac{\partial a}{\partial z}$

1 求 $\frac{\partial L}{\partial a}$

由 $L = -lna_j$ 得，loss只与a_j有关
$\frac{\partial L }{\partial a} = [0,0,...,-\frac{1}{a_j},..0]$

2 求 $\frac{\partial a}{\partial z}$

a是一个向量，z是一个向量， $\frac{\partial a}{\partial z} = \left[ \begin{matrix} \frac{\partial a_1}{\partial z_1} & \frac{\partial a_1}{\partial z_2} & \cdots & \frac{\partial a_1}{\partial z_n}\\ \frac{\partial a_2}{\partial z_1} & \frac{\partial a_2}{\partial z_2} & \cdots & \frac{\partial a_2}{\partial z_n}\\ \vdots & \vdots & \vdots & \vdots \\ \frac{\partial a_n}{\partial z_1} & \frac{\partial a_n}{\partial z_2} & \cdots & \frac{\partial a_n}{\partial z_n}\\ \end{matrix} \right]$
由于 $\frac{\partial l}{\partial a}$ 只有第j列不为0，我们只需要求 $\frac{\partial a}{\partial z}$ 的第行,即 $\frac{\partial a_j}{\partial z}$
$\frac{\partial L}{\partial Z} = -\frac{1}{a_j}*\frac{\partial a_j}{\partial Z}$ ，其中 $a_j = \frac{e^{z_j}}{\sum_{i=1}^{n}e^{z_k}}$

当 $i \not= j$
$\frac{\partial a_j}{\partial z_i} = \frac{0-e^{z_j}*e^{z_i}}{(\sum_{i=1}^{n}e^{z_k})^2} = -a_j*a_i$
$\frac{\partial L}{\partial z_i} = -\frac{1}{a_j}*\frac{\partial a_j}{\partial z} = -\frac{1}{a_j}*(-a_j*a_i) = a_i$
当 $i = j$
$\frac{\partial a_j}{\partial z_j} = \frac{e^{z_j}*\sum_{i=1}^{n}e^{z_k}-e^{z_j}*e^{z_j}}{(\sum_{i=1}^{n}e^{z_k})^2} = a_j- a_j^2$
$\frac{\partial L}{\partial z_j} = (a_j-a_j^2)*(-\frac{1}{a_j}) = a_j-1$