Derivative of Softmax Loss Function

Derivative of Softmax Loss Function

A softmax classifier:

\[p_j = \frac{\exp{o_j}}{\sum_{k}\exp{o_k}} \]

It has been used in a loss function of the form

\[L = - \sum_{j} y_j \log p_j \]

where \(o\) is a vector. We need the derivative of \(L\) with respect to \(o\). We can get the partial of \(o_i\) :

\[\frac{\partial{p_j}}{\partial{o_i}} = p_i (1-p_i), \quad i = j \\ \frac{\partial{p_j}}{\partial{o_i}} = - p_i p_j, \quad i \ne j \]

Hence the derivative of Loss with respect to \(o\) is:

\[\begin{align} \frac{\partial{L}}{\partial{o_i}} & = - \sum_k y_k \frac{\partial{\log p_k}}{\partial{o_i}} \\ & = - \sum_k y_k \frac{1}{p_k} \frac{\partial{p_k}}{\partial{o_i}} \\ & = -y_i(1-p_i) - \sum_{k\ne i} y_k \frac{1}{p_k} (-p_kp_i) \\ & = -y_i + y_i p_i + \sum_{k\ne i} y_k p_i \\ & = p_i (\sum_k y_k) - y_i \\ \end{align} \]

Given that \(\sum_k y_k = 1\) as \(y\) is a vector with only one non-zero element, which is 1. By other words, this is a classification problem.

\[\frac{\partial L}{\partial o_i} = p_i - y_i \]

Reference

Derivative of Softmax loss function