Neural Network and DeepLearning (1.2)使用神经网络识别手写数字
1.Learning with gradient descent(使用梯度下降法进行学习)
cost function(代价函数)
训练神经网络的目的是找到能最小化二次代价函数C(w,b)的权重和偏置。梯度下降:
when we move the ball a small amount Δv1 in the v1 direction, and a small amount Δv2 in the v2 direction. Calculus tells us that C changes as follows:
defined the gradient of C to be the vector of partial derivatives:
the expression for ΔC can be rewritten as:
choose Δv as:
then:
this guarantees that ΔC≤0, C will always decrease, never increase.
hat is, we'll use Equation (10) to compute a value for Δv, then move the ball's position v by that amount:
Then we'll use this update rule again, to make another move. If we keep doing this, over and over, we'll keep decreasing C until - we hope - we reach a global minimum.
stochastic gradient descent(随机梯度下降):
works by picking out a randomly chosen mini-batch of training inputs, and training with those:
随机记梯度下降通过随机的选取并训练输入的小批量数据来工作,其中两个求和符号是在当前小批量数据中的所有训练样本Xj,上进行的。
然后我们再挑选另一随机选定的小批量数据去训练。直到我们用完了所有的训练输入,这被称为完成了一个训练迭代期(epoch)。然后我们就会开始一个新的训练迭代期。
备:我们不能提前知道训练数据量的情况下,舍弃1/n和1/m是有效的。
小批量数据的大小设置为1时,成为online学习,或递增学习