jrae源代码解析（二）

本文细述上文引出的RAECost和SoftmaxCost两个类。

SoftmaxCost

我们已经知道。SoftmaxCost类在给定features和label的情况下（超參数给定），衡量给定权重（hidden×catSize）的误差值cost,并指出当前的权重梯度。看代码。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
@Override
    public double valueAt(double[] x)
    {
        if( !requiresEvaluation(x) )
            return value;
        int numDataItems = Features.columns;
         
        int[] requiredRows = ArraysHelper.makeArray(0, CatSize-2);
        ClassifierTheta Theta = new ClassifierTheta(x,FeatureLength,CatSize);
        DoubleMatrix Prediction = getPredictions (Theta, Features);
         
        double MeanTerm = 1.0 / (double) numDataItems;
        double Cost = getLoss (Prediction, Labels).sum() * MeanTerm;
        double RegularisationTerm = 0.5 * Lambda * DoubleMatrixFunctions.SquaredNorm(Theta.W);
         
        DoubleMatrix Diff = Prediction.sub(Labels).muli(MeanTerm);
        DoubleMatrix Delta = Features.mmul(Diff.transpose());
     
        DoubleMatrix gradW = Delta.getColumns(requiredRows);
        DoubleMatrix gradb = ((Diff.rowSums()).getRows(requiredRows));
         
        //Regularizing. Bias does not have one.
        gradW = gradW.addi(Theta.W.mul(Lambda));
         
        Gradient = new ClassifierTheta(gradW,gradb);
        value = Cost + RegularisationTerm;
        gradient = Gradient.Theta;
        return value;
    }<br><br>public DoubleMatrix getPredictions (ClassifierTheta Theta, DoubleMatrix Features)<br>    {<br>        int numDataItems = Features.columns;<br>        DoubleMatrix Input = ((Theta.W.transpose()).mmul(Features)).addColumnVector(Theta.b);<br>        Input = DoubleMatrix.concatVertically(Input, DoubleMatrix.zeros(1,numDataItems));<br>        return Activation.valueAt(Input); <br>    }

 是个典型的2层神经网络，没有隐层，首先依据features预測labels，预測结果用softmax归一化，然后依据误差反向传播算出权重梯度。

此处添加200字。

这个典型的2层神经网络，label为一列向量，目标label置1，其余为0；转换函数为softmax函数，输出为每一个label的概率。

计算cost的函数为getLoss。如果目标label的预測输出为p∗，则每一个样本的cost也即误差函数为：

cost=E(p∗)=−log(p∗)

依据前述的神经网络后向传播算法，我们得到(j为目标label时，否则为0)：

∂E∂wij=∂E∂pj∂hj∂netjxi=−1pjpj(1−pj)xi=−(1−pj)xi=−(labelj−pj)featurei

因此我们便理解了以下代码的含义：

1
DoubleMatrix Delta = Features.mmul(Diff.transpose());

 

RAECost

先看实现代码：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
@Override
    public double valueAt(double[] x)
    {
        if(!requiresEvaluation(x))
            return value;
         
        Theta Theta1 = new Theta(x,hiddenSize,visibleSize,dictionaryLength);
        FineTunableTheta Theta2 = new FineTunableTheta(x,hiddenSize,visibleSize,catSize,dictionaryLength);
        Theta2.setWe( Theta2.We.add(WeOrig) );
         
        final RAEClassificationCost classificationCost = new RAEClassificationCost(
                catSize, AlphaCat, Beta, dictionaryLength, hiddenSize, Lambda, f, Theta2);
        final RAEFeatureCost featureCost = new RAEFeatureCost(
                AlphaCat, Beta, dictionaryLength, hiddenSize, Lambda, f, WeOrig, Theta1);
     
        Parallel.For(DataCell,
            new Parallel.Operation<LabeledDatum<Integer,Integer>>() {
                public void perform(int index, LabeledDatum<Integer,Integer> Data)
                {
                    try {
                        LabeledRAETree Tree = featureCost.Compute(Data);
                        classificationCost.Compute(Data, Tree);                
                    } catch (Exception e) {
                        System.err.println(e.getMessage());
                    }
                }
        });
         
        double costRAE = featureCost.getCost();
        double[] gradRAE = featureCost.getGradient().clone();
             
        double costSUP = classificationCost.getCost();
        gradient = classificationCost.getGradient();
             
        value = costRAE + costSUP;
        for(int i=0; i<gradRAE.length; i++)
            gradient[i] += gradRAE[i];
         
        System.gc();    System.gc();
        System.gc();    System.gc();
        System.gc();    System.gc();
        System.gc();    System.gc();
         
        return value;
    }

cost由两部分组成，featureCost和classificationCost。程序遍历每一个样本，用featureCost.Compute(Data)生成一个递归树，同一时候累加cost和gradient。然后用classificationCost.Compute(Data, Tree)依据生成的树计算并累加cost和gradient。因此关键类为RAEFeatureCost和RAEClassificationCost。

RAEFeatureCost类在Compute函数中调用RAEPropagation的ForwardPropagate函数生成一棵树。然后调用BackPropagate计算梯度并累加。详细的算法过程。下一章分解。

posted @ 2016-04-01 10:49  mfrbuaa  阅读(286)  评论(0)  编辑  收藏  举报