2、Vectorization
这个实验其实主要是对于sparseAutoencoderCost.m向量化，但是由于之前已经进行了向量化，所以这其实没什么内容（之前其实感觉向量化，挺难的，但是第一个实验就向量化是有难度的，不过这个实验之前有很多向量化的提示）。就是自编码的结构参数，按照文档改一下，下载给的帮助代码。
- 问题：
1）
2）
3）
4）
5）
% Change the filenames if you've saved the files under different names
% On some platforms, the files might be saved as
% train-images.idx3-ubyte / train-labels.idx1-ubyte
images = loadMNISTImages('train-images-idx3-ubyte');
labels = loadMNISTLabels('train-labels-idx1-ubyte');
% We are using display_network from the autoencoder code
display_network(images(:,1:100)); % Show the first 100 images
disp(labels(1:10));
下载MNIST数据集，和帮助导入MNIST的函数http://ufldl.stanford.edu/wiki/resources/mnistHelper.zip.
基本就是没什么内容了，然后自己根据前一节的函数sampleIMAGES.m，改写一下train.m 就可以了。
train.m
clear;clc;close all;
%% CS294A/CS294W Programming Assignment Starter Code
% Instructions
% ------------
%
% This file contains code that helps you get started on the
% programming assignment. You will need to complete the code in sampleIMAGES.m,
% sparseAutoencoderCost.m and computeNumericalGradient.m.
% For the purpose of completing the assignment, you do not need to
% change the code in this file.
%
%%======================================================================
%% STEP 0: Here we provide the relevant parameters values that will
% allow your sparse autoencoder to get good filters; you do not need to
% change the parameters below.
visibleSize = 28*28; % number of input units
hiddenSize = 196; % number of hidden units
sparsityParam = 0.1; % desired average activation of the hidden units.
% (This was denoted by the Greek alphabet rho, which looks like a lower-case "p",
% in the lecture notes).
% lambda = 0;
lambda = 3e-3; % weight decay parameter
% beta = 0;
beta = 3; % weight of sparsity penalty term
%从http://deeplearning.stanford.edu/wiki/index.php/Using_the_MNIST_Dataset获得的代码。
%下面这两个函数还要下载对应的
% Change the filenames if you've saved the files under different names
% On some platforms, the files might be saved as
% train-images.idx3-ubyte / train-labels.idx1-ubyte
images = loadMNISTImages('train-images-idx3-ubyte');
labels = loadMNISTLabels('train-labels-idx1-ubyte');
% We are using display_network from the autoencoder code
display_network(images(:,1:100)); % Show the first 100 images
disp(labels(1:10));
set(gcf,'NumberTitle','off');
set(gcf,'Name','显示MNIST前100个数据');
%%======================================================================
%% STEP 1:
%下面为根据本次试验修改的读取patches
%patches = first 10000 images from the MNIST dataset
%
numpatches = 10000;
patches = zeros(visibleSize, numpatches);
for imageNum=1:numpatches
patches(:,imageNum)=reshape(images(:,imageNum),visibleSize,1);
end
%%======================================================================
%% STEP 2: Implement sparseAutoencoderCost
%
% You can implement all of the components (squared error cost, weight decay term,
% sparsity penalty) in the cost function at once, but it may be easier to do
% it step-by-step and run gradient checking (see STEP 3) after each step. We
% suggest implementing the sparseAutoencoderCost function using the following steps:
%
% (a) Implement forward propagation in your neural network, and implement the
% squared error term of the cost function. Implement backpropagation to
% compute the derivatives. Then (using lambda=beta=0), run Gradient Checking
% to verify that the calculations corresponding to the squared error cost
% term are correct.
%
% (b) Add in the weight decay term (in both the cost function and the derivative
% calculations), then re-run Gradient Checking to verify correctness.
%
% (c) Add in the sparsity penalty term, then re-run Gradient Checking to
% verify correctness.
%
% Feel free to change the training settings when debugging your
% code. (For example, reducing the training set size or
% number of hidden units may make your code run faster; and setting beta
% and/or lambda to zero may be helpful for debugging.) However, in your
% final submission of the visualized weights, please use parameters we
% gave in Step 0 above.
theta = initializeParameters(hiddenSize, visibleSize);
[costBegin, grad] = sparseAutoencoderCost(theta, visibleSize, hiddenSize, lambda, ...
sparsityParam, beta, patches);
%%======================================================================
%下面这个为检查函数computeNumericalGradient和函数sparseAutoencoderCost的代码
%检查完毕，就可以注释掉，不用再运行
%{
%% STEP 3: Gradient Checking
%
% Hint: If you are debugging your code, performing gradient checking on smaller models
% and smaller training sets (e.g., using only 10 training examples and 1-2 hidden
% units) may speed things up.
% First, lets make sure your numerical gradient computation is correct for a
% simple function. After you have implemented computeNumericalGradient.m,
% run the following:
%检查梯度的函数 computeNumericalGradient.m，开始编写computeNumericalGradient的时候开始验证
%然后就不用再检查
%checkNumericalGradient();
% Now we can use it to check your cost function and derivative calculations
% for the sparse autoencoder.
%下面这句语法上理解,其实一开始不懂，后面问了王鑫师姐，就完全懂了。
%下面的函数有两个参数，用逗号作为间隔，前一个参数是函数，后一个就是一个常数
%看 computeNumericalGradient(J, theta) 的函数定义，前一个参数J就是一个函数。
%这里就是用一个匿名函数作为参数，然后定义的x作为调用函数中的某一个参数变量
%所以后面如果调用这个参数的函数，那么送入的变量就会替换掉x
%搞懂这块真不容易呀，不过弄懂了是很爽呀，真就看懂了
%不过当年MATLAB也不熟，看这块代码确实是有点费劲
numgrad = computeNumericalGradient( @(x) sparseAutoencoderCost(x, visibleSize, ...
hiddenSize, lambda, ...
sparsityParam, beta, ...
patches), theta);
%下面就直接输出和theta[3289,1]维数一样的两个向量
% Use this to visually compare the gradients side by side
disp([numgrad grad]);
% Compare numerically computed gradients with the ones obtained from backpropagation
diff = norm(numgrad-grad)/norm(numgrad+grad);
fprintf('Norm of the difference between numerical and analytical gradient (should be < 1e-9)\n\n');
disp(diff); % Should be small. In our implementation, these values are
% usually less than 1e-9.
% When you got this working, Congratulations!!!
%%======================================================================
%}
%% STEP 4: After verifying that your implementation of
% sparseAutoencoderCost is correct, You can start training your sparse
% autoencoder with minFunc (L-BFGS).
% Randomly initialize the parameters
theta = initializeParameters(hiddenSize, visibleSize);
% Use minFunc to minimize the function
addpath minFunc/
options.Method = 'lbfgs'; % Here, we use L-BFGS to optimize our cost
% function. Generally, for minFunc to work, you
% need a function pointer with two outputs: the
% function value and the gradient. In our problem,
% sparseAutoencoderCost.m satisfies this.
options.maxIter = 400; % Maximum number of iterations of L-BFGS to run
options.display = 'on';
[opttheta, costEnd] = minFunc( @(p) sparseAutoencoderCost(p, ...
visibleSize, hiddenSize, ...
lambda, sparsityParam, ...
beta, patches), ...
theta, options);
%%======================================================================
%% STEP 5: Visualization
W1 = reshape(opttheta(1:hiddenSize*visibleSize), hiddenSize, visibleSize);
figure;
display_network(W1', 12);
set(gcf,'NumberTitle','off');
set(gcf,'Name','稀疏自编码后的第一层的权系数');
print -djpeg weights.jpg % save the visualization to a file
看这个输出的cost，也可以看出，模型越大，cost越大。
实验结果为如下，这些笔画。
