DeepLearnToolbox使用总结
转自:http://blog.csdn.net/lifeitengup/article/details/10219075
GitHub链接:DeepLearnToolbox
DeepLearnToolbox
A Matlab toolbox for Deep Learning.
Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data. It is inspired by the human brain's apparent deep (layered, hierarchical) architecture. A good overview of the theory of Deep Learning theory is Learning Deep Architectures for AI
Directories included in the toolbox
NN/
- A library for Feedforward Backpropagation Neural Networks
CNN/
- A library for Convolutional Neural Networks
DBN/
- A library for Deep Belief Networks
SAE/
- A library for Stacked Auto-Encoders
CAE/
- A library for Convolutional Auto-Encoders
util/
- Utility functions used by the libraries
data/
- Data used by the examples
tests/
- unit tests to verify toolbox is working
For references on each library check REFS.md
Setup
- Download.
- addpath(genpath('DeepLearnToolbox'));
Windows下把文件夹加入 path 即可
- %LiFeiteng
- path = pwd;
- files = dir(path);
- for i = 1:length(files)
- if files(i).isdir
- file = files(i).name;
- addpath([path '/' file])
- disp(['add ' file ' to path!'])
- end
- end
我不打算解析代码,想从代码里面学算法是stupid的;有相应的论文,readlist,talk等可以去学习。
DeepLearnToolbox单隐藏层NN的优化策略:mini-Batch SGD
- function [nn, L] = nntrain(nn, train_x, train_y, opts, val_x, val_y)
- %NNTRAIN trains a neural net
- % [nn, L] = nnff(nn, x, y, opts) trains the neural network nn with input x and
- % output y for opts.numepochs epochs, with minibatches of size
- % opts.batchsize. Returns a neural network nn with updated activations,
- % errors, weights and biases, (nn.a, nn.e, nn.W, nn.b) and L, the sum
- % squared error for each training minibatch.
- assert(isfloat(train_x), 'train_x must be a float');
- assert(nargin == 4 || nargin == 6,'number ofinput arguments must be 4 or 6')
- loss.train.e = [];
- loss.train.e_frac = [];
- loss.val.e = [];
- loss.val.e_frac = [];
- opts.validation = 0;
- if nargin == 6
- opts.validation = 1;
- end
- fhandle = [];
- if isfield(opts,'plot') && opts.plot == 1
- fhandle = figure();
- end
- m = size(train_x, 1);
- batchsize = opts.batchsize;
- numepochs = opts.numepochs;
- numbatches = m / batchsize;
- assert(rem(numbatches, 1) == 0, 'numbatches must be a integer');
- L = zeros(numepochs*numbatches,1);
- n = 1;
- for i = 1 : numepochs
- tic;
- kk = randperm(m);
- for l = 1 : numbatches
- batch_x = train_x(kk((l - 1) * batchsize + 1 : l * batchsize), :);
- %Add noise to input (for use in denoising autoencoder)
- if(nn.inputZeroMaskedFraction ~= 0)
- batch_x = batch_x.*(rand(size(batch_x))>nn.inputZeroMaskedFraction);
- end
- batch_y = train_y(kk((l - 1) * batchsize + 1 : l * batchsize), :);
- nn = nnff(nn, batch_x, batch_y);
- nn = nnbp(nn);
- nn = nnapplygrads(nn);
- L(n) = nn.L;
- n = n + 1;
- end
- t = toc;
- if ishandle(fhandle)
- if opts.validation == 1
- loss = nneval(nn, loss, train_x, train_y, val_x, val_y);
- else
- loss = nneval(nn, loss, train_x, train_y);
- end
- nnupdatefigures(nn, fhandle, loss, opts, i);
- end
- disp(['epoch ' num2str(i) '/' num2str(opts.numepochs) '. Took ' num2str(t) ' seconds' '. Mean squared error on training set is ' num2str(mean(L((n-numbatches):(n-1))))]);
- nn.learningRate = nn.learningRate * nn.scaling_learningRate;
- end
- end
1.不管是在 nntrain、nnbp还是nnapplygrads中我都没看到 对算法收敛性的判断,
而且在实测的过程中 有观察到 epoch过程中 mean-squared-error有 下降-上升-下降 的走势——微小抖动在SGD中 算是正常
多数还都是在下降(epoch我一般设为 10-40,这个值可能偏小;Hinton 06 science的文章代码记得epoch了200次,我跑了3天也没跑完)
在SAE/CNN等中 也没看到收敛性的判断。
2.CAE 没有完成
3.dropout的优化策略也可以选择
我测试了 SAE CNN等,多几次epoch(20-30),在MNIST上正确率在 97%+的样子。
其实cost-function 可以有不同的选择,如果使用 UFLDL的优化方式(固定的优化方法,传入cost-function的函数句柄),在更改cost-function上会更自由。
可以改进的地方:
1. mini-Bathch SGD算法 增加收敛性判断
2.增加 L-BFGS/CG等优化算法
3.完善CAE等
4.增加min KL-熵的 Sparse Autoencoder等
5.优化算法增加对 不同cost-function的支持