基于Excel的神经网络工具箱（之一）——DNN神经网络数据结构的算法实现
近三个月一直在研究ANN，为了真正深入地理解ANN的细节，我摈弃了那种直接调用算法库构建ANN的做法，而是选择了一条布满荆棘的小路，从数据结构和算法出发，从0开始一步步构建ANN的结构大厦。因为最熟悉的“开发工具”就是VBA（才疏学浅，其他的语言除了C以外都学艺不精），因此就从三个月前的国庆长假开始正式启动了开发一个Excel的神经网络工具箱的工作，现在已经完成了DNN的部分，可以在Excel中建立一个DNN，并提供了三种不同的训练算法进行训练，最后可以将DNN保存并发布。未来还将增加CNN、RNN的内容。这里记录的东西都是一些学习心得，附上部分源码，以跟同好共同学习
我们一般所谓的DNN其实名字应该叫做全连接多层感知器，每个感知器模拟大脑里神经元的工作方式，通过“树突”上的突触端接受化学电信号，经过“轴突”的突触端将信号传递到下一个神经元。神经元传递信号与否取决于神经元本身是否被输入信号激活，而神经元是否被激活又取决于它本身的特性与输入信号的模式。我个人的理解，神经元就像一个选择性导通的电阻，当接收到“合适”的输入信号时就导通（激活），然后传递信号到下个神经元，否则就不导通。
人工神经网络就是由许多模拟上述神经元工作模式的人工模拟神经元组成的，每一个神经元被称为“感知器”，它其实是一类包含大量输入参数的函数，这个函数其实就是将输入向量的每一个分量乘以一个权值后加总，再加上一个偏置（bias）后经一次非线性变换后再输出。下面这张图大家肯定都看熟了：
将许多感知器单元分层互联起来，使第一层的每个单元的输入来自系统的输入，而后续各层的输入分别来自它的前一层的输出信号，这样就构建了一个完整的MLP网络（Multi-Level Perceptron多层感知器，不过貌似现在大家比较喜欢使用DNN这个名字，当然，广义的DNN还可以有其他更多的结构）。
基本数据结构
言归正传，我需要在Excel的VBA里建立这样一个MLP的数据结构，并且实现这个数据结构上的前向传播和反向传播训练算法，其实这个数据结构还是很简单的，首先定义每个单元，每个单元内部仅包含两部分内容，一个是双精度浮点型数据偏置bias，另一个就是权值，由于权值数量较多所以可以用数组表示，因此最简单的NNUnit就可以如下定义
Public Type NNUnit
b As Double ' bias value of unit
w(1 to w_size) As Double ' weights of unit
End Type
其实还有更简单的定义方法，那就是将b定义为w(0)，那样连b都省掉了，这样做有个好处就是更加方便后续进行反向传播时进行矩阵计算
定义好Unit之后可以直接定义Layer（层）以及整个网络的数据结构：
Public Type layer
units() As NNUnit ' all units in the layer
ActiF As Byte ' unit activation function type, identified with UTF constants
End Type
Public Type NN
inSize As Long ' size of input vector
hLayer() As layer ' array of hidden layers
hLayerCount As Long ' count of hidden layers, in order to facilitate hidden layer location
OutLayer As layer ' the output layer of the net
End Type
网络数据结构
去掉不关键的部分后，网络的结构和层的结构也很简单，ActiF定义了单元的激活函数，因为可能在定义网络的时候需要用到不同的激活函数。整个数据结构是一个级联的数组结构，即一个网络包含若干层，而每一层都包含若干单元。大家可以看到我在NN的数据结构中显式地声明了一个输出层以及一个包含若干隐藏层的Layer型数组，实际上更简单的做法是直接声明一个Layer型数组包含所有层，这样还能够简化前向和后向传播算法的代码。
现在数据结构已经搭好了，为了让这个数据结构有内容，我们可以创建并初始化任意一个网络了，我创建了两个基本函数，NewNN()和NNAddLayer()，分别用来创建一个单层网络以及用来向一个网络中添加感知器层。
下面是新建网络函数，接受两个参数
Public Function newNN(name As String, inSize As Long, outSize As Long, outF As Byte) As NN
Dim net As NN, l As layer
Dim i As Long
With l
ReDim .units(1 To outSize)
For i = 1 To outSize
.units(i) = newUnit(inSize) ' insert empty units to the output layer
.utf = outF
Next
End With
With net
.inSize = inSize
.OutLayer = l
.hLayerCount = 0
End With
newNN = net
End Function
而NNAddLayer函数在任意网络的输出层前插入一个新的层，接受的参数是新层的单元数量，这时候需要注意，新插入的层可能会打乱输出层的单元连接，因此需要将输出层的所有单元重置一下，比如，原来输出层的前一层有10个单元，输出层有3个单元，每个单元有10个权值与前一层连接，现在插入一个新层，该层有7个单元，这时候应该将新插入的层的每个单元置10个权值与前一个单元连接，而输出层的每个单元的权值应该改成7个，以便与新插入的层全连接。这个过程通过下面函数实现，其中调用了一些基本单元操作函数，就不详细展开了：
Public Function NNAddLayer(net As NN, ByVal s As Long, f As Byte) As NN
' this function adds/inserts one layer before the output layer, returns the new net
' size of the layer should be provided as argument
' net is the neuron net to be modified
' s is the size (number of NNUnit units) of the layer, number of weights are defined
' f is used to define the unit transfering function of the added layer
Dim curUnit As NNUnit
Dim outUnitSize As Long
Dim outUnitCount As Long
Dim curLayerInput As Long ' size of current layer input
Dim i As Long, j As Long
With net
' insert new layer after the last hidden layer
If .hLayerCount = 0 Then ' insert a new layer before the out layer
curLayerInput = .inSize
.hLayerCount = 1
ReDim .hLayer(1 To 1)
ReDim .hLayer(1).units(1 To s)
Else ' insert a new layer before the outlayer and after tha last hidden layer
curLayerInput = UBound(.hLayer(.hLayerCount).units)
.hLayerCount = .hLayerCount + 1
ReDim Preserve .hLayer(1 To .hLayerCount)
ReDim .hLayer(.hLayerCount).units(1 To s)
End If
For i = 1 To s
curUnit = newUnit(curLayerInput)
.hLayer(.hLayerCount).units(i) = curUnit
Next
.hLayer(.hLayerCount).utf = f
' re-define the output layer if new layer size differs from previous last layer
outUnitSize = .OutLayer.units(1).w_size
outUnitCount = UBound(.OutLayer.units)
If outUnitSize <> s Then
For i = 1 To outUnitCount
.OutLayer.units(i) = resizeUnit(.OutLayer.units(i), s)
Next
End If
End With
NNAddLayer = net
End Function
通过调用上面两个函数，一个完整的DNN结构就可以在Excel的内存中创建出来。下一篇，我将分享这个DNN的前向传播和反向传播算法的实现，同时，由于Excel本身就是一个非常不错的数据处理工具，我也将分享工具箱中的一个数据处理模块，可以很容易地从Excel的单元格中创建可以用于DNN的训练的数据集，这样有数据分析需要的同学就不可以直接将数据导入Excel处理后便很方便地用于神经网络的训练了。
实际算法源码
在ANN Toolbox中，完整的DNN定义为了兼容CNN卷积网络，实际上每个单元不仅仅可以以一维数组形式存储，同时还能存储为二维或三维数组，用于卷积网络的卷积层，同时，还实现了不同的激活函数。另外，为了便于计算每一层的梯度，又定义了两个专门的数据结构，MultiArr和TripleArr，MultiArr其实就是“数组的数组”，而TripleArr则是MultiArr的数组。完整的Unit和Layer定义的代码如下：
基本数据结构及Unit和layer的定义：
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'' Data Structure for artifitial neural networks
'' 1, Common public structure used for multiple arrays
' a data type that used to store multiple arrays, usful to store delta of units in each layer _
' also useful for store all outputs of each layer of a net
Public Type MultiArr
d() As Double
End Type
Public Type TripleArr 'triple arr structure
m() As MultiArr
End Type
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'' 2, Define Structure NN Unit, class properties of perceptron units
Public Type NNUnit ' the basic element of neural network, to reduce resource consumption, thus type is to be as simple as possible
b As Double ' bias value of unit
w() As Double ' weights of unit
End Type
'' 2.2, define layer structure
Public Type layer
layerType As Byte ' type of layer identifies the connection type of layer
units() As NNUnit ' all units in the layer
unitSize1 As Long ' count of weights in dimension 1: number of rows
unitSize2 As Long ' count of weights in dimension 2: number of columns
unitSize3 As Long ' count of weights in dimension 3: number of levels
unitDim As Byte ' dimension of weight array
utf As Byte ' unit activation function type, identified with UTF constants
outSize1 As Long ' dimension 1 size of output array: number of rows
outSize2 As Long ' dimension 2 size of output array: number of columns
outSize3 As Long ' dimension 3 size of output array: number of levels
End Type
基本的Unit和Layer创建、初始化和激活函数：
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'' functions of neural networks
Private Function newUnit(ByVal ws As Long, Optional ByVal ws_2 As Long = 0, Optional ByVal ws_3 As Long = 0) As NNUnit
' create a new NNUnit unit, returns created NNUnit
' the unit weight array can be created as 1/2/3 dimensional, d as dimension is a passed parameter
Dim NU As NNUnit
With NU
If ws_2 = 0 Then ' unit is 1-D
ReDim .w(1 To ws)
ElseIf ws_3 = 0 Then ' unit is 2-D
ReDim .w(1 To ws, 1 To ws_2)
Else ' unit is 3-D
ReDim .w(1 To ws, 1 To ws_2, 1 To ws_3)
End If
.b = 0
End With
newUnit = NU
End Function
Private Function initUnit(unit As NNUnit, Optional ByVal wType As Long = wTypeRand, _
Optional ByVal lLimit As Double = -0.05, Optional ByVal uLimit As Double = 0.05, Optional ByVal d As Byte = 1) As NNUnit
' initialize the weights of the sigmoid unit according to given weight type
' d as dimension of the unit weights
Dim i As Long, j As Long, k As Long, m As Long, n As Long, o As Long
Dim rndRange As Double, rndCentre As Double
'On Error Resume Next
rndRange = uLimit - lLimit
With unit
.b = 0
If wType = wTypeRand Then .b = Rnd() * rndRange + lLimit 'randomize bias value
Select Case d
Case 1
For i = 1 To UBound(.w, 1)
.w(i) = 0
If wType = wTypeRand Then .w(i) = Rnd() * rndRange + lLimit 'randomize weights
Next
Case 2
For i = 1 To UBound(.w, 1)
For j = 1 To UBound(.w, 2)
.w(i, j) = 0
If wType = wTypeRand Then .w(i, j) = Rnd() * rndRange + lLimit 'randomize weights
Next
Next
Case 3
For i = 1 To UBound(.w, 1)
For j = 1 To UBound(.w, 2)
For k = 1 To UBound(.w, 3)
.w(i, j, k) = 0
If wType = wTypeRand Then .w(i, j, k) = Rnd() * rndRange + lLimit 'randomize weights
Next
Next
Next
End Select
End With
initUnit = unit
End Function
单元的激活函数(为了防止溢出错误，对输入的大小做了限制防止exp(x)溢出)：
Private Function UTFunction(fType As Byte, X As Double) As Double
' this function calculates the Unit Transfering Function
' to prevent from overflow error, input x should be larger than 700
If X > -700 And X < 700 Then
Select Case fType
Case UTFLOG
UTFunction = 1 / (1 + Exp(-1 * X))
Case UTFTanH
'UTFunction = Tanh(x)
UTFunction = (1 - Exp(-1 * X)) / (1 + Exp(-1 * X))
Case UTFReL
UTFunction = max(0, X)
Case UTFLkyReL
UTFunction = max(0, X) + 0.01 * min(0, X)
Case UTFLin
UTFunction = X
Case UTFSoftMax
' softmax is calculated later when results of all units are calculated, here only gives an intermediate result
UTFunction = Exp(X)
End Select
ElseIf X < -700 Then 'if x<-700 exp function will return "overflow" error, then value should be calculated manually
Select Case fType
Case UTFLOG
UTFunction = 0
Case UTFTanH
'UTFunction = Tanh(x)
UTFunction = -1
Case UTFReL
UTFunction = 0
Case UTFLkyReL
UTFunction = 0.01 * X
Case UTFLin
UTFunction = X
Case UTFSoftMax
UTFunction = 0
End Select
ElseIf X > 700 Then
Select Case fType
Case UTFLOG
UTFunction = 1
Case UTFTanH
'UTFunction = Tanh(x)
UTFunction = 1
Case UTFReL
UTFunction = X
Case UTFLkyReL
UTFunction = X
Case UTFLin
UTFunction = X
Case UTFSoftMax
UTFunction = 1E+200
End Select
End If
End Function
创建新的层，对于普通的DNN来说，只需要创建全连接层即可，使用newFullConLayer()函数，而在卷积网络中，需要用到三种不同的层：全连接层、卷积层或Pooling层，因此需要分别调用newFullConLayer(), newConvLayer()和newPoolingLayer()三个函数：
Private Function newFullConLayer(unitCount As Long, actiFunction As Byte, inSize As Long, Optional inSize2 As Long = 0 _
, Optional inSize3 As Long = 0, Optional typeOfANN As Byte = NNTypeMLP) As layer
' creates a full connection layer with give parameters
' layerForCNN = 1 if the layer is a cnn layer, then output is 3D, otherwize layerForCNN = 0 then output is 1-D
Dim l As layer, i As Long
With l
.layerType = NNFullConLayer
ReDim .units(1 To unitCount)
.outSize1 = unitCount
If typeOfANN = NNTypeCNN Then
.outSize2 = 1
.outSize3 = 1
Else 'type of ann = nnTypeMLP
.outSize2 = 0
.outSize3 = 0
End If
.utf = actiFunction
.unitSize1 = inSize
.unitSize2 = inSize2
.unitSize3 = inSize3
' the input data is either 1-D or 3-D
If inSize2 <= 0 Then ' 1-D input
For i = 1 To unitCount
.units(i) = newUnit(inSize) ' insert empty units to the output layer
Next
.unitDim = 1
ElseIf inSize3 <= 0 Then ' 2-D input
For i = 1 To unitCount
.units(i) = newUnit(inSize, inSize2)
Next
.unitDim = 2
Else ' 3-D input
For i = 1 To unitCount
.units(i) = newUnit(inSize, inSize2, inSize3)
Next
.unitDim = 3
End If
End With
newFullConLayer = l
End Function
Private Function newConvLayer(kernelCount As Long, kSize As Byte, inSize As Long, Optional kSize2 As Byte = 0 _
, Optional inSize2 As Long = 0, Optional channel As Long = 0, Optional ActFunc As Byte = UTFReL) As layer
' creates a convolutional layer with give parameters
Dim l As layer, d As Byte ' d as dimension of inputs (in conv layer case channel is not counted as one of the dimensions)
Dim i As Long
d = 1 ' set initial d = 1
If inSize2 >= 0 Then d = 2 ' d = 2 if ksize >=0
With l
.layerType = NNConvLayer
ReDim .units(1 To kernelCount)
.outSize1 = inSize - kSize + 1 ' convolution on all rows
If d = 1 Then ' input is 1-D
.outSize2 = 1
Else ' when input is 2-D
.outSize2 = inSize2 - kSize2 + 1 ' convolution on all columns when input is 2-D
End If
.outSize3 = kernelCount
.utf = UTFReL
.unitSize1 = kSize
.unitSize2 = kSize2
.unitSize3 = channel
Select Case d ' determine the input data dimensions
Case 1 ' if input data is 1-D then units will be 2-D with the second dimension for channel
For i = 1 To kernelCount
.units(i) = newUnit(kSize, channel) ' insert empty units to the output layer, even if there's only one channel
Next
.unitDim = 2
Case 2 ' 2-D input
For i = 1 To kernelCount
.units(i) = newUnit(kSize, kSize2, channel)
Next
.unitDim = 3
End Select
End With
newConvLayer = l
End Function
Private Function newPoolingLayer(ByVal rate As Byte, ByVal inSize As Long, Optional ByVal rate2 As Byte = 1, _
Optional inSize2 As Long = 1, Optional channel As Long = 1, Optional Func As Byte = UTFMaxP) As layer
' creates a pooling layer with give parameters, the pooling function can be max-pool or avg-pool
' Func is the pooling function with value: 7 = Max pooling, 8 = Average pooling, 7 is the default value
Dim l As layer 'd As Byte ' d as input data dimensions, the array defined for pooling layer will always be 3-D
With l
.layerType = NNPoolLayer
ReDim .units(1 To 1) ' a pool layer has always one unit with empty weights
.outSize1 = inSize / rate
.outSize2 = inSize2 / rate2 '
.outSize3 = channel ' output on each channel or level
.utf = Func
.unitDim = 3 ' the dimension of unit is always 3-D
.unitSize1 = rate ' two numbers are to be stored
.unitSize2 = rate2
.unitSize3 = channel
.units(1) = newUnit(rate, rate2, channel) _
' the pooling layer has only one unit, the unit has 3-D weight arrays and contains the same number of elements as _
the kernel size
End With
newPoolingLayer = l
End Function
最后这部分是神经网络的创建函数，newNN()和killNN()用于创建一个神经网络或删除网络并释放内存，initNN()对一个创建好的神经网络进行权值的初始化，NNAddLayer()用于在一个网络中插入一个新的层，不管是哪一种层都用同一个函数，参数不同。
''''''''''''''''''Define Class Neuron Net'''''''''''''
'''Class method of Neuron Nets
Public Function newNN(name As String, inSize As Long, outSize As Long, outF As Byte, Optional annType As Byte = NNTypeMLP, _
Optional inSize2 As Long = 0, Optional inSize3 As Long = 0) As NN
' This function creates an "Empty" Neuron Net that contains only one full connection perceptron layer according to _
input vector / array size and size of output vector
' the function returns the reference of the created network
' inSize defines the size of input vector, if input is an array, inSize_2 and inSize_3 defines the 2/3-D size
' outSize defines the size of output vector, it is defined that the output of a full connection perceptron layer can _
ONLY be a 1-dimensional vector (multiple rows and only one column)
' outF defines the unit transfering function of output layer
Dim net As NN, l As layer
Dim curUnit As NNUnit
Dim i As Long
l = newFullConLayer(outSize, outF, inSize, inSize2, inSize3, annType) ' the last layer is always a full connection layer
With net
.annType = annType
.annName = name
.createdOn = Now()
.createdBy = VBA.Environ("username")
.annVersion = VERSION
.cycleTrained = 0
.inSize = inSize
.inSize2 = inSize2
.inSize3 = inSize3
.OutLayer = l
.hlayercount = 0
End With
With net.TrainParams
.maxEpochs = 1000
.maxCycle = 1000000
.trainAlgorithms = NNTrainSGD
.learningSpeed = 0.1
.anneal = NNLearnSpeedANNONE
.maxValidationROC = 0.95
.maxTestROC = 0.9
.minValidationError = 0.1
.minTestError = 0.1
End With
newNN = net
End Function
Public Function initNN(net As NN, Optional ByVal wType As Long = 1, Optional ByVal lLimit As Double = -0.05, Optional ByVal uLimit As Double = 0.05) As NN
' initialize all NNUnit units in the net, wType defines init type of hidden layer units, output units are always set to rand number
' normal net works will be always initialized as small random numbers, 0 initialization is only used in special occasions
Dim outputUCount As Long
Dim lUnitCount As Long
Dim curUnit As NNUnit, curLayer As layer
Dim i As Long, j As Long
With net
For i = 1 To .hlayercount + 1 ' initialize all hidden layers
If i > .hlayercount Then
curLayer = .OutLayer
Else
curLayer = .hLayer(i)
End If
With curLayer
If .layerType <> NNPoolLayer Then ' weights of pooling layer should not be initialized or modified
If wType = wTypeRand Then
lUnitCount = UBound(.units)
For j = 1 To lUnitCount
.units(j) = initUnit(.units(j), wTypeRand, lLimit, uLimit, .unitDim) ' set all hidden layer units to be rand number
Next j
Else
lUnitCount = UBound(.units)
For j = 1 To lUnitCount
.units(j) = initUnit(.units(j), , , , .unitDim) ' set all hidden layer units to be 0
Next j
End If
End If
End With
If i > .hlayercount Then
.OutLayer = curLayer
Else
.hLayer(i) = curLayer
End If
Next i
End With
initNN = net
End Function
Public Function killNN(net As NN) As Long
' release all resources bound to the net
End Function
Public Function getLayerCount(net As NN) As Long
On Error GoTo errorHandler
With net
getLayerCount = .hlayercount + 1
End With
errorHandler:
If Err.Number <> 0 Then getLayerCount = 0
End Function
Public Function NNAddLayer(net As NN, ByVal s As Long, ByVal f As Byte, Optional ByVal t As Byte = 1, _
Optional ByVal p1 As Byte, Optional ByVal p2 As Byte) As NN
' this function adds/inserts a full connection layer before the output layer, the layer can ONLY be added just before _
the output layer and after all hidden layers previously added
' size of the layer should be provided as argument
' net is the neuron net to be modified
' s is the size (number of units) of the layer, number of weights are defined
' f is used to define the unit transfering function of the added layer
' t is the type of layers to be added, p1 and p2 means kernel size or pooling rate in different circumstances
Dim curUnit As NNUnit, outL As layer, curL As layer
Dim outUnitSize As Long
Dim outUnitCount As Long
Dim curLayerInput As Long, h As Long ' size of current layer input
Dim i As Long, j As Long
With net
' create and insert a new layer after the last hidden layer
h = .hlayercount
If h = 0 Then ' insert a new layer before the out layer
.hlayercount = 1
ReDim .hLayer(1 To 1)
Select Case t
Case 1 ' create normal full connection layer
.hLayer(1) = newFullConLayer(s, f, .inSize, .inSize2, .inSize3, .annType)
Case 2 ' create convolutional layer
.hLayer(1) = newConvLayer(s, p1, .inSize, p2, .inSize2, .inSize3)
Case 3 ' create pooling layer
.hLayer(1) = newPoolingLayer(p1, .inSize, p2, .inSize2, .inSize3)
End Select
Else ' insert a new layer before the outlayer and after tha last hidden layer
.hlayercount = h + 1
ReDim Preserve .hLayer(1 To h + 1)
With .hLayer(h)
Select Case t
Case 1
net.hLayer(h + 1) = newFullConLayer(s, f, .outSize1, .outSize2, .outSize3, net.annType)
Case 2
net.hLayer(h + 1) = newConvLayer(s, p1, .outSize1, p2, .outSize2, .outSize3)
Case 3
net.hLayer(h + 1) = newPoolingLayer(p1, .outSize1, p2, .outSize2, .outSize3)
End Select
End With
End If
' re-define the output layer if new layer size differs from previous last layer
outL = .OutLayer
curL = .hLayer(.hlayercount)
If curL.outSize1 <> outL.unitSize1 Or curL.outSize2 <> outL.unitSize2 Or curL.outSize3 <> outL.unitSize3 Then
With .OutLayer
.unitSize1 = curL.outSize1
.unitSize2 = curL.outSize2
.unitSize3 = curL.outSize3
If .unitSize2 = 0 Then
.unitDim = 1
ElseIf .unitSize3 = 0 Then
.unitDim = 2
Else
.unitDim = 3
End If
For i = 1 To .outSize1
.units(i) = resizeUnit(.units(i), curL.outSize1, curL.outSize2, curL.outSize3)
Next
End With
End If
End With
NNAddLayer = net
End Function
在下一篇文章中，我将详细介绍神经网络运行的关键：前向传播和反向传播算法的实现。
