Reducing the Dimensionality of Data with Neural Networks

 

http://www.cs.toronto.edu/~hinton/science.pdf

High-dimensional data can be converted to low-dimensional codes by training a multilayer neural

network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent

can be used for fine-tuning the weights in such ‘‘autoencoder’’ networks, but this works well only if

the initial weights are close to a good solution. We describe an effective way of initializing the

weights that allows deep autoencoder networks to learn low-dimensional codes that work much

better than principal components analysis as a tool to reduce the dimensionality of data.

 

 

D

imensionality reduction facilitates the

classification, visualization, communi-

cation, and storage of high-dimensional

data. A simple and widely used method is

principal components analysis (PCA), which

 

finds the directions of greatest variance in the

data set and represents each data point by its

coordinates along each of these directions. We

describe a nonlinear generalization of PCA that

uses an adaptive, multilayer encoder  network to

transform the high-dimensional data into a

low-dimensional code and a similar decoder

network to recover the data from the code.

 

 

 

 

 

Fig. 1.Pretraining consists of learning a stack of restricted Boltzmann machines (RBMs), eachhaving only one layer of feature detectors. Thelearned feature activations of one RBM are usedas the ‘‘data’’ for training the next RBM in the stack. After the pretraining, the RBMs are‘‘unrolled’’ to create a deep autoencoder, which is then fine-tuned using backpropagation oferror derivatives.

 

 

 

 

 

Starting with random weights in the twonetworks, they can be trained together byminimizing the discrepancy between the orig-inal data and its reconstruction. The requiredgradients are easily obtained by using the chainrule to backpropagate error derivatives firstthrough the decoder network and then throughthe encoder network (1). The whole system is called an autoencoderand is depicted in Fig. 1.

It is difficult to optimize the weights innonlinear autoencoders that have multiplehidden layers (2–4). With large initial weights,autoencoders typically find poor local minima;with small initial weights, the gradients in theearly layers are tiny, making it infeasible totrain autoencoders withmany hidden layers. If the initial weights are close to a good solution,gradient descent works well, but finding suchinitial weights requires a very different type ofalgorithm that learns one layer of features at atime. We introduce this pretraining procedurefor binary data, generalize it to real-valued data,and show that it works well for a variety of data sets.

An ensemble of binary vectors (e.g., im-ages) can be modeled using a two-layer net-work called a restricted Boltzmann machine (RBM) (5,6) in which stochastic, binary pixelsare connected to stochastic, binary featuredetectors using symmetrically weighted con-nections. The pixels correspond toBvisible[units of the RBM because their states are observed; the feature detectors correspond to hidden units. A joint configuration (v,h) of the visible and hidden units has an energy (7) given by

 

 

where

v

i

and

h

j

are the binary states of pixel

i

and feature

j

,

b

i

and

b

j

are their biases, and

w

ij

is the weight between them. The network as-

signs a probability to every possible image via

this energy function, as explained in (

8

).

 

 

The probability of a training image can be raised by adjusting the weights and biases to lower theenergy of that image and to raise the energy ofsimilar, confabulated images that the network would prefer to the real data.