Format

Send to

Choose Destination
Neural Comput. 2012 Aug;24(8):1967-2006. doi: 10.1162/NECO_a_00311. Epub 2012 Apr 17.

An efficient learning procedure for deep Boltzmann machines.

Author information

1
Department of Statistics, University of Toronto, Toronto, Ontario M5S 3G3, Canada. rsalakhu@utstat.toronto.edu

Abstract

We present a new learning algorithm for Boltzmann machines that contain many layers of hidden variables. Data-dependent statistics are estimated using a variational approximation that tends to focus on a single mode, and data-independent statistics are estimated using persistent Markov chains. The use of two quite different techniques for estimating the two types of statistic that enter into the gradient of the log likelihood makes it practical to learn Boltzmann machines with multiple hidden layers and millions of parameters. The learning can be made more efficient by using a layer-by-layer pretraining phase that initializes the weights sensibly. The pretraining also allows the variational inference to be initialized sensibly with a single bottom-up pass. We present results on the MNIST and NORB data sets showing that deep Boltzmann machines learn very good generative models of handwritten digits and 3D objects. We also show that the features discovered by deep Boltzmann machines are a very effective way to initialize the hidden layers of feedforward neural nets, which are then discriminatively fine-tuned.

PMID:
22509963
DOI:
10.1162/NECO_a_00311

Supplemental Content

Full text links

Icon for Atypon
Loading ...
Support Center