C O D E: DigitDemo v1.0
: This code package contains an interactive demo of Gibbs sampling, mean-field
reconstruction and classification for restricted Boltzmann machines trained using four
different inductive principles including stochastic maximum likelihood (SML), contrastive divergence (CD),
ratio matching (RM) and pseudo-likelihood (PL). The MNIST data set was used as training data.
See our AISTATS 2010 paper Inductive Principles for Restricted Boltzmann Machine Learning
for details. Note that this demo uses trained models and does not
include code for learning.
- Because of differences in GUI handling in different versions of Matlab, this code package
requires Matlab 7.2 or higher to function properly.
- The GUI was designed for a 1280x1024 screen resolution. Some GUI elements may not
automatically re-position themselves nicely on smaller screens.
This example shows the mean-field reconstruction and classification of a seven with noise added.
We see that SML and CD have significantly degraded mean-field reconstructions in the presence of
noise compared to RM and PL. All of the models make the correct classification, but RM is much
more certain that the image is a seven.
Launch the GUI by entering "DigitDemo" at the Matlab prompt.
You can draw in the "Input Image" panel by clicking and dragging with the mouse.
You can select different input tools from the "Input Tool" control including:
You can erase the input image by clicking on the "Reset" button.
You can select RBM models trained using different inductive principles using the
"Inductive Principle" control. You can switch between models at any time, including
while the Gibbs sampler is running. The inductive principles include:
- The Pen Tool: Sets 2x2 blocks to solid white
- The Brush Tool: Sets 3x3 blocks to white with a soft fall-off
- The Noise Tool: Adds 3x3 blocks of white noise to the image
- The Eraser Tool: Set 2x2 blocks to solid black
As you draw, the GUI displays the mean-field reconstruction in the
"Mean-Field Reconstruction" panel for the model currently selected using the "Inductive
Principle" control. The mean-field reconstruction is obtained by going from the input
image to the hidden unit activations and back to the input image. It is often able to
fill-in gaps in digits as well as remove noise.
A linear classification model also attempts to predict the digit you are drawing in real time as
you draw it. A separate classifier was trained using RBM hidden unit activations as input for
each of the four inductive principles. The results are displays as a distribution over the digits
0 to 9 in the "Classification" panel.
The un-normalized free energy of the input image under the currently selected model is displayed
in real time in the "Free Energy History" panel. Relative changes in free energy correspond to
changes in log probability of the input. Remember that lower energy corresponds to higher
The most active hidden units are also displayed for the currently selected model as filters
(matrix of all weights leaving a hidden unit) and are updated in real time as you draw.
The filters are a representation of the input features that each hidden unit has learned to pay
attention to. The degree to which these filters are active is the input to the classifier, while
the mean-field reconstruction of the input image is obtained by a hidden-unit-activation weighted
linear combination of the filters.
You can run a Gibbs sampler initialized from the current input image at any time by clicking the
"Start Sample" button. You can stop the sampler by clicking the "Stop Sample" button. The Gibbs
sampler samples binary values for the input image pixels. The soft visible unit probabilities
are instead displayed in the "Mean-Field Reconstruction" panel to reduce noise in the visualization.
All of the above outputs (classification, energy history, filters) are updated while the Gibbs
sampler runs. You can also switch between trained models using the "Inductive Principles" control
while the sampler runs.
- SML: Stochastic Maximum Likelihood. Also called Persistent Contrastive Divergence or PCD
- CD: Contrastive Divergence
- PL: Pseudo-Likelihood
- RM: Ratio Matching