RBM Inductive Principles Digit Demo
Version: 1.0 - Nov 22, 2010
Author:  Benjamin M. Marlin

Description: This code package contains an interactive demo of Gibbs sampling, mean-field 
reconstruction and classification for restricted Boltzmann machines trained using four 
different inductive principles including stochastic maximum likelihood, contrastive divergence,
ratio matching and pseudo-likelihood. The MNIST data set was used as training data. See 
reference [1] below for details. Note that this demo uses trained models and does not 
include code for learning.

System Requirements:

* Because of differences in GUI handling in different versions of Matlab, this code package 
  requires Matlab 7.2 or higher to function properly. 

* The GUI was designed for a 1280x1024 screen resolution. Some GUI elements may not 
  automatically re-position themselves nicely on smaller screens.

Usage: 

* Launch the GUI by entering "DigitDemo" at the Matlab prompt. 

* You can draw in the "Input Image" panel by clicking and dragging with the mouse.
  You can select different input tools from the "Input Tool" control including:

  - The Pen Tool: Sets 2x2 blocks to solid white 
  - The Brush Tool: Sets 3x3 blocks to white with a soft fall-off  
  - The Noise Tool: Adds 3x3 blocks of white noise to the image 
  - The Eraser Tool: Set 2x2 blocks to solid black

* You can erase the input image by clicking on the "Reset" button.

* You can select RBM models trained using different inductive principles using the 
  "Inductive Principle" control. You can switch between models at any time, including 
   while the Gibbs sampler is running. The inductive principles include:
  
  - SML: Stochastic Maximum Likelihood. Also called Persistent Contrastive Divergence or PCD
  - CD: Contrastive Divergence
  - PL: Pseudo-Likelihood
  - RM: Ratio Matching  

* As you draw, the GUI displays the mean-field reconstruction in the "Mean-Field 
  Reconstruction" panel for the model currently selected using the "Inductive Principle" 
  control. The mean-field reconstruction is obtained by going from the input image to 
  the hidden unit activations and back to the input image. It is often able to fill-in 
  gaps in digits as well as remove noise.

* A linear classification model also attempts to predict the digit you are drawing in real time as
  you draw it. A separate classifier was trained using RBM hidden unit activations as input for 
  each of the four inductive principles. The results are displays as a distribution over the digits
  0 to 9 in the "Classification" panel.

* The un-normalized free energy of the input image under the currently selected model is displayed 
  in real time in the "Free Energy History" panel. Relative changes in free energy correspond to
  changes in log probability of the input. Remember that lower energy corresponds to higher 
   probability.

* The most active hidden units are also displayed for the currently selected model as filters
  (matrix of all weights leaving a hidden unit) and are updated in real time as you draw. 
  The filters are a representation of the input features that each hidden unit has learned to pay
  attention to. The degree to which these filters are active is the  input to the classifier, while 
  the mean-field reconstruction of the input image is obtained by a hidden-unit-activation weighted 
  linear combination of the filters.

* You can run a Gibbs sampler initialized from the current input image at any time by clicking the
  "Start Sample" button. You can stop the sampler by clicking the "Stop Sample" button. The Gibbs
  sampler samples binary values for the input image pixels. The soft visible unit probabilities 
  are instead displayed in the "Mean-Field Reconstruction" panel to reduce noise in the visualization. 
  All of the above outputs (classification, energy history, filters) are updated while the Gibbs 
  sampler runs. You can also switch between trained models using the "Inductive Principles" control
  while the sampler runs.

Contact:

* For questions, comments, bug reports, etc... please contact Ben Marlin at
bmarlin[at]cs[dot]ubc[dot]ca.
 
References:

[1] Benjamin M. Marlin, Kevin Swersky, Bo Chen and Nando de Freitas. 
Inductive Principles for Restricted Boltzmann Machine Learning. In Proceedings of the 
Thirteenth International Conference on Artificial Intelligence and Statistics. 2010. 