Machine Learning and Friends Lunch
Learning in Markov Random Fields with Contrastive Free Energies
Charles Sutton
UMass
Abstract
Learning Markov random field (MRF) models is notoriously hard due to the presence of a global normalization factor. We present a new framework for learning MRF models based on the contrastive free energy (CF) objective function. In this scheme the parameters are updated in an attempt to match the average statistics of the data distribution and a distribution which is (partially or approximately) "relaxed" to the equilibrium distribution. We show that maximum likelihood, mean field, contrastive divergence and pseudo-likelihood objectives can be understood in this paradigm. Moreover, we propose and study a new learning algorithm: the "kstep Kikuchi/Bethe approximation". This algorithm is then tested on a conditional random field model with "skip-chain" edges to model long range interactions in text data. It is demonstrated that with no loss in accuracy, the training time is brought down from 19 hours (BP based learning) to 83 minutes, an order of magnitude improvement. Joint work with Max Welling.