UMass Machine Learning and Friends Lunch | Main / Think Globally And Locally Hybrid Models For Segmentation And Labeling

Conditional Random Fields are a powerful tool for building models to segment natural images. They are particularly appropriate for modeling local interactions among labels in a segmentation. Complementary to these are Restricted Boltzmann Machines and their extensions which have been used to model global shapes produced by segmentation models. In this work we present a new model that uses the combined power of these two types of networks to build a state-of-the-art labeler for the parts of complex face images. Specifically, we investigate the problem of segmenting the Labeled Faces in the Wild (LFW) data set into hair, skin and background regions. The CRF is a good baseline labeler but we show how an RBM can be added to the architecture and provide a global shape bias that complements the local modeling provided by the CRF. This hybrid model produces results that are both quantitatively and qualitatively better than the CRF alone. In addition to high quality segmentation results for LFW, we demonstrate that the hidden unit activations in the RBM portion of our model can be interpreted as face attributes which have been learned without any attribute-specific training data.

Andrew Kae is a Ph.D candidate in the Computer Science Department, at the University of Massachusetts Amherst, advised by Erik Learned-Miller. He is a member of the Computer Vision Lab and received a MEng degree and BA degree in Computer Science from Cornell University. His research interests include computer vision and machine learning.