Abstract :
Unsupervised methods are evaluated in one of two different ways - intrinsically and extrinsically. By intrinsic evaluations we mean methods such as perplexity that require access only to more raw data without additional human annotations. Unfortunately, these methods are often weakly correlated to a model’s performance on a particular task. Extrinsic metrics, on the other hand, evaluate the model on a particular task but are dependent on ground truth annotations and often involve traversing over ranked lists.
In this talk, I will present a new extrinsic evaluation metric that compares histograms, computed over model’s similarity metric, between human and randomly generated document pairs in log space. We call this approach Histogram Slope Analysis (HSA). Across two families of topic models I will demonstrate that HSA, unlike perplexity, achieves very high correlation with traditional performance metrics such as Mean Average Precision (MAP), while being more efficient to compute. As this is work in progress any feedback is welcome.
Bio:
Kriste Krstovski is a Ph.D candidate in the School of Computer Science, at the University of Massachusetts Amherst and a Predoctoral Fellow at the Harvard-Smithsonian Center for Astrophysics. An advisee of Prof. David A. Smith, he is a member of the Information Retrieval (IR) and Information Extraction and Synthesis (IESL) Laboratories. Before starting his PhD studies Kriste was a Staff Scientist in the Speech and Language Processing Department at BBN Technologies for four years. He finished his B.S. and M.S. in Electrical Engineering at the University of New Hampshire while being a member of the Project54 team under the supervision of Andrew L. Kun and W. Tom Miller III.
