• View
• Edit
• History
• Print

Whats In Your Embedding And How It Predicts Task Performance

Abstract: Word embeddings are the most widely used kind of distributional meaning representations in both industrial and academic NLP systems, and they can make dramatic difference in the performance of the system. However, the absence of a reliable intrinsic evaluation metric makes it hard to choose between dozens of models and their parameters. This work presents Linguistic Diagnostics (LD), a new methodology for evaluation, error analysis and development of word embedding models that is implemented in an open-source Python library. In a large-scale experiment with 14 datasets LD successfully highlights the differences in the output of GloVe and word2vec algorithms that correlate with their performance on different NLP tasks.

Bio: Anna is a post-doctoral associate in the Computer Science Department at Text Machine lab, University of Massachusetts (Lowell). She works at the intersection of linguistics, natural language processing, and machine learning. She holds a Ph.D. degree from the Department of Language and Information Sciences at the University of Tokyo (Japan). Her current research focuses on interpretability of deep learning, evaluation of distributional meaning representations, and semantic compositionality. She also leads annotation projects for sentiment analysis and temporal reasoning.