CMPSCI 585 : Introduction to Natural Language Processing
Fall 2007
Homework #6: Probabilistic Context Free Grammars

In this homework assignment you will implement and experiment with a probabilistic (weighted) context free grammar, and write a short report about your experiences and findings.

You may begin with your context free CYK parser from homework #2. Change this implementation so that it accepts a grammar with weights (log-probabilities, or negative log-probabilities) on production rules, and implement dynamic programming so that is properly accumulates the total weights of various parses.

You should then explore some parsing issue that leverages such a weighted grammar. There are several suggestions below. You are also free to show your creativity by creating your own substantive exploratory question.

Everyone should estimate the parameters of an HMM from counts (as we did in class on the board), and implement Viterbi, as described in the first bullet below. There are additional bullets below describing further optional exercises. As usual, you need not be limited by the suggestions of these extra bullets. I you are free to come up with your own tasks.

Please re-check this page as well as the course Web site syllabus, in the homework column for any updates and clarifications to this assignment.

Python and Data Infrastructure available

You may begin with cfg.py which is available at http://www.cs.umass.edu/~mccallum/courses/inlp2007/code. And/or your own assignment for HW#2.A

As with HW#2, we are not providing training or testing data, but you may make up your own data.

Tasks

• (Required part.) Finish an implemenetation of weighted (probabilistic) CYK to parse sentences using a context free grammar. You should provide a function that takes a sequence of words as input, and prints the most likely parse tree (in some convenient format) as a result. If there are several equally likely parse trees, you may arbitrarily choose one of them. Demonstrate this parser on some grammar on data that show off its interesting properties. Write a report about your findings. Write additional code for measuring the accuracy of your parser--being sure to test on data different than the data you trained on. (The remaining bullets are possible topics for investigation.)
• Demonstrate prepositional phrase (PP) attachment happening correctly for several different circumstances. Consider "low attachment", lexical preferences, etc.
• Consider some words with ambiguous parts-of-speech, and some ambiguous sentences with multiple parses, but set the weights on production rules to make the more likely parts-of-speech and parses come out on top. For example, you could build a grammar to parse all the "ambiguous newspaper headlines" that I showed on the first day of class.
• Experiment with relaxations of the PCFG assumptions. For example, include ancestors in the tags, etc. Tricks like this can be used to express preferences for certain structures as the OBJECT of the verb as opposed to its SUBJECT.
• Implement beam search for weighted CYK.
• (Optional extra. Not a substitute for an exploration of parsing.) Experiment with generation from your weighted grammar. Generate sentences from your grammar. Now that it is weighted, you may have better luck including recursive rules than you did when you completed HW#2.

What to hand in, and how

The homework should be emailed to cs585-staff@cs.umass.edu.

In addition to writing your Python program, write a short report about your experiences. Feel free to suggest other additional things you might like to to next that build on what you've done so far. This report should be clear, well-written, but needn't be long--one page is fine. Also, no need for fancy formatting. In fact, we prefer to receive this report as the body of your email. Your program can also be included in the body, or included as an email attachment.

Grading

The assignment will be graded for (a) correctness of your implementation, (b) quality/clarity of your written report, and (d) creativity, effort and success in the task(s) you choose.

Questions?

Feel free to ask! Send email to cs585-staff@cs.umass.edu, or if you'd like your classmates to be able to help answer your question, use cs585-class@cs.umass.edu.