Final Project

Introduction

The final project is to either build a natural language processing system, or apply one for some task. The project must use or develop a dataset, and report empirical results or analyses with the dataset. It may use machine learning or rule-based approaches. It may use any type of open-source or widely available software.

You can choose to emphasize:

• Implementing and developing algorithms and features.
• Defining a new linguistic / text analysis task, and tackling it with off-the-shelf NLP software.
• Collect and explore a new textual dataset to address research hypotheses about it.

Different projects will have different balances of these three things.

This course does not have a final exam. The final project is the focus for the final part of the course.

The project will be completed in groups of 1-3. We encourage size 2, which often works well.

The project has four components over the second half of the semester: Proposal, Progress Report, Presentation, and Final report.

(Requirements for the items after the proposal are subject to revision as we get closer to them.)

Proposal (due 3/10)

A 2-4 page document outlining the problem, your approach, possible dataset(s) and/or software systems to use. This proposal

• Provide a title for your project.
• Describes the scope of the proposed work, which we will use to help give feedback and define what is necessary to complete for the project.
• Cites and describes at least two pieces of relevant prior work that are peer-reviewed research papers. (Of course you can add more.)
• Proposes at least one dataset to use or try using. You must learn a bit about it and convince us that it is available for you, and that you can easily get it, and that it is appropriate for the task and research questions you care about.
• Proposes what pre-existing software will be used to accomplish the analysis task.
• Says whether human annotation will be required, and if so, how much. We encourage to make this only a small component of the overall project, since it can potentially take a lot of time.
• Proposes a preliminary experiment to run on the data (this will be reported on in the progress report), as well as the scope of the final total project.

In general, you should illustrate that you have learned about and thought through some of the problem space and possible avenues of analysis and approaches to the problem.

Ideally, try to answer the following questions as well.

• Will your project require human annotation, or will you use a ready-made (or “pre-baked”) dataset? We think it is absolutely great to make your own annotations or labels — this lets you be more creative with defining the task you care about. However, make sure to build in time for this. We suggest you use yourselves (the project group members) as the human annotators (though if you can get others to do it, that’s great, but be aware it can be difficult to manage depending on the situation.)
• How will the train/test split be done? Some datasets have one built in. Some do not. Will you split examples by document? Author? Location? Book? Time?
• What baseline algorithm will you use? A baseline algorithm is one that is very simple and trivial to implement. For example, “predict the most common class,” or “tag all capitalized words as names,” or “select the first sentence in the document”. Sometimes it can be difficult to get a fancy algorithm to beat a baseline. “Always ask yourself, ‘What’s the simplest experiment I could do to (in)validate my hypothesis?’ Talented researchers have a knack for coming up with simple baselines.”

Formatting: please use a 10 to 12 point font with single spacing.

Groups: All group members must submit a copy on Gradescope. Clearly list all group members on the document. Gradescope should let you mark group members.

In special cases, some groups may want to change after this point. That's OK, but please be very clear when doing later turn-ins.

Progress Report (due 4/7)

You’ve had a few weeks to work on the project! You have now clarified and revised your proposed idea. You have started working on it and have some preliminary results to report.

The progress report is a 5-10 page document that describes your preliminary work and results. You should do and report on work including

• Acquire your dataset. Report its source, its basic statistics (source, size, number of words/sentences/documents) and other important properties.
• If your project involves annotation, you’ve started a pilot annotation experiment, annotating a few dozen or few hundred examples. What major issues have come up? Do you and your project partner agree or disagree on examples? (At this stage, qualitative findings about these questions are fine.)
• Run some sort of NLP algorithm — classifier, parser, etc. — on the data, and report its result. If you are using a ready-made dataset, you should define a train/test split, and you should have at least one accuracy number to report at this point.
• The report must contain at least one table or graph that conveys numerical information – for example, statistics about your data or annotations, accuracy or other results of running an algorithm on the data, or something else.
• You now have a better idea of how much you can accomplish in the rest of the semester. Lay out the major items you want to accomplish. Provide a timeline to finish them by.

In-class presentations (5/2)

Final Report (due 5/11 at end of semester)

The final report is a 12 to 20 page document that describes your project and final results. Unlike the proposal (which was only about a possible project and related work), or the progress report (which was only about results), the final report must be a complete, standalone document. Conceptually, it should include the content of both the proposal and progress report, though they will be changed. The final report describes and motivates the problem, places it in context of related work, describes the dataset and your approach, and reports results with discussion and thoughts for future work.

Submit your PDF on Moodle. Also submit a zip file with your implementation code. (Moodle limits the size of the zip file, so don’t include large data files, but feel free to provide us a URL to them.)

Here is a sample outline for your final report. There are different possible ways to structure it (for example, if you can, you can weave related work into the other sections), but we suggest you follow this outline unless you have substantial prior experience writing technical reports and research papers.

• Abstract: summarize the main components of your work in one paragraph (no more than 5 sentences). What problem are you solving? What is the key to your approach? What results did you achieve? Your abstract should draw the reader in and interest them in reading the rest of your paper to understand the details of your work.
• Introduction: explain the problem, motivate it (why is it important?), and briefly describe your approach. State a research question that your project seeks to answer: what are you trying to learn from this research project? You may also report some of your results without discussing the details of your method.
• Related work and literature review: explain what other approaches have been to the problem. You must cite at least 10 relevant research papers, and describe them and how they relate to your work. (They can be in the related work section, or woven throughout the paper if convenient.)

• Data: Describe the dataset that you are using.
• Method: Describe your approach to handling the problem. This should should include any models you used and any modeling assumptions you made. If you’ve developed new models for this project, you may even want to split a description/analysis of your models into its own section.
• Results: Describe the experiments you ran and identify your baseline method(s). Include the results you achieved with the various methods you are comparing making. This section will probably also include some figures that succinctly summarize your results. Analyze your results (including your models). If you did exploratory analysis or a significant amount of feature engineering, your analysis may merit its own section. After reading this section (and your dataset and methods), an interested reader should be able to duplicate your experiments and results.
• Discussion and Future Work: discuss any implications of your analysis for the problem as a whole, and what are the next steps for future work. Any other concluding remarks should go here.

Some things to remember:

• Write clearly. A good paper is direct, unambiguous and describes all stages of the work in complete—but not superfluous—detail. You must provide this, while not causing confusion to the reader. Quality of writing will affect your grade.
• All plots, figures, and tables must have a title, labeled axes and a caption. There should be no confusion about what you’re trying to express, or why the plot was included.
• Do not include multiple figures that all convey the same information. Be succinct.
• Avoid redundancy. Do not describe the same component of your work twice.
• Do not include much code in your report. If your approach contains a new algorithm you developed, it may be appropriate for you to include its pseudocode, but do not copy and paste code from your editor. Do not include well-known algorithms like the perceptron training algorithm. (If you feel like you need to, instead include a citation to the literature that describes the algorithm, and provide a brief English description of what it does.)
• You may include back-references to things you’ve already discussed. Avoid forward references.
• Provide a title for your report.

Writing a paper is like composing a piece of art. Be deliberate in your choice of what to include, when to include it and how to express it. For example, don’t include every plot you generated; pick the ones that best demonstrate your results. Work on the clarity of your writing. At the end of the day, your report is all the reader has to generate an opinion about your work. You don’t want good work to be obscured by poor writing.

Formatting: write using the ACL stylesheet. Please divide your report clearly into sections/subsections.