Final Project

Introduction

The final project is to either build a natural language processing system, or apply one for some task. The project must use or develop a dataset, and report empirical results or analyses with the dataset. It may use machine learning or rule-based approaches. It may use any type of open-source or widely available software.

You can choose to emphasize:

• Implementing and developing algorithms and features.
• Defining a new linguistic / text analysis task, and tackling it with off-the-shelf NLP software.
• Collect and explore a new textual dataset to address research hypotheses about it.

Different projects will have different balances of these three things.

The key requirement is to investigate, analyze, and come to research findings about new methods, or insights about previously existing methods.

This course does not have a final exam. The final project is the focus for the final part of the course.

The project will be completed in groups of 2-4.

(Rarely, working alone may be appropriate for work tied to an external project like independent study. Working alone requires more work, and special permission from instructors and you need to ask long before the main deadlines.)

The project has four components over the second half of the semester: Proposal, Progress Report, Presentation, and Final report.

(Requirements for the items after the proposal are subject to revision as we get closer to them.)

Proposal (due Monday 10/18 Wednesday 10/20)

There are two dates:

• Wed 10/13 Fri 10/15: submit your group members (you've found a group, but may not know yet your topic).
• Mon 10/18 Wed 10/20: submit your project proposal, described as follows.

The project proposal is a 2-4 page document outlining the problem, your approach, possible dataset(s) and/or software systems to use. This proposal

• Describes one or a few research questions that you're trying to answer in your project.
• Describes the scope of the proposed work, which we will use to help give feedback and define what is necessary to complete for the project.
• Cites and briefly describes at least two pieces of relevant prior work (typically research papers).
• Proposes at least one dataset to use or try using. You must learn a bit about it and convince us that it is available for you, and that you can easily get it, and that it is appropriate for the task and research questions you care about.
• Proposes what pre-existing software will be used to accomplish the analysis task.
• Says whether human annotation will be required, and if so, how much.
• Proposes a preliminary experiment to run on the data (this will be reported on in the progress report), as well as the scope of the final total project.

In general, you should illustrate that you have learned about and thought through some of the problem space and possible avenues of analysis and approaches to the problem.

Ideally, try to answer the following questions as well.

• Will your project require human annotation, or will you use a ready-made (or “pre-baked”) dataset? We think it is absolutely great to make your own annotations or labels — this lets you be more creative with defining the task you care about. However, make sure to build in time for this. We suggest you use yourselves (the project group members) as the human annotators (though if you can get others to do it, that’s great, but be aware it can be difficult to manage depending on the situation.)
• How will the train/test split be done? Some datasets have one built in. Some do not. Will you split examples by document? Author? Location? Book? Time?
• What baseline algorithm will you use? A baseline algorithm is one that is very simple and trivial to implement. For example, “predict the most common class,” or “tag all capitalized words as names,” or “select the first sentence in the document”. Sometimes it can be difficult to get a fancy algorithm to beat a baseline. “Always ask yourself, ‘What’s the simplest experiment I could do to (in)validate my hypothesis?’ Talented researchers have a knack for coming up with simple baselines.”
  • Also note some papers use "baseline" to mean "recent state-of-the-art result I'm trying to beat." We don't require you to have this at this point, but if you can, great!

Formatting: please use a 10 to 12 point font with single spacing.

In special cases, some groups may want to change after this point. That's OK, but please be very clear when doing later turn-ins.

Project proposal meeting (~first week of Nov)

After you've submitted your project proposal, you'll receive grading and feedback. Then your entire group will schedule a meeting with one TA or UCA to discuss the feedback, answer questions, and figure out next steps. This meeting is mandatory and part of the project proposal grade. It will likely be scheduled during the week of Nov 1.

Progress Report (due Monday 11/22 Tues 11/23)

You’ve had a few weeks to work on the project! You have now clarified and revised your proposed idea. You have started working on it and have some preliminary results to report.

The progress report is a 4-8 page document that describes your preliminary work and results. You should do the following and report on them:

• Restate or revise from the proposal: what research questions are you answering? What progress have you made toward them? It is likely and totally OK that your questions or plan have shifted by now! Please report how.
• Acquire your dataset. Report its source, its basic statistics (source, size, number of words/sentences/documents) and other important properties.
• If your project involves annotation, you’ve started a pilot annotation experiment, annotating a few dozen or few hundred examples. What major issues have come up? Do you and your project partner agree or disagree on examples? (At this stage, qualitative findings about these questions are fine.)
• Run some sort of NLP algorithm — classifier, parser, etc. — on the data, and report its result. If you are using a ready-made dataset, you should define a train/test split, and you should have at least one accuracy number to report at this point.
• The report must contain at least one table or graph that conveys numerical information – for example, statistics about your data or annotations, accuracy or other results of running an algorithm on the data, or something else.
• You now have a better idea of how much you can accomplish in the rest of the semester. Lay out the major items you want to accomplish. Provide a timeline to finish them by.
• Finally, also propose a work allocation among your team members for the final phase of work. Who is doing what, and when is their work due? If you have dependencies among the subtasks for different group members, now is the time to figure it out.

Class poster session or presentations

See 11/23 slides. Poster pdf/image due Dec 1 (directions will be posted to Piazza). The poster session will be virtual on Dec 7, 4:00-5:30pm, on GatherTown! This session is always a lot of fun!

Final Report (due 12/16, end of semester)

The final report is an 8-12 page document that describes your project and final results. Unlike the proposal (which was only about a possible project and related work), or the progress report (which was only about results), the final report must be a complete, standalone document. Conceptually, it should include the content of both the proposal and progress report, though they will be changed. The final report describes and motivates the problem, places it in context of related work, describes the dataset and your approach, and reports results with discussion and thoughts for future work.

Here is a sample outline for your final report. There are different possible ways to structure it (for example, if you can, you can weave related work into the other sections), but we suggest you follow this outline if you're new to this type of technical research writing.

• Abstract: summarize the main components of your work in one paragraph (no more than 5 sentences). What problem are you solving? What is the key to your approach? What results did you achieve? Your abstract should draw the reader in and interest them in reading the rest of your paper to understand the details of your work.
• Introduction: explain the problem, motivate it (why is it important?), and briefly describe your approach. State a research question that your project seeks to answer: what are you trying to learn from this research project? You may also overview some of the coming results without discussing the details of your method.
• Related work: explain what other approaches have been to the problem. Cite specific instances of previous work.
• Data: Describe the dataset that you are using.
• Method: Describe your approach to handling the problem. This should should include any models you used and any modeling assumptions you made. If you’ve developed new models for this project, you may even want to split a description/analysis of your models into its own section.
• Results: Describe the experiments you ran and identify your baseline method(s). Include the results you achieved with the various methods you are comparing making. This section will probably also include some figures that succinctly summarize your results. Analyze your results (including your models). If you did exploratory analysis or a significant amount of feature engineering, your analysis may merit its own section. After reading this section (and your dataset and methods), an interested reader should be able to duplicate your experiments and results.
• Discussion and Future Work: discuss any implications of your analysis for the problem as a whole, and what are the next steps for future work. Any other concluding remarks should go here.

Some things to remember:

• WRITE CLEARLY. A good paper is direct, unambiguous and describes all stages of the work in complete—but not superfluous—detail. You must provide this, while not causing confusion to the reader. Quality of writing will affect your grade.
• All plots, figures, and tables must have a title, labeled axes and a caption. There should be no confusion about what you’re trying to express, or why the plot was included.
• Do not include multiple figures that all convey the same information. Be succinct.
• Avoid redundancy. Do not describe the same component of your work twice.
• Do not include much code in your report. If your approach contains a new algorithm you developed, it may be appropriate for you to include its pseudocode, but do not copy and paste code from your editor. Do not include well-known algorithms like the perceptron training algorithm. (If you feel like you need to, instead include a citation to the literature that describes the algorithm, and provide a brief English description of what it does.)
• You may include back-references to things you’ve already discussed. Avoid forward references.
• Provide a title for your report.

Writing a paper is like composing a piece of art. Be deliberate in your choice of what to include, when to include it and how to express it. For example, don’t include every plot you generated; pick the ones that best demonstrate your results. Work on the clarity of your writing. At the end of the day, your report is all the reader has to generate an opinion about your work. You don’t want good work to be obscured by poor writing.

Formatting: please use a 10 to 12 point font with single spacing. Please divide your report clearly into sections/subsections. We suggest the ACL stylesheets, though they're not necessary to use.