COMP 790.139: Natural Language Processing Seminar (Fall 2016)

Instructor: Mohit Bansal
Units: 3
Office: SN 258
Lectures: Wed 10:10am-12:40pm, Rm FB-008
Office Hours: Wed 12:40pm-1:40pm (by appointment) (SN 258)
Course Webpage:
Course Email: nlpcomp790unc -at-



This course will be based on the artificial intelligence and machine learning field of natural language processing (NLP, or computational linguistics), and its important multimodal+embodied connections to computer vision and robotics. After discussing some basic foundations and core tasks of NLP (e.g., tagging, parsing, coreference resolution, sentiment analysis, etc.), we will cover more recent research topics of multimodal, grounded, and embodied semantics (i.e., language with vision and speech, for robotics), human-like language understanding+generation, question answering, and dialogue models, and interpretable and structured deep learning models for NLP.
This will be a research-oriented grad-level seminar course, where we will read lots of interesting research papers, brainstorm about ideas on latest research topics, and code and write up fun and novel projects!
Please email me or drop by my office if you have any questions!

Topics (tentative, based on time)


No formal course pre-requisites. No NLP background needed but some machine learning and coding experience highly recommended (see references below). Previous courses in probability and linear algebra are also highly recommended.


Grading will consist of:

Details in first class intro lecture slides. There will not be any exams. All submissions should be emailed to:

Lateness Policy

Written summaries are due *before class* by email to First paper summary submission (for week of 09/14) will have no late penalties. 09/21 onwards, for every week's summary submission, there will be a 25% value reduction per late day.

Collaboration Policy

Paper summaries have to be written and submitted individually. Projects are encouraged to be done in pairs (but individual projects are fine too, e.g., if it relates to your current research), with clearly outlined contributions from each team member.

Reference Books


DateTopic Readings Discussion LeadersTodo's
Aug 24Intro to the Course (slides) SLP2 Chapter1 Mohit -
Aug 31NLP Basics and Core Tasks 1: POS Tagging, Parsing, Coreference, Distributional Semantics (slides) SLP2 Chapters 5-6, 13-14, 19-21; SLP3 Chapters 8-13, 15-16, 23 Mohit -
Sep 7NLP Basics and Core Tasks 2: Compositional Semantics, Sentiment, Language Modeling, Machine Translation (slides) SLP2 Chapters 4, 17-18, 25; SLP3 Chapters 4-5, 7, 16-18, 21, 26 Mohit -
Sep 14Question Answering (KB-based and Passage-based) (1) SLP2 Chapter 23; SLP3 Chapter 27;
(2) "Semantic Parsing via Paraphrasing";
(3) "Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks";
(4) "Thorough Examination of the CNN/Daily Mail Reading Comprehension Task"
Anna Z (1, 2),
Marquis H (3, 4)
2 paper summaries (except readings)
Sep 21 Language Generation (Summarization, Dialogue) SLP2 Chapters 23-24; SLP3 Chapters 25, 28;
(1) "Abstractive Sentence Summarization with Attentive Recurrent Neural Networks";
(2) "What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment";
(3) "A Persona-Based Neural Conversation Model";
(4) "On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems"
Cheng-Yang F (1, 2),
Yen-Chun C (3, 4)
2 paper summaries
Sep 28 Human-like Language (Non-literal, Ambiguous) (1) "Black Holes and White Rabbits: Metaphor Identification with Visual Features";
(2) "An LSTM Framework for Predicting Humor in Dialogues";
(3) "Interpreting Neural Networks to Improve Politeness Comprehension";
(4) "Harnessing Context Incongruity for Sarcasm Detection"
Patrick H (1, 2),
Ying L (3, 4)
2 paper summaries
Oct 5 Image-Text Alignment and Language Disambiguation via Visual Features (1) "What are you talking about? Text-to-Image Coreference";
(2) "Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models";
(3) "Resolving Language and Vision Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes"
Scotty C (2, 3) 2 paper summaries
Oct 12 Image/Video Captioning and Referring Expressions (1) "BabyTalk: Understanding and Generating Simple Image Descriptions";
(2) "Show, Attend and Tell: Neural Image Caption Generation with Visual Attention";
(3) "Modeling Context in Referring Expressions";
(4) "Sequence to Sequence -- Video to Text"
Chris Horning (1, 2),
Ramakanth P (3, 4)
2 paper summaries
Oct 19 Visual Question Answering and Madlibs (1) "VQA: Visual Question Answering";
(2) "Visual7W: Grounded Question Answering in Images" (part of "Visual Genome");
(3) "Multimodal Compact Bilinear Pooling for Visual Q&A and Visual Grounding";
(4) "Visual Madlibs: Fill in the blank Description Generation and Question Answering"
Fan J (1, 2),
Micah M (3, 4)
2 paper summaries
Oct 26 Project Proposals (10 min presentations) NLP Colloquium (Dipanjan Das, Google Research), then extended class after that --
Nov 2 Traveling to EMNLP (make-up class on Oct 26); Project midterm write-ups due Nov06! - - Project midterm write-ups due Nov06
Nov 9 Text-to-Image Generation, Visual Storytelling (1) "Text to 3D Scene Generation with Rich Lexical Grounding";
(2) "Generative Adversarial Text to Image Synthesis";
(3) "Visual Storytelling";
(4) "Sort Story: Sorting Jumbled Images and Captions into Stories"
Marc E (1, 2),
Bhavya V (3, 4)
2 paper summaries
Nov 16 Language for Robotics 1: Instructions for Navigation, Manipulation, Skill Learning (1) "Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences" (based on the task from MacMahonEtAl2006 and Chen&Mooney2011);
(2) "Interpreting and Executing Recipes with a Cooking Robot";
(3) "Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation";
(4) "Tell Me Dave: Context-Sensitive Grounding of Natural Language to Manipulation Instructions"
Adam A (1, 2),
Jaesung P (3, 4)
2 paper summaries
Nov 23 Thanksgiving Recess (no class) - --
Nov 30 Language for Robotics 2: Grounding via Dialogue-based Games (1) "Back to the Blocks World: Learning New Actions through Situated Human-Robot Dialogue";
(2) "Learning to Interpret Natural Language Commands through Human-Robot Dialog";
(3) "Natural Language Communication with Robots";
(4) "Learning Language Games through Interaction"
Shiwei F (1, 2),
Hao T (3, 4)
2 paper summaries
Dec 7 Last Class: Final Project Presentation (and some tips on research-style writing) - --
Dec 16 Project Write-ups Due - --


The professor reserves the right to make changes to the syllabus, including project due dates. These changes will be announced as early as possible.