Teaching Machines to Ask Clarification Questions Sudha Rao Thesis Proposal Presentation May 12th
Natural Language Understanding
2
Natural Language Understanding Understan ding Tell me the recipe for lasagna
When was Barack Obama born?
3
Natural Language Understanding Understan ding Tell me the recipe for lasagna
When was Barack Obama born?
Please bring me my coffee mug from the kitchen
4
Natural Language Understanding Understan ding Tell me the recipe for lasagna
When was Barack Obama born?
Please bring me my coffee mug from the kitchen
5
Humans Interactions
6
Humans Interactions Hey Marge!
Hey Homer
7
Humans Interactions Hey Marge!
Hey Homer Today’s Today’s math ma th class clas s was sure sure fun
You You bet!
8
Grounding in Communication Hey Marge!
Hey Homer Today’s Today’s math ma th class clas s was sure sure fun
You You bet!
Clark, Herbert H., and Susan E. Brennan. "Grounding "Grounding in communication." Perspectives Perspectives on socially shared cognition 13.1991 (1991): 127-149. 9
Absence of of shared shared knowledge: knowledge: Ask a clarification question! question!
Hey Marge!
Hey Homer Today’s Today’s math ma th class clas s was sure sure fun
You You bet! Let’s Let’s meet m eet tomorrow tom orrow at 10 am to discuss our group assignment Sure. Where do you want to meet meet though?
10
Absence of of shared shared knowledge: knowledge: Ask a clarification question! question!
Hey Marge!
Hey Homer Today’s Today’s math ma th class clas s was sure sure fun
You You bet! Let’s Let’s meet m eet tomorrow tom orrow at 10 am to discuss our group assignment Sure. Where do you want to meet meet though? In 3rd floor grad lounge Sounds good!
11
Teach Machines to Ask Clarification Clarification Questions
12
Teach Machines to Ask Clarification Clarification Questions Please bring me my coffee mug from the kitchen
What color is your coffee mug?
13
Teach Machines to Ask Clarification Clarification Questions Please bring me my coffee mug from the kitchen
What color is your coffee mug?
Tell me how to bake a cake! Sure! Which cake do you want to bake?
14
Teach Machines to Ask Clarification Clarification Questions Please bring me my coffee mug from the kitchen
What color is your coffee mug?
Context-aware questions about missing information in text
Tell me how to bake a cake! Sure! Which cake do you want to bake?
15
This is a hard problem!
16
PRIOR WORK WORK IN QUESTION GENERATION
17
RELATED WORK
Reading Comprehension Questions My class is going to the movies on a field trip next week. We have to get permission permi ssion slips signed before we go. We also need to ask our parents if they will drive to the movie theatre. We are going to see a movie that tells the story from a book we read. We love it when movies are made from books. It is fun to compare movie to the book. I usually like the book better. We get to the movie early so we can buy popcorn. Some of us buy candy and slushes too. We all enjoy watching the movie. When we return to the school, we talk about things that were in the movie and the book. The movie and book are similar.
Michael Heilman. 2011. Automatic factual question generation from text Ph.D. thesis, CMU 18
RELATED WORK
Reading Comprehension Questions My class is going to the movies on a field trip next week. We have to get permission permi ssion slips signed before we go. We also need to ask our parents if they will drive to the movie
Q: What do the students need to do before going to the movies?
theatre. We are going to see a movie that tells the story from a book we read. We love it when movies are made from books. It is fun to compare movie to the book. I usually like the book better. We get to the movie early so we can buy popcorn. Some of us buy candy and slushes too. We all enjoy watching the movie. When we return to the school, we talk about things that were in the movie and the book. The movie and book are similar.
Michael Heilman. 2011. Automatic factual question generation from text Ph.D. thesis, CMU 19
RELATED WORK
Reading Comprehension Questions My class is going to the movies on a field trip next week. We have to get permission permi ssion slips signed before we go. We also need to ask our parents if they will drive to the movie
Q: What do the students need to do before going to the movies?
theatre. We are going to see a movie that tells the story from a book we read. We love it when movies are made from books. It is fun to compare movie to the book. I usually like the book better. We get to the movie early so we can buy popcorn. Some of us buy candy and slushes too. We all enjoy watching the movie. When we return to the school, we talk about
GOAL
Assess someone’s someone’s understanding of the text
things that were in the movie and the book. The movie and book are similar.
Michael Heilman. 2011. Automatic factual question generation from text Ph.D. thesis, CMU 20
RELATED WORK
Questions for f or Literature Literature Review Writing Support writ e a better literature review review Goal: Help students write
Liu, Ming, Rafael A. Calvo, and Vasile Rus. "Automatic question generation for literature review writing support." International Conference on Intelligent Tutoring Systems . Springer Berlin Heidelberg, 2010. 21
RELATED WORK
Questions for f or Literature Literature Review Writing Support writ e a better literature review review Goal: Help students write
Cannon (1927) challenged this view mentioning that physiological changes were were not sufficient to discriminate emotions
Liu, Ming, Rafael A. Calvo, and Vasile Rus. "Automatic question generation for literature review writing support." International Conference on Intelligent Tutoring Systems . Springer Berlin Heidelberg, 2010. 22
RELATED WORK
Questions for f or Literature Literature Review Writing Support writ e a better literature review review Goal: Help students write
Cannon (1927) challenged this view mentioning that physiological changes were were not sufficient to discriminate emotions
Why did Cannon challenge this view?
Liu, Ming, Rafael A. Calvo, and Vasile Rus. "Automatic question generation for literature review writing support." International Conference on Intelligent Tutoring Systems . Springer Berlin Heidelberg, 2010. 23
RELATED WORK
Questions for f or Literature Literature Review Writing Support writ e a better literature review review Goal: Help students write
Cannon (1927) challenged this view mentioning that physiological changes were were not sufficient to discriminate emotions
Why did Cannon challenge this view?
What evidence is provided by Cannon to prove the opinion?
Liu, Ming, Rafael A. Calvo, and Vasile Rus. "Automatic question generation for literature review writing support." International Conference on Intelligent Tutoring Systems . Springer Berlin Heidelberg, 2010. 24
RELATED WORK
Bootstrapping semantic parsing parsi ng from conversations conversations SYSTEM : How can I help you? you? USER:
I would ould lik like to fly fly fr from At Atla lant nta a Geor Georgia gia to London England on September 24th in the early evening. I would like to return on October 1 st departing from London in the late morning.
Artzi, Yoav oav,, and Luke Zettlemoyer Zettlemoyer.. "Bootstrapping semantic parsers from from conversations” EMNLP 2011. 25
RELATED WORK
Bootstrapping semantic parsing parsi ng from conversations conversations SYSTEM : How can I help you? you? USER:
I would ould lik like to fly fly fr from At Atla lant nta a Geor Georgia gia to London England on September 24th in the early evening. I would like to return on October 1 st departing from London in the late morning.
SYSTEM : Leaving what city? USER:
Atlanta Georgia
Artzi, Yoav oav,, and Luke Zettlemoyer Zettlemoyer.. "Bootstrapping semantic parsers from from conversations” EMNLP 2011. 26
RELATED WORK
Bootstrapping semantic parsing parsi ng from conversations conversations SYSTEM : How can I help you? you? USER:
I would ould lik like to fly fly fr from At Atla lant nta a Geor Georgia gia to London England on September 24th in the early evening. I would like to return on October 1 st departing from London in the late morning.
SYSTEM : Leaving what city? USER:
Atlanta Georgia
SYSTEM : Going to which city? USER:
London
[conversation [conversation continues]
Artzi, Yoav oav,, and Luke Zettlemoyer Zettlemoyer.. "Bootstrapping semantic parsers from from conversations” EMNLP 2011. 27
RELATED WORK
Natural Questions about Images
Mostafazadeh, Nasrin, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. "Generating natural questions about an image." Association of Computational Linguistics 2016 28
RELATED WORK
Natural Questions about Images
Caption : A man standing standin g next to a motorcycle
Mostafazadeh, Nasrin, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. "Generating natural questions about an image." Association of Computational Linguistics 2016 29
RELATED WORK
Natural Questions about Images
Q: Was anyone injured in the crash? Q: Is the motorcyclist alive? Q: What caused the accident?
Caption : A man standing standin g next to a motorcycle
Mostafazadeh, Nasrin, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. "Generating natural questions about an image." Association of Computational Linguistics 2016 30
Talk Outline o
Problem Overview
o
Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l
o
Clarification Questions for Question-Answering forums
o
Clarification Questions for Dialogues
o
Proposed Work
o
Generalizability bey beyond Q&A forums and Dialogues
31
Talk Outline o
Problem Overview
o
Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l
o
Clarification Questions for Question-Answering forums
o
Clarification Questions for Dialogues
o
Proposed Work
o
Generalizability bey beyond Q&A forums and Dialogues
32
Talk Outline o
Problem Overview
o
Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l
o
Clarification Questions for Question-Answering forums
o
Clarification Questions for Dialogues
o
Proposed Work
o
Generalizability bey beyond Q&A forums and Dialogues
33
Talk Outline o
Problem Overview
o
Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l
o
Clarification Questions for Question-Answering forums
o
Clarification Questions for Dialogues
o
Proposed Work
o
Generalizability bey beyond Q&A forums and Dialogues
34
Talk Outline o
Problem Overview
o
Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l
o
Clarification Questions for Question-Answering forums
o
Clarification Questions for Dialogues
o
Proposed Work
o
Generalizability bey beyond Q&A forums and Dialogues
35
Talk Outline o
Problem Overview
o
Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l
o
Clarification Questions for Question-Answering forums
o
Clarification Questions for Dialogues
o
Proposed Work
o
Generalizability bey beyond Q&A forums and Dialogues
36
Talk Outline o
Problem Overview
o
Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l
o
Clarification Questions for Question-Answering forums
o
Clarification Questions for Dialogues
o
Proposed Work
o
Generalizability bey beyond Q&A forums and Dialogues
37
Problem Overview Information Content
38
Problem Overview Information Content
process
Achieve Achieve a goal
39
Problem Overview Information Content
process
Achieve Achieve a goal
!
Information retrieval retrieval ! Problem solving !
Taking an action
40
Problem Overview Information Content
process
Achieve Achieve a goal
!
Information retrieval retrieval ! Problem solving !
Taking an action
Hey Marge, Ma rge, Let us meet at 10 am tomorrow to discuss our group assignment?
Setup a meeting
41
Problem Overview Information Content
process
Achieve Achieve a goal
!
Information retrieval retrieval ! Problem solving !
Taking an action
Missing Information
Hey Marge, Ma rge, Let us meet at 10 am tomorrow to discuss our group assignment?
Setup a meeting
42
Problem Overview Information Content
process
Achieve Achieve a goal
!
Information retrieval retrieval ! Problem solving !
Taking an action
Missing Information
Hey Marge, Ma rge, Let us meet at 10 am tomorrow to discuss our group assignment?
Setup a meeting
43
Problem Overview Information Content
process
Achieve Achieve a goal
!
Information retrieval retrieval ! Problem solving !
Missing Information
Taking an action
Clarification Question
Hey Marge, Ma rge, Let us meet at 10 am tomorrow to discuss our group assignment?
Setup a meeting
44
Problem Overview Information Content
process
Achieve Achieve a goal
!
Information retrieval retrieval ! Problem solving !
Missing Information
Taking an action
Clarification Question
Hey Marge, Ma rge, Let us meet at 10 am tomorrow to discuss our group assignment?
Setup a meeting
Hey Homer, Sure. Where do you want to meet though? 45
Problem Overview Information Content
process
Achieve Achieve a goal
!
Information retrieval retrieval ! Problem solving !
Missing Information
Taking an action
Clarification Question
Hey Marge, Ma rge, Let us meet at 10 am tomorrow to discuss our group assignment?
Setup a meeting
Hey Homer,
In 3rd floor grad lounge
Sure. Where do you want to meet though? 46
Problem Overview 1
Information Content
process
Achieve Achieve a goal
!
Information retrieval retrieval ! Problem solving !
Missing Information
Taking an action
Clarification Question
Hey Marge, Ma rge, Let us meet at 10 am tomorrow to discuss our group assignment?
Setup a meeting
Hey Homer,
In 3rd floor grad lounge
Sure. Where do you want to meet though? 47
Problem Overview 1
Information Content
process
Achieve Achieve a goal
!
Information retrieval retrieval ! Problem solving !
Missing Information
2
Taking an action
Clarification Question
Hey Marge, Ma rge, Let us meet at 10 am tomorrow to discuss our group assignment?
Setup a meeting
Hey Homer,
In 3rd floor grad lounge
Sure. Where do you want to meet though? 48
Problem Overview 1
Information Content
process
Achieve Achieve a goal
!
Information retrieval retrieval ! Problem solving !
3
Missing Information
2
Taking an action
Clarification Question
Hey Marge, Ma rge, Let us meet at 10 am tomorrow to discuss our group assignment?
Setup a meeting
Hey Homer,
In 3rd floor grad lounge
Sure. Where do you want to meet though? 49
Problem Overview 1
Information Content 3
Missing Information
2
Clarification Question
50
Problem Overview 1
Information Content 3
Missing Information
1
2
Clarification Question
2
Context
3
Question
Answer
51
Problem Overview 1
Information Content 3
Missing Information
1
2
Clarification Question
2
Context
Let us meet at 10 am tomorrow to discuss our group assignment?
3
Question
Where do you want to meet though?
Answer
In 3rd floor grad lounge
52
Problem Formulation
Context
Let us meet at 10 am tomorrow to discuss our group assignment?
53
Problem Formulation Generate Question Candidates Context
Let us meet at 10 am tomorrow to discuss our group assignment?
Who are you?
Where do you want to meet though?
What will we be discussing?
54
Problem Formulation Generate Question Candidates Context
Let us meet at 10 am tomorrow to discuss our group assignment?
Rank the question candidates
Who are you?
1
Where do you want to meet though?
Where do you want to meet though?
2
What will we be discussing?
What will we be discussing?
10
Who are you?
55
Talk Outline o
Problem Overview
o
Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l
o
Clarification Questions for Question-Answering forums
o
Clarification Questions for Dialogues
o
Proposed Work
o
Generalizability bey beyond Q&A forums and Dialogues
56
Expected Value Value of Perfect Information (EVPI) inspired model
Key Idea Hey Marge, M arge, Let us meet at 10 am tomorrow to discuss our group assignment
Possible questions (a) Where do you want to meet?
" Just right
57
Expected Value Value of Perfect Information (EVPI) inspired model
Key Idea Hey Marge, M arge, Let us meet at 10 am tomorrow to discuss our group assignment
Possible questions (a) Where do you want to meet?
" Just right
(b) Is (b) Is the moon waning waning or waxing? waxing?
"
Not useful
(c) Did (c) Did you you see the new new homework homework? ?
"
Does not add value
58
Expected Value Value of Perfect Information (EVPI) inspired model
Definition of EVPI
Avriel, Avriel, Mordecai, Mordecai, and A. C. Williams. "The value of information and stochastic programming." programming." Operations Operations Research 18.5 (1970): 947-954. 59
Expected Value Value of Perfect Information (EVPI) inspired model
Definition of EVPI
o
What is the value of gathering an additional information inf ormation x?
Avriel, Avriel, Mordecai, Mordecai, and A. C. Williams. "The value of information and stochastic programming." programming." Operations Operations Research 18.5 (1970): 947-954. 60
Expected Value Value of Perfect Information (EVPI) inspired model
Definition of EVPI
o
What is the value of gathering an additional information inf ormation x?
Avriel, Avriel, Mordecai, Mordecai, and A. C. Williams. "The value of information and stochastic programming." programming." Operations Operations Research 18.5 (1970): 947-954. 61
Expected Value Value of Perfect Information (EVPI) inspired model
Definition of EVPI
o
What is the value of gathering an additional information inf ormation x?
o
Since we have not acquired x, we define its value in expectation
Avriel, Avriel, Mordecai, Mordecai, and A. C. Williams. "The value of information and stochastic programming." programming." Operations Operations Research 18.5 (1970): 947-954. 62
Expected Value Value of Perfect Information (EVPI) inspired model
Definition of EVPI
o
What is the value of gathering an additional information inf ormation x?
o
o
Since we have not acquired x, we define its value in expectation Expectation is over all possible x, weighted by each x’s likelihood.
Avriel, Avriel, Mordecai, Mordecai, and A. C. Williams. "The value of information and stochastic programming." programming." Operations Operations Research 18.5 (1970): 947-954. 63
Expected Value Value of Perfect Information (EVPI) inspired model
Definition of EVPI
o
What is the value of gathering an additional information inf ormation x?
o
o
Since we have not acquired x, we define its value in expectation Expectation is over all possible x, weighted by each x’s likelihood.
EVPI =
Avriel, Avriel, Mordecai, Mordecai, and A. C. Williams. "The value of information and stochastic programming." programming." Operations Operations Research 18.5 (1970): 947-954. 64
Expected Value Value of Perfect Information (EVPI) inspired model
Definition of EVPI
o
What is the value of gathering an additional information inf ormation x?
o
o
Since we have not acquired x, we define its value in expectation Expectation is over all possible x, weighted by each x’s likelihood. Likelihood Likelihood of x
EVPI =
Avriel, Avriel, Mordecai, Mordecai, and A. C. Williams. "The value of information and stochastic programming." programming." Operations Operations Research 18.5 (1970): 947-954. 65
Expected Value Value of Perfect Information (EVPI) inspired model
Definition of EVPI
o
What is the value of gathering an additional information inf ormation x?
o
o
Since we have not acquired x, we define its value in expectation Expectation is over all possible x, weighted by each x’s likelihood. Likelihood Likelihood of x
EVPI = Value Value or Utility of x Avriel, Avriel, Mordecai, Mordecai, and A. C. Williams. "The value of information and stochastic programming." programming." Operations Operations Research 18.5 (1970): 947-954. 66
Expected Value Value of Perfect Information (EVPI) inspired model
EVPI formulation for our problem
67
Expected Value Value of Perfect Information (EVPI) inspired model
EVPI formulation for our problem
c:
given given information content
q: question from a set of question candidates Q
68
Expected Value Value of Perfect Information (EVPI) inspired model
EVPI formulation for our problem Ask question ‘q’ that maximizes the expected expected utility of the updated information
c:
given given information content
q: question from a set of question candidates Q
69
Expected Value Value of Perfect Information (EVPI) inspired model
EVPI formulation for our problem Ask question ‘q’ that maximizes the expected expected utility of the updated information
c:
given given information content
q: question from a set of question candidates Q a:
answer from a set of answer candidates A
70
Expected Value Value of Perfect Information (EVPI) inspired model
EVPI formulation for our problem Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’
c:
given given information content
from a set of question candidates candidates Q q: question from a:
answer from a set of
answer candidates candid ates A
71
Expected Value Value of Perfect Information (EVPI) inspired model
EVPI formulation for our problem Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’
Utility of updating content ‘c’ with answer ‘a’
c:
given given information content
from a set of question candidates candidates Q q: question from a:
answer from a set of
answer candidates candid ates A
72
Expected Value Value of Perfect Information (EVPI) inspired model
EVPI formulation for our problem Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’
Utility of updating content ‘c’ with answer ‘a’
c:
given given information content
from a set of question candidates candidates Q q: question from a:
answer from a set of
1
answer candidates candid ates A
73
Expected Value Value of Perfect Information (EVPI) inspired model
EVPI formulation for our problem 2
Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’
Utility of updating content ‘c’ with answer ‘a’
c:
given given information content
from a set of question candidates candidates Q q: question from a:
answer from a set of
1
answer candidates candid ates A
74
Expected Value Value of Perfect Information (EVPI) inspired model
EVPI formulation for our problem 2
Answer Modeling
Utility of updating content ‘c’ with answer ‘a’
c:
given given information content
from a set of question candidates candidates Q q: question from a:
answer from a set of
1
answer candidates candid ates A
75
Expected Value Value of Perfect Information (EVPI) inspired model
EVPI formulation for our problem 2
Answer Modeling
3 c:
Utility of updating content ‘c’ with answer ‘a’
given given information content
from a set of question candidates candidates Q q: question from a:
answer from a set of
1
answer candidates candid ates A
76
Expected Value Value of Perfect Information (EVPI) inspired model
EVPI formulation for our problem 2
Answer Modeling
3 c:
Utility Calculator
given given information content
from a set of question candidates candidates Q q: question from a:
answer from a set of
1
answer candidates candid ates A
77
Expected Value Value of Perfect Information (EVPI) inspired model
1
Question & Answer candidates
78
Expected Value Value of Perfect Information (EVPI) inspired model
1
Question & Answer candidates
79
Expected Value Value of Perfect Information (EVPI) inspired model
1
Question & Answer candidates
80
Expected Value Value of Perfect Information (EVPI) inspired model
1
Question & Answer candidates
81
Expected Value Value of Perfect Information (EVPI) inspired model
1
Question & Answer candidates
82
Expected Value Value of Perfect Information (EVPI) inspired model
1
Question & Answer candidates
83
Expected Value Value of Perfect Information (EVPI) inspired model
84
Expected Value Value of Perfect Information (EVPI) inspired model
2
Answer Answer Modeling
Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’
85
Expected Value Value of Perfect Information (EVPI) inspired model
2
Answer Answer Modeling
Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’
Approach: !
We define the likelihood function as: Pr(ak|c j,q j) = closeness(a k , F(c j, q j) )
86
Expected Value Value of Perfect Information (EVPI) inspired model
2
Answer Answer Modeling
Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’
Approach: !
We define the likelihood function as: Pr(ak|c j,q j) = closeness(a k , F(c j, q j) )
! F(c j, q j): Trained to be close to the true answer a j
87
Expected Value Value of Perfect Information (EVPI) inspired model
2
Answer Answer Modeling
Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’
Approach: !
We define the likelihood function as:
Let us meet at 10 am tomorrow to discuss our group assignment?
Where do you want to meet though?
Pr(ak|c j,q j) = closeness(a k , F(c j, q j) ) ! F(c j, q j): Trained to be close to the true answer a j
In 3rd floor grad lounge
88
Expected Value Value of Perfect Information (EVPI) inspired model
2
Answer Answer Modeling
Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’
Approach: !
We define the likelihood function as: Pr(ak|c j,q j) = closeness(a k , F(c j, q j) )
! F(c j, q j): Trained to be close to the true answer a j
and to answers whose questions are similar to q j
89
Expected Value Value of Perfect Information (EVPI) inspired model
2
Answer Answer Modeling
Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’
Let us meet at 10 am tomorrow to discuss our group assignment?
Approach: !
We define the likelihood function as:
Do you have a place in mind?
Pr(ak|c j,q j) = closeness(a k , F(c j, q j) ) ! F(c j, q j): Trained to be close to the true answer a j
In a coffee shop?
and to answers whose questions are similar to q j
90
Expected Value Value of Perfect Information (EVPI) inspired model
2
Answer Answer Modeling
Likelihood of ‘a’ being the answer to t o the question questi on ‘q’ asked on ‘c’
Approach: !
We define the likelihood function as: Pr(ak|c j,q j) = closeness(a k , F(c j, q j) )
! F(c j, q j): Trained to be close to the true answer a j
and to answers whose questions are similar to q j
91
Expected Value Value of Perfect Information (EVPI) inspired model
2
1
Answer Answer Modeling
Question & Answer candidates
92
Expected Value Value of Perfect Information (EVPI) inspired model
2
1
Answer Answer Modeling
Question & Answer candidates
3
Utility Calculator
93
Expected Value Value of Perfect Information (EVPI) inspired model
3
Utility Calculator
Utility of updating content ‘c’ with answer ‘a’
94
Expected Value Value of Perfect Information (EVPI) inspired model
3
Utility Calculator
Utility of updating content ‘c’ with answer ‘a’ Approach
Initial contents
low utility
Updated contents
high utility
95
Expected Value Value of Perfect Information (EVPI) inspired model
3
Utility Calculator
Utility of updating content ‘c’ with answer ‘a’ Approach
!
Initial contents
low utility
Updated contents
high utility
Let us meet tomorrow at 10am to discuss the next group assignment?
Let us meet tomorrow at 10am in 3rd floor grad lounge to discuss the next group assignment?
96
Expected Value Value of Perfect Information (EVPI) inspired model
3
Utility Calculator
Utility of updating content ‘c’ with answer ‘a’ Approach
Initial contents
low uti utility
y=0
Updated contents
high utility
y=1
Let us meet tomorrow at 10am to discuss the next group assignment?
Let us meet tomorrow at 10am in 3rd floor grad lounge to discuss the next group assignment?
97
Expected Value Value of Perfect Information (EVPI) inspired model
3
Utility Calculator
Utility of updating content ‘c’ with answer ‘a’ Approach
!
Initial contents
low uti utility
y=0
Updated contents
high utility
y=1
!
Train F(c,a) to minimize cross-entropy
!
U(c+a) U(c+a ) = Value between 0 and 1
Let us meet tomorrow at 10am to discuss the next group assignment?
Let us meet tomorrow at 10am in 3rd floor grad lounge to discuss the next group assignment?
98
Expected Value Value of Perfect Information (EVPI) inspired model
3
Utility Calculator
Utility of updating content ‘c’ with answer ‘a’ Approach !
Initial contents
low uti utility
y=0
!
Updated contents
high utility
y=1
!
Train F(c,a) to minimize cross-entropy
!
U(c+a) U(c+a ) = Value between 0 and 1 99
Expected Value Value of Perfect Information (EVPI) inspired model
Our EVPI inspired model
Question & Answer candidates
Answer Modeling
Utility Calculator
100
Expected Value Value of Perfect Information (EVPI) inspired model
Our EVPI inspired model
Question & Answer candidates
Answer Modeling
Trained using a joint loss function =
Utility Calculator
!i loss ans + loss utiil 101
Talk Outline o
Problem Overview
o
Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l
o
Clarification Questions for Question-Answering forums
o
Clarification Questions for Dialogues
o
Proposed Work
o
Generalizability bey beyond Q&A forums and Dialogues
102
Clarification Questions for f or Question-Answering Forums Forums
Dataset
103
Clarification Questions for f or Question-Answering Forums Forums
How to configure path or set environment variables for installation? i'm aiming to install ape, a simple code for pseudopotential generation. i'm having this error message while running ./configure
Initial Post
So I have the library but the program installation isn't finding it. Any help? help? Thanks Thanks in advance! advance!
104
Clarification Questions for f or Question-Answering Forums Forums
How to configure path or set environment variables for installation? i'm aiming to install ape, a simple code for pseudopotential generation. i'm having this error message while running ./configure
Initial Post
So I have the library but the program installation isn't finding it. Any help? help? Thanks Thanks in advance! advance!
Question comment
What version of ubuntu do you have?
105
Clarification Questions for f or Question-Answering Forums Forums
How to configure path or set environment variables for installation? i'm aiming to install ape, a simple code for pseudopotential generation. i'm having this error message while running ./configure
Initial Post
Question comment
So I have the library but the program installation isn't finding it. Any help? help? Thanks Thanks in advance! advance!
What version of ubuntu do you have? i'm aiming to install ape in Ubuntu 14.04 LTS, a simple code for pseudopotential generation. i'm having this error message while running ./configure So I have the library but the program installation isn't finding it.
Updated Post
Any help? help? Thanks Thanks in advance! advance!
106
Clarification Questions for f or Question-Answering Forums Forums
How to configure path or set environment variables for installation? i'm aiming to install ape, a simple code for pseudopotential generation. i'm having this error message while running ./configure
Initial Post
Question comment
So I have the library but the program installation isn't finding it. Any help? help? Thanks Thanks in advance! advance!
What version of ubuntu do you have? i'm aiming to install ape in Ubuntu 14.04 LTS, a simple code for pseudopotential generation. i'm having this error message while
Edit as an answer to the question
running ./configure So I have the library but the program installation isn't finding it.
Updated Post
Any help? help? Thanks Thanks in advance! advance!
107
Clarification Questions for f or Question-Answering Forums Forums
How to configure path or set environment variables for installation? i'm aiming to install ape, a simple code for pseudopotential generation. i'm having this error message while running ./configure
Initial Post
Question comment
So I have the library but the program installation isn't finding it. Any help? help? Thanks Thanks in advance! advance!
What version of ubuntu do you have? i'm aiming to install ape in Ubuntu 14.04 LTS, a simple code for pseudopotential generation. i'm having this error message while
Edit as an answer to the question
running ./configure So I have the library but the program installation isn't finding it.
Updated Post
Any help? help? Thanks Thanks in advance! advance!
108
Clarification Questions for f or Question-Answering Forums Forums
Dataset Creation (post, question, answer) triples Post:
Original post
Question:
Clarification question posted in comments
Answer:
Edit made to the post answering the question
109
Clarification Questions for f or Question-Answering Forums Forums
Dataset Creation (post, question, answer) triples Post:
Original post
Context
Question:
Clarification question posted in comments
Question
Answer:
Edit made to the post answering the question
Answer
110
Clarification Questions for f or Question-Answering Forums Forums
Dataset Creation (post, question, answer) triples Post:
Original post
Context
Question:
Clarification question posted in comments
Question
Answer:
Edit made to the post answering the question
Answer
We extract a total of 37,000 such triples from three related domains on StackExchange: askubuntu, unix & superuser 111
Clarification Questions for f or Question-Answering Forums Forums USING OUR EVPI MODEL
112
Clarification Questions for f or Question-Answering Forums Forums USING OUR EVPI MODEL
Question & Answer Candidate Generator
Using Lucene
113
Clarification Questions for f or Question-Answering Forums Forums USING OUR EVPI MODEL
Question & Answer Candidate Generator
Using Lucene
Answer Modeling
Utility Calculator
Using Long Short Term Memory (LSTM) Model
114
Clarification Questions for f or Question-Answering Forums Forums NEURAL NETWORK MODEL
Answer Modeling
Utility Calculator
115
Clarification Questions for f or Question-Answering Forums Forums LONG SHORT TERM MEMORY (LSTM)
Sepp Hochreiter and J¨urgen Schmidhuber. 1997. Long short-term memory. Neural computation , 9(8):1735–1780. Jeffrey Jeffrey Pennington, Pennington, Richard Socher, Socher, and Christopher D Manning. 2014. “Glove: “Glove: Global vectors for word word representation” representation” In Empirical Methods on Natural Language Processing. Processing. 116
Clarification Questions for f or Question-Answering Forums Forums
Answer Answer Modeling
117
Clarification Questions for f or Question-Answering Forums Forums
Answer Answer Modeling
118
Clarification Questions for f or Question-Answering Forums Forums
Answer Answer Modeling
119
Clarification Questions for f or Question-Answering Forums Forums
Answer Answer Modeling
120
Clarification Questions for f or Question-Answering Forums Forums
Answer Answer Modeling
121
Clarification Questions for f or Question-Answering Forums Forums
Answer Answer Modeling
122
Clarification Questions for f or Question-Answering Forums Forums
Utility Calculator
123
Clarification Questions for f or Question-Answering Forums Forums
Utility Calculator i'm aiming to install ape, a simple code
F(p, a)
!
0
for pseudopotential generation. i'm having this error message while running ./configure
Initial Post
So I have the library but the program installation isn't finding it. Any help? help? Thanks Thanks in advance! advance!
124
Clarification Questions for f or Question-Answering Forums Forums
Utility Calculator i'm aiming to install ape, a simple code
F(p, a)
!
0
for pseudopotential generation. i'm having this error message while running ./configure
Initial Post
So I have the library but the program installation isn't finding it. Any help? help? Thanks Thanks in advance! advance!
i'm aiming to install ape in Ubuntu 14.04 LTS,
F(p, a)
!
1
a simple code for f or pseudopotential generation. i'm having this error message
Updated Post
while running running ./configure ./configure So I have the library but the program installation isn't finding it. Any help? help? Thanks Thanks in advance! advance! 125
Clarification Questions for f or Question-Answering Forums Forums
Experiments
126
Clarification Questions for f or Question-Answering Forums Forums
Experimental Results Accuracy
MRR
Recall@3
Recall@5
Random Bag-of-ngrams Neural baseline EVPI Neural
Accuracy: Accuracy: How How often is correct correct the top 1 in 10 MRR:
Mean Reciprocal Rank
Recall@3: How often is correct in top 3
Dataset Askubuntu Unix Superuser
~14K ~7K ~17 K
Recall@5: How often is correct in top 5
127
Clarification Questions for f or Question-Answering Forums Forums
Experimental Results Random
Accuracy
MRR
Recall@3
Recall@5
10.0
29.3
30.0
50.0
Bag-of-ngrams Neural baseline EVPI Neural
Accuracy: Accuracy: How How often is correct correct the top 1 in 10 MRR:
Mean Reciprocal Rank
Recall@3: How often is correct in top 3
Dataset Askubuntu Unix Superuser
~14K ~7K ~17 K
Recall@5: How often is correct in top 5
128
Clarification Questions for f or Question-Answering Forums Forums
Experimental Results Accuracy
MRR
Recall@3
Recall@5
Random
10.0
29.3
30.0
50.0
Bag-of-ngrams
11.6
31.3
32.5
54.6
Neural baseline EVPI Neural
Accuracy: Accuracy: How How often is correct correct the top 1 in 10 MRR:
Mean Reciprocal Rank
Recall@3: How often is correct in top 3
Dataset Askubuntu Unix Superuser
~14K ~7K ~17 K
Recall@5: How often is correct in top 5
129
Clarification Questions for f or Question-Answering Forums Forums
Neural Baseline
EVPI Neural
Answer Modeling
Utility Calculator
130
Clarification Questions for f or Question-Answering Forums Forums
Experimental Results Accuracy
MRR
Recall@3
Recall@5
Random
10.0
29.3
30.0
50.0
Bag-of-ngrams
11.6
31.3
32.5
54.6
Neural baseline
17.4
37.8
43.2
63.9
EVPI Neural
Accuracy: Accuracy: How How often is correct correct the top 1 in 10 MRR:
Mean Reciprocal Rank
Recall@3: How often is correct in top 3
Dataset Askubuntu Unix Superuser
~14K ~7K ~17 K
Recall@5: How often is correct in top 5
131
Clarification Questions for f or Question-Answering Forums Forums
Experimental Results Accuracy
MRR
Recall@3
Recall@5
Random
10.0
29.3
30.0
50.0
Bag-of-ngrams
11.6
31.3
32.5
54.6
Neural baseline
17.4
37.8
43.2
63.9
EVPI Neural
23.3
43.4
51.0
70.3
Accuracy: Accuracy: How How often is correct correct the top 1 in 10 MRR:
Mean Reciprocal Rank
Recall@3: How often is correct in top 3
Dataset Askubuntu Unix Superuser
~14K ~7K ~17 K
Recall@5: How often is correct in top 5
132
Clarification Questions for f or Question-Answering Forums Forums
How does our model compare to non-expert humans?
133
Clarification Questions for f or Question-Answering Forums Forums
Annotation scheme
o
o
14 computer science graduate students perform annotations on 50 examples Given a post and a set of ten candidate questions #
Mark the one you think is the right one # Mark all the ones that you think may be valid o
Human annotators found this task very hard!
134
Clarification Questions for f or Question-Answering Forums Forums
Sample human annotation task
135
Clarification Questions for f or Question-Answering Forums Forums
Our model choses the right question
136
Clarification Questions for f or Question-Answering Forums Forums
Our model choses a valid question
137
Clarification Questions for f or Question-Answering Forums Forums
Our model choses a wrong question
138
Clarification Questions for f or Question-Answering Forums Forums
Findings from human annotations
o
o
o
Annotator’s Annotator’s accuracy was 36%, significantly better than our best system (23%) If we limit to examples on which they were more confident, their accuracy raised to 42%, but never surpassed that Aver Average age number of “valid” answers for a single post was 4.26 (out of ten)
139
Talk Outline o
Problem Overview
o
Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l
o
Clarification Questions for Question-Answering forums
o
Clarification Questions for Dialogues
o
Proposed Work
o
Generalizability bey beyond Q&A forums and Dialogues
140
Clarification Questions for Dialogues
Ubuntu Chat Dataset
is anyone having trouble with ubuntu ubuntu server 8.04 with a slow apt-get update? what country country do you you live live in and where are you getting the sources from ?
us.archive.ubuntu.com live in the US which state? state? michigan
thanks!
you could try using a mirror that is closer to home: https://launchpad.net/ ubuntu/+archivemirrors
Ryan Lowe, Nissan Pow, Iulian V Serban, and Joelle Pineau. 2015. “The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems” In Special Interest Group on Discourse and Dialogue.
141
Clarification Questions for Dialogues
Ubuntu Chat Dataset
is anyone having trouble with ubuntu ubuntu server 8.04 with a slow apt-get update? what country country do you you live live in and where are you getting the sources from ?
CONTEXT
us.archive.ubuntu.com live in the US which state? state? michigan
thanks!
you could try using a mirror that is closer to home: https://launchpad.net/ ubuntu/+archivemirrors
Ryan Lowe, Nissan Pow, Iulian V Serban, and Joelle Pineau. 2015. “The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems” In Special Interest Group on Discourse and Dialogue.
142
Clarification Questions for Dialogues
Ubuntu Chat Dataset
is anyone having trouble with ubuntu ubuntu server 8.04 with a slow apt-get update? what country country do you you live live in and where are you getting the sources from ?
CONTEXT
us.archive.ubuntu.com live in the US which state? state?
RESPONSE
michigan
thanks!
you could try using a mirror that is closer to home: https://launchpad.net/ ubuntu/+archivemirrors
Ryan Lowe, Nissan Pow, Iulian V Serban, and Joelle Pineau. 2015. “The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems” In Special Interest Group on Discourse and Dialogue.
143
Clarification Questions for Dialogues
Our Dialogue Dataset
is anyone having trouble with ubuntu ubuntu server 8.04 with a slow apt-get update? what country country do you you live live in and where are you getting the sources from ?
CONTEXT
us.archive.ubuntu.com live in the US which state? state?
QUESTION
michigan
thanks!
you could try using a mirror that is closer to home: https://launchpad.net/ ubuntu/+archivemirrors
144
Clarification Questions for Dialogues
Our Dialogue Dataset
is anyone having trouble with ubuntu ubuntu server 8.04 with a slow apt-get update? what country country do you you live live in and where are you getting the sources from ?
CONTEXT
us.archive.ubuntu.com live in the US which state? state?
ANSWER
QUESTION
michigan
thanks!
you could try using a mirror that is closer to home: https://launchpad.net/ ubuntu/+archivemirrors
145
Clarification Questions for Dialogues (CONTEXT, QUESTION, ANSWER) TRIPLES
Context:
Context of the conversation
Question:
Clarification question following the context
Answer:
Response Respons e following the question questi on
146
Clarification Questions for Dialogues (CONTEXT, QUESTION, ANSWER) TRIPLES
Context:
Context of the conversation
Question:
Clarification question following the context
Question
Answer:
Response Respons e following the question questi on
Answer
Context
147
Clarification Questions for Dialogues EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
148
Clarification Questions for Dialogues EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
is anyone having trouble with ubuntu ubuntu server 8.04 8.04 with a slow slow apt-get apt-get update? what country country do you you live in and where are you getting the sources from ? us.archive.ubuntu.com live in the US which state? state?
149
Clarification Questions for Dialogues EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications
150
Clarification Questions for Dialogues EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications
hi, what is the best filesystem to use on a usb thumb drive? ext2 and ext3 seem to constantly write to the drive
what's wrong wrong with vfat ?
isnt that too microsoftish?
i'd just use whatever works, works, honestly honestly :)
151
Clarification Questions for Dialogues EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications Collect human annotations on a subset : Is this a clarification question?
152
Clarification Questions for Dialogues EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications Collect human annotations on a subset : Is this a clarification question?
Train a classifier to identify clarifications
153
Clarification Questions for Dialogues EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications Collect human annotations on a subset : Is this a clarification question?
Train a classifier to identify clarifications
o
Not all clarifications clar ifications are answer answered ed in the next turn
154
Clarification Questions for Dialogues EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications Collect human annotations on a subset : Is this a clarification question?
Train a classifier to identify clarifications
o
Not all clarifications clar ifications are answer answered ed in the next turn Collect human annotations on a subset : Is the clarification answered answered in next turn?
155
Clarification Questions for Dialogues
ONGOING WORK
EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications Collect human annotations on a subset : Is this a clarification question?
Train a classifier to identify clarifications
o
Not all clarifications clar ifications are answer answered ed in the next turn Collect human annotations on a subset : Is the clarification answered answered in next turn?
156
Clarification Questions for Dialogues
ONGOING WORK
EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications
90%
Collect human annotations on a subset : Is this a clarification question?
Train a classifier to identify clarifications
o
Not all clarifications clar ifications are answer answered ed in the next turn Collect human annotations on a subset : Is the clarification answered answered in next turn?
157
Clarification Questions for Dialogues
ONGOING WORK
EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications
90%
Collect human annotations on a subset :
45%
Is this a clarification question?
Train a classifier to identify clarifications
o
Not all clarifications clar ifications are answer answered ed in the next turn Collect human annotations on a subset : Is the clarification answered answered in next turn?
158
Clarification Questions for Dialogues
ONGOING WORK
EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications
90%
Collect human annotations on a subset :
45%
Is this a clarification question?
Train a classifier to identify clarifications
o
Not all clarifications clar ifications are answer answered ed in the next turn Collect human annotations on a subset : Is the clarification answered answered in next turn?
35%
159
Clarification Questions for Dialogues
ONGOING WORK
EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications
90%
Collect human annotations on a subset :
45%
Is this a clarification question?
Train a classifier to identify clarifications
o
Not all clarifications clar ifications are answer answered ed in the next turn Collect human annotations on a subset : Is the clarification answered answered in next turn?
o
35%
35% of 1 million conversations from Ubuntu Dataset
160
Clarification Questions for Dialogues
ONGOING WORK
EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications
90%
Collect human annotations on a subset :
45%
Is this a clarification question?
Train a classifier to identify clarifications
o
Not all clarifications clar ifications are answer answered ed in the next turn Collect human annotations on a subset : Is the clarification answered answered in next turn?
o
35%
35% of 1 million conversations from Ubuntu Dataset
Classifier confidence 161
Clarification Questions for Dialogues EXTRACTING (CONTEXT, QUESTION, ANSWER) TRIPLES o
Identify responses with ‘?’
o
Not all questions are clarifications
90%
Collect human annotations on a subset :
45%
Is this a clarification question?
Train a classifier to identify clarifications
o
Not all clarifications clar ifications are answer answered ed in the next turn Collect human annotations on a subset : Is the clarification answered answered in next turn?
o
35%
35% of 1 million conversations from Ubuntu Dataset
Classifier confidence
~200K (context, question, answer) 162
Clarification Questions for Dialogues USING OUR EVPI MODEL
163
Clarification Questions for Dialogues USING OUR EVPI MODEL
Question & Answer Candidate Generator
Using Lucene
164
Clarification Questions for Dialogues USING OUR EVPI MODEL
Question & Answer Candidate Generator
Using Lucene
Answer Modeling
Utility Calculator
Using an Ensemble of Neural Dialogue Model
165
Clarification Questions for Dialogues LONG SHORT TERM MEMORY MEMORY (LSTM) BASELINE
Context LSTM C
Hello there there how how are you you ?
σ(cTMr
+ b)
Response LSTM
I am fine thank you you .
r
Ryan Lowe, Nissan Pow, Iulian V Serban, and Joelle Pineau. 2015. “The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems” In Special Interest Group on Discourse and Dialogue. 166
Clarification Questions for Dialogues LONG SHORT TERM MEMORY MEMORY (LSTM) BASELINE
Context LSTM o
C
Hello there there how how are you you ?
o
σ(cTMr
+ b)
Two LSTMs with tied parameters Trained to minimize crossentropy between all context, response pairs
Response LSTM
I am fine thank you you .
r
Ryan Lowe, Nissan Pow, Iulian V Serban, and Joelle Pineau. 2015. “The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems” In Special Interest Group on Discourse and Dialogue. 167
Clarification Questions for Dialogues ATTENTION OVER OVER THE CONTEXT
C
Σ
Hello there how are
you
?
168
Clarification Questions for Dialogues ATTENTION OVER OVER THE CONTEXT
C
C
Σ
Hello there how are
A
you
?
Hello there how are
you
?
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conf erence on Learning Representations . 169
Clarification Questions for Dialogues HIERARCHICAL MODEL OVER THE UTTERANCES
Token level LSTM
A
hum ,
just
mail
A
me
.
I’ll
follow the mail on the internal list
A
Yes Yes .
Aaron Courville, and Yoshua Bengio.“A hierarchical latent latent variable encoder-decoder model for generating dialogues AAAI AAAI 2017 170
Clarification Questions for Dialogues HIERARCHICAL MODEL OVER THE UTTERANCES A
C
Utterance level LSTM
Token level LSTM
A
hum ,
just
mail
A
me
.
I’ll
follow the mail on the internal list
A
Yes Yes .
Aaron Courville, and Yoshua Bengio.“A hierarchical latent latent variable encoder-decoder model for generating dialogues AAAI AAAI 2017 171
Clarification Questions for Dialogues CHARACTER TRIGRAM HISTOGRAM
What? Extract features below token level
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conf erence on Conference on information & knowledge management . ACM, ACM, pages 2333–2338. 2333–2338. 172
Clarification Questions for Dialogues CHARACTER TRIGRAM HISTOGRAM
What? Extract features below token level Why?
Misspelling, elided whitespaces, tokenization errors
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conf erence on Conference on information & knowledge management . ACM, ACM, pages 2333–2338. 2333–2338. 173
Clarification Questions for Dialogues CHARACTER TRIGRAM HISTOGRAM
What? Extract features below token level Why?
Misspelling, elided whitespaces, tokenization errors
How?
Trigram histogram
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conf erence on Conference on information & knowledge management . ACM, ACM, pages 2333–2338. 2333–2338. 174
Clarification Questions for Dialogues CHARACTER TRIGRAM HISTOGRAM Example: What? Extract features below token level
“Bananas”
ban - 1 ana - 2
Why?
Misspelling,
nan - 1
elided whitespaces,
nas - 1
tokenization errors How?
Trigram histogram
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conf erence on Conference on information & knowledge management . ACM, ACM, pages 2333–2338. 2333–2338. 175
Clarification Questions for Dialogues CHARACTER TRIGRAM HISTOGRAM Example: What? Extract features below token level
“Bananas”
ban - 1 ana - 2
Why?
Misspelling,
nan - 1
elided whitespaces,
nas - 1
tokenization errors How?
Trigram histogram
Trigram histogram 0 1 0 0 10 2 0 0 1 0 Size of trigram vocabulary
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conf erence on Conference on information & knowledge management . ACM, ACM, pages 2333–2338. 2333–2338. 176
Clarification Questions for Dialogues CHARACTER TRIGRAM HISTOGRAM Example: What? Extract features below token level
“Bananas”
ban - 1 ana - 2
Why?
Misspelling,
nan - 1
elided whitespaces,
nas - 1
tokenization errors How?
Trigram histogram 0 1 0 0 10 2 0 0 1 0
Trigram histogram
Size of trigram vocabulary
Input representation
Vocabulary Vocabulary
OOV
Trigram
OOT
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conf erence on Conference on information & knowledge management . ACM, ACM, pages 2333–2338. 2333–2338. 177
Clarification Questions for Dialogues CHARACTER TRIGRAM HISTOGRAM Example: What? Extract features below token level
ban - 1 “Bannas”
ana - 2
Why?
Misspelling,
nan - 1
elided whitespaces,
nas - 1
tokenization errors How?
“Bananas”
Trigram histogram 0 1 0 0 10 2 0 0 1 0
Trigram histogram
Size of trigram vocabulary
Input representation
Vocabulary Vocabulary
OOV
Trigram
OOT
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conf erence on Conference on information & knowledge management . ACM, ACM, pages 2333–2338. 2333–2338. 178
Clarification Questions for Dialogues
Experiments
179
Clarification Questions for Dialogues PRELIMINARY PRELIMINARY RESULTS ON NEXT UTTERANCE CLASSIFICATION
TASK: ASK:
Giv Given con conte text xt of of a con convers versat atio ion, n, select the correct response from 10 responses chosen at random
Ryan Lowe, Nissan Pow, Iulian V Serban, and Joelle Pineau. 2015. “The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems” In Special Interest Group on Discourse and Dialogue. 180
Clarification Questions for Dialogues PRELIMINARY PRELIMINARY RESULTS ON NEXT UTTERANCE CLASSIFICATION
TASK: ASK:
Giv Given con conte text xt of of a con convers versat atio ion, n, select the correct response from 10 responses chosen at random
Model
Accuracy
Recall @2
Recall @5
TF-IDF
0.48
0.58
0.76
LSTM
0.55
0.72
0.92
+ Attention
0.60
0.77
0.95
+ Hierarchical Hierarchical
0.62
0.79
0.95
+ Trigram Histogram
0.64
0.80
0.96
Ryan Lowe, Nissan Pow, Iulian V Serban, and Joelle Pineau. 2015. “The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems” In Special Interest Group on Discourse and Dialogue. 181
Talk Outline o
Problem Overview
o
Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l
o
Clarification Questions for Question-Answering forums
o
Clarification Questions for Dialogues
o
Proposed Work
o
Generalizability bey beyond Q&A forums and Dialogues
182
1. TEMPLATE BASED QUESTION QUESTION GENERATION
183
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION KEY IDEA
“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt-get are you running?”
184
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION KEY IDEA
“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt-get are you running?”
Unseen context about ‘gcc’
I built a cross compiler on Ubuntu 14.04 32 bit for MIPS platform before, but now I can’t compile ordinary C programs with GCC. I removed and reinstalled everything but every time I get this error as: unrecognized option ‘--32’
185
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION KEY IDEA
“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt-get are you running?”
Unseen context about ‘gcc’
Template:
I built a cross compiler on Ubuntu 14.04 32 bit for MIPS platform before, but now I can’t compile ordinary C programs with GCC. I removed and reinstalled everything but every time I get this error as: unrecognized option ‘--32’
What version of ____ are are you using?
186
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION KEY IDEA
“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt-get are you running?”
Unseen context about ‘gcc’
Template:
I built a cross compiler on Ubuntu 14.04 32 bit for MIPS platform before, but now I can’t compile ordinary C programs with GCC. I removed and reinstalled everything but every time I get this error as: unrecognized option ‘--32’
What version of ____ are are you using?
What version of gcc are you using?
187
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION PROPOSED METHOD
1.
Identify groups of similar questions
“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt -get are you running?” running ?”
188
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION PROPOSED METHOD
1.
Identify groups of similar questions Clustering algorithms
“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt -get are you running?” running ?”
189
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION PROPOSED METHOD
1.
Identify groups of similar questions Clustering algorithms
2.
“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt -get are you running?” running ?”
Generate a template for each group
190
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION PROPOSED METHOD
1.
Identify groups of similar questions Clustering algorithms
2.
“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt-get are you running?” running ?”
Generate a template for each group !
Iden Id enti tify fy top topic ic spe speci cifi fic c wor words ds
!
Remove them to form template
191
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION PROPOSED METHOD
1.
Identify groups of similar questions Clustering algorithms
2.
“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt-get are you running?” running ?”
Generate a template for each group !
Iden Id enti tify fy top topic ic spe speci cifi fic c wor words ds
!
Remove them to form template
What version of ____ are you using?
192
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION PROPOSED METHOD
1.
Identify groups of similar questions Clustering algorithms
2.
3.
“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt-get are you running?” running ?”
Generate a template for each group !
Iden Id enti tify fy top topic ic spe speci cifi fic c wor words ds
!
Remove them to form template
What version of ____ are you using?
Given a context, select a template templat e from a candidate candid ate set of templates templa tes
193
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION PROPOSED METHOD
1.
Identify groups of similar questions Clustering algorithms
2.
3.
“What version of Ubuntu are you using?” “What version of yum are you using?” “What version of apt-get are you running?” running ?”
Generate a template for each group !
Iden Id enti tify fy top topic ic spe speci cifi fic c wor words ds
!
Remove them to form template
What version of ____ are you using?
Given a context, select a template templat e from a candidate candid ate set of templates templa tes !
Using our EVPI model
194
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION PROPOSED METHOD
1.
Identify groups of similar questions Clustering algorithms
2.
3.
“What version of yum are you using?” “What version of apt-get are you running?” running ?”
Generate a template for each group !
Iden Id enti tify fy top topic ic spe speci cifi fic c wor words ds
!
Remove them to form template
What version of ____ are you using?
Given a context, select a template templat e from a candidate candid ate set of templates templa tes !
4.
“What version of Ubuntu are you using?”
Using our EVPI model
Fill in the blanks of the template templa te using topic words words from the context
195
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION PROPOSED METHOD
1.
Identify groups of similar questions Clustering algorithms
2.
3.
“What version of yum are you using?” “What version of apt-get are you running?” running ?”
Generate a template for each group !
Iden Id enti tify fy top topic ic spe speci cifi fic c wor words ds
!
Remove them to form template
What version of ____ are you using?
Given a context, select a template templat e from a candidate candid ate set of templates templa tes !
4.
“What version of Ubuntu are you using?”
Using our EVPI model
Fill in the blanks of the template templa te using topic words words from the context !
Identify candidate topic words from the context
196
PROPOSED WORK: TEMPLATE BASED QUESTION QUESTION GENERATION PROPOSED METHOD
1.
Identify groups of similar questions Clustering algorithms
2.
3.
“What version of yum are you using?” “What version of apt-get are you running?” running ?”
Generate a template for each group !
Iden Id enti tify fy top topic ic spe speci cifi fic c wor words ds
!
Remove them to form template
What version of ____ are you using?
Given a context, select a template templat e from a candidate candid ate set of templates templa tes !
4.
“What version of Ubuntu are you using?”
Using our EVPI model
Fill in the blanks of the template templa te using topic words words from the context !
Identify candidate topic words from the context
!
Train a model to select the correct topic word to fill in the template
197
2. SEQUENCE-TO-SEQUENCE BASED QUESTION GENERATION
198
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION KEY IDEA o
Sequence-to-sequence neural network models !
Machine Translation
!
Dialog Generation
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems 2014 Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative generative hierarchical neural network models. Proceedings of AAAI 2014 199
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION KEY IDEA o
o
Sequence-to-sequence neural network models !
Machine Translation
!
Dialog Generation
Given an input sequence, generate output sequence one word at a time
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems 2014 Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative generative hierarchical neural network models. Proceedings of AAAI 2014 200
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION KEY IDEA o
o
Sequence-to-sequence neural network models !
Machine Translation
!
Dialog Generation
Given an input sequence, generate output sequence one word at a time
A
B
C
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems 2014 Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative generative hierarchical neural network models. Proceedings of AAAI 2014 201
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION KEY IDEA o
o
Sequence-to-sequence neural network models !
Machine Translation
!
Dialog Generation
Given an input sequence, generate output sequence one word at a time W
A
B
C
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems 2014 Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative generative hierarchical neural network models. Proceedings of AAAI 2014 202
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION KEY IDEA o
o
Sequence-to-sequence neural network models !
Machine Translation
!
Dialog Generation
Given an input sequence, generate output sequence one word at a time W
A
B
C
X
W
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems 2014 Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative generative hierarchical neural network models. Proceedings of AAAI 2014 203
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION KEY IDEA o
o
Sequence-to-sequence neural network models !
Machine Translation
!
Dialog Generation
Given an input sequence, generate output sequence one word at a time W
A
B
C
X
Y
W
X
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems 2014 Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative generative hierarchical neural network models. Proceedings of AAAI 2014 204
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION KEY IDEA o
o
Sequence-to-sequence neural network models !
Machine Translation
!
Dialog Generation
Given an input sequence, generate output sequence one word at a time W
A
B
C
X
Y
Z
W
X
Y
Z
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems 2014 Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative generative hierarchical neural network models. Proceedings of AAAI 2014 205
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION Neural Generative Question Answering Model
He is 2.29 m and visible from space
How Ho w tall is Yao Ming ?
Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016. Neural generative generative question answering. answering. In North American Association of Computational Linguistics . 206
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION Neural Generative Question Answering Model
He is 2.29 m and visible from space
How Ho w tall is Yao Ming ?
Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016. Neural generative generative question answering. answering. In North American Association of Computational Linguistics . 207
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION Neural Generative Question Answering Model
Knowledge Base
He is 2.29 m and visible from space
How Ho w tall is Yao Ming ?
Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016. Neural generative generative question answering. answering. In North American Association of Computational Linguistics . 208
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION PROPOSED METHOD o
We propose to build a sequence-to-sequence model to generate generate clarification question one word at a time, given the context
Which version of Ubuntu ar are e you using?
How to configure environment variable? ….
209
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION PROPOSED METHOD o
o
We propose to build a sequence-to-sequence model to generate generate clarification question one word at a time, given the context At each time step, step, our model will decide whether to
Which version of Ubuntu ar are e you using?
How to configure environment variable? ….
210
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION PROPOSED METHOD o
o
We propose to build a sequence-to-sequence model to generate generate clarification question one word at a time, given the context At each time step, step, our model will decide whether to !
Generate a template word OR
Which version version of ____ are are you you using?
How to configure environment variable? ….
211
PROPOSED WORK: SEQUENCE-TO-SEQ SEQUENCE-TO-SEQUENCE UENCE BASED QUESTION GENERATION PROPOSED METHOD o
o
We propose to build a sequence-to-sequence model to generate generate clarification question one word at a time, given the context At each time step, step, our model will decide whether to !
Generate a template word OR
!
Generate a topic specific word from context
Which version of Ubuntu ar are e you using?
How to configure environment variable? ….
212
3. USING KNOWLEDGE SOURCES TO IDENTIFY MISSING INFORMATION
213
PROPOSED WORK: USING KNOWLEDGE SOURCES TO IDENTIFY MISSING INFORMATION
KEY IDEA
Post related to Ubuntu Operating System
214
PROPOSED WORK: USING KNOWLEDGE SOURCES TO IDENTIFY MISSING INFORMATION
KEY IDEA
Operating systems !
!
Knowledge Base
Post related to Ubuntu Operating System
215
PROPOSED WORK: USING KNOWLEDGE SOURCES TO IDENTIFY MISSING INFORMATION
KEY IDEA
Operating systems !
!
Knowledge Base
Post related to Ubuntu Operating System
What version of Ubuntu are you using?
216
PROPOSED WORK: USING KNOWLEDGE SOURCES TO IDENTIFY MISSING INFORMATION
KEY IDEA
Operating systems !
!
Knowledge Base
Songs !
!
Post related to Ubuntu
Conversation
Operating System
about a song
What version of Ubuntu are you using?
When was the song released?
217
PROPOSED WORK: USING KNOWLEDGE SOURCES TO IDENTIFY MISSING INFORMATION
PROPOSED METHOD
1. Identify topics of discussion in a given context
218
PROPOSED WORK: USING KNOWLEDGE SOURCES TO IDENTIFY MISSING INFORMATION
PROPOSED METHOD
1. Identify topics of discussion in a given context
I built a cross compiler on Ubuntu 14.04 32 bit for MIPS platform before, but now I can’t compile ordinary C programs with GCC. I removed and reinstalled everything but every time I get this error as: unrecognized option ‘--32’
219
PROPOSED WORK: USING KNOWLEDGE SOURCES TO IDENTIFY MISSING INFORMATION
PROPOSED METHOD
1. Identify topics of discussion in a given context GCC
2. Extract attribute information from Dbpedia
!
!
220
PROPOSED WORK: USING KNOWLEDGE SOURCES TO IDENTIFY MISSING INFORMATION
PROPOSED METHOD
1. Identify topics of discussion in a given context
I built a cross compiler on Ubuntu 14.04 32 bit for MIPS platform before, but now I can’t compile ordinary C programs
2. Extract attribute information from Dbpedia
with GCC. I removed and reinstalled everything but every time I get this error as:
3. Identify the missing attribute in the context
unrecognized option ‘--32’
221
PROPOSED WORK: USING KNOWLEDGE SOURCES TO IDENTIFY MISSING INFORMATION
PROPOSED METHOD
1. Identify topics of discussion in a given context
I built a cross compiler on Ubuntu 14.04 32 bit for MIPS platform before, but now I can’t compile ordinary C programs
2. Extract attribute information from Dbpedia
with GCC. I removed and reinstalled everything but every time I get this error as:
3. Identify the missing attribute in the context
unrecognized option ‘--32’
4. Ask 4. Ask a question about the missing attribute What version of gcc are you using?
222
Talk Outline o
Problem Overview
o
Expected Value Value of Perfect Information Inf ormation (EVPI) inspired inspired model mode l
o
Clarification Questions for Question-Answering forums
o
Clarification Questions for Dialogues
o
Proposed Work
o
Generalizability bey beyond Q&A forums and Dialogues
223
GENERALIZABILITY BEYOND Q&A FORUMS AND DIALOGUES INTERACTIVE QUERY SEARCH o
Traditional search approaches rely heavily on relevance feedback
224
GENERALIZABILITY BEYOND Q&A FORUMS AND DIALOGUES INTERACTIVE QUERY SEARCH o
o
Traditional search approaches rely heavily on relevance feedback However with the increase of interactive search agents like Siri, Alexa etc, we will no longer longer be clicking on links
225
GENERALIZABILITY BEYOND Q&A FORUMS AND DIALOGUES INTERACTIVE QUERY SEARCH o
o
o
Traditional search approaches rely heavily on relevance feedback However with the increase of interactive search agents like Siri, Alexa etc, we will no longer longer be clicking on links Hence there is a need for the system to narrow down on the search criteria by asking questions
226
GENERALIZABILITY BEYOND Q&A FORUMS AND DIALOGUES INTERACTIVE QUERY SEARCH o
o
o
o
Traditional search approaches rely heavily on relevance feedback However with the increase of interactive search agents like Siri, Alexa etc, we will no longer longer be clicking on links Hence there is a need for the system to narrow down on the search criteria by asking questions Ask clarification questions is one way of doing that!
227
GENERALIZABILITY BEYOND Q&A FORUMS AND DIALOGUES WRITING ASSISTANCE
Hi Marge, Let us meet tomorrow tomorrow at 10am to discuss the next group assignment? Hey Homer, H omer, Sure. Where do you want to meet though? Oh right. Forgot to mention that. Let us meet in the 3 rd floor grad lounge.
228
GENERALIZABILITY BEYOND Q&A FORUMS AND DIALOGUES WRITING ASSISTANCE
Hi Marge,
Do you want to suggest a place?
Let us meet tomorrow tomorrow at 10am to discuss the next group assignment?
Hi Marge, Let us meet tomorro tomor row w at 10am in 3rd floor grad lounge to discuss the next group assignment? Sounds good!
229
CONCLUSION !
Identify the importance of teaching machines to ask clarification clar ification questions
!
Introduce two novel datasets for this problem o o
Question-Answering Forums Dialogues
!
Introduce a novel model inspired by Expected Value of Perfect Perfect Information to rank candidate questions
!
Propose methods to generate questions (instead of select)
230
TIMELINE November 2017
Clarification Questions for Dialogues
March 2017
Template based question generation
June 2018
Sequence-to-sequence based question generation generation
September 2018
Identify missing information using u sing knowledge knowledge bases
December 2018
Thesis writing and defense
231
Sudha Rao, Yogarshi Vyas, Hal Daumé III, and Philip Resnik, ”Parser for Abstract Meaning Representation using Learning to Search", NAACL 2016 Workshop Workshop on Meaning Representation Parsing
SEMANTICS Sudha Rao, Daniel Marcu, Kevin Knight and Hal Daumé III, "Biomedical Event Extraction using Abstract Meaning Representation" In Submission
Sudha Rao, Hal Daumé III, " Are you asking the the right questions? Automatically Generating Generating Clarification Questions" In submission
Rao, Sudha, Allyson Ettinger, Hal Daumé III, and Philip Resnik. "Dialogue focus tracking for zero pronoun resolution.” NAACL N AACL 2015
QUESTION GENERATION
DIALOGUES Sudha Rao, Paul Mineiro, " A Play on Words: Redefining Vocabulary for Dialogue Modeling" In submission
232
233