Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information

Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Expect ed Value of Perfect Perf ect Info Information rmation Sudha Rao1 and Hal Daumé III1,2 1University

of Maryland

Colle leg ge Pa Park 2Microsoft

Research

New Yor York k

CODE + DATA: https://github.com/raosudha89/ranking_clarification_questions

Natural Language Understanding

2

Natural Language Understanding

How long does it take to get a PhD?

3

Natural Language Understanding Give me a recipe for lasagna


4



Please bring me my coffee mug from f rom the kitchen

5



Please bring me my coffee mug from f rom the kitchen

6

Human Interactions

7

Human Interactions

Please bring me my coffee mug from the kitchen

8

Human Interactions


9

Human Interactions


What color is your your coffee coff ee mug?

10

Teach Machines to Ask Clarification Clarification Questions

11

Teach Machines to Ask Clarification Questions Questions

How long does it take to get a PhD ? In which field?

12



Give me a recipe for lasagna Any dietary restrictions?

13





What color is your coffee coff ee mug?

14

Teach Machines to Ask Clarification Questions Questions Context-aware questions about missing information




What color is your coffee coff ee mug?

15

Reading Comprehension Question Generation My class is going to the movies on a field trip tr ip next week. We have to get permission slips signed before we go. We are going to see a movie that tells the story from a

someone’s Goal: Assess someone’s understanding of the text

book we read.

Q: What do the students need to do before before going to the movies? movies?

o

o

Heilman. “Automatic factual question generation from text” Ph.D. thesis 2011 Vasile, Vasile, et al. "The first question question generation generation shar shared ed task evaluation evaluation challenge. challenge.” ” NLG 2010

o

Olney, Graesser, and Person. "Question generation from concept maps." Dialogue & Discourse 2012

o

Chali and Hasan. "Towards Topic-to-Question Generation." ACL 2015

o

Serban, et al. "Generating Factoid Questions With Recurrent Recur rent Neural Networks” ACL 2016

o

Du, Shao & Cardie "Learning to ask: Neural question generation for reading comprehension" ACL 2017

o

Tang et al. "Learning "Learni ng to Collaborate for Question Answering and Asking." NAACL 2018

o

Mrinmaya and Xing. "Self-Training for Jointly Learning to Ask and Answer Questions." NAACL 2018

16

Question Generation Generation for Slot Filling SLOTS USER:

I want to go to Melbourne on July 14

SYSTEM:

What time do you want to leave?

USER:

I must be in Melbourne rne by 11 am

SYSTEM SYS TEM::

Would ould you like like a Delta flight that arrive arr ives s at 10.15 am?

USER:

Sure

SYS YST TEM: EM:

In what name should I make the reservation?

o

Goddeau, et al. "A form-based dialogue manager for spoken language applications.” 1996

o

Bobrow., et al. "GUS, a frame-driven dialog system." Artificial intelligence 1977

o

Lemon, et al. "An ISU dialogue system exhibiting reinforcement learning of dialogue policies: generic slot-filling in the TALK in-car system.” EACL 2006

o

o

o

Williams, et al. The Dialog State Tracking Challenge” SIGDIAL 2013 Young Young,, et al. “Pomdp-based “Pomdp-based statistical statistical spoken spoken dialog systems: systems: A review review..” IEEE 2013 Dhingra, et al. "Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access.” Access.” ACL 2017

o

Bordes, et al. "Learning "Learni ng end-to-end goal-oriented goal-or iented dialog.” dialog.” ICLR 2017

17

Other types of Question Generation o

Liu, et al. “Automatic question generation for literature review writing support." International Conference on Intelligent Tutoring Systems. 2010

o

Penas and Hovy, “Filling knowledge gaps in text for machine reading” International Conference on Computational Linguistics: Posters ACL 2010

o

Artzi & Zettlemoyer Zettlemoyer,, “Bootstrapping semantic semantic parsers from conversations” conversations” EMNLP 2011

o

Labutov, Labutov, et al.“Deep questions question s without deep understanding” understa nding” ACL 2015

o

Mostafazadeh et al. "Generating "Generating natural questions about an image." i mage." ACL 2016

o

Mostafazadeh et al. a l. "Multimodal Context for Natural Nat ural Question and Response Generation.” Generation.” IJCNLP 2017. 2017 .

o

Rothe, Lake and Gureckis. “Question asking as program generation” NIPS 2017.

18

Talk Outline o

Clarification Questions Dataset

o

Problem Formulation: ormulatio n: Question Ranking

o

Expected Value of Perfect Information (EVPI) inspired model

o

Evaluation

o

Conclusion

19

Talk Outline o


o


o


o

Evaluation

o

Conclusion

20


StackExchange Question-Answer Forum

21


How to configure path or set environment variables for installation? I'm aiming to install ape, a simple code for f or pseudopotential generation. I'm having this error message while running ./configure So I have the library but the program installation isn't finding it. Any help? help? Thanks in advance! advance!

22



Finding: Questions go unanswered for a long time if they are not clear enough Asaduzzaman, Asaduzzaman, Muhammad, et al. "Answering "Answering questions about about unanswer unanswered ed questions of stack overflo overflow w.” Working Conference on Mining Software Repositories. IEEE Press, 2013.

23



Question comment

What version of ubuntu do you have?

24


How to configure path or set environment variables for installation? I'm aiming to install ape, a simple code for f or pseudopotential generation.

Initial Post

I'm having this error message while running ./configure

Question comment

So I have the library but the program installation isn't finding it. Any help? help? Thanks in advance! advance! What version of ubuntu do you have? I'm aiming to install ape in Ubuntu 14.04 LTS, a simple code for pseudopotential generation. I'm having this error message while running ./configure

Updated Post

So I have the library but the program program installation isn't finding it. Any help? Thanks in advance!

25



Initial Post


Question comment

So I have the library but the program installation isn't finding it. Any help? help? Thanks in advance! advance! What version of ubuntu do you have? I'm aiming to install ape in Ubuntu 14.04 LTS, a simple code for pseudopotential generation.

Edit as an answer to the question


Updated Post


26



Initial Post


Question comment

So I have the library but the program installation isn't finding it. Any help? help? Thanks in advance! advance! What version of ubuntu do you have? I'm aiming to install ape in Ubuntu 14.04 LTS, a simple code for pseudopotential generation.

Edit as an answer to the question


Updated Post


27

Clarification Questions Dataset Dataset Creation

(

post

,

question

,

answer

) triples

post

Original post

question

Clarification question posted in comments

answer

Edit made to the post in response response to the question OR author’s author’s rreply eply to the question comment

Dataset Size: ~77 K triples trip les Domains:

Askubuntu, Unix, Superuser

Note: We identify a question questio n using the question mark (?) token. We match the edit to the answer using timestamp & word word embedding similarity based heuristics. 28

Talk Outline o


o


o


o

Evaluation

o

Conclusion

29

Problem Formulati ormulation: on: Question Question Ranking Ranking

Post

How to configure path or set environment variables? …

30

Problem Formulati ormulation: on: Question Question Ranking Ranking Generate Question Candidates Post


What is the make of your wifi card?

What version of Ubuntu do you have?

What OS are you using?

31

Problem Formulati ormulation: on: Question Question Ranking Ranking Generate Question Candidates Post



What version of Ubuntu do you have?


Rank the question candidates What version of Ubuntu do you have?



32

Problem Formulati ormulation: on: Question Question Ranking Ranking


What version of Ubuntu do you have? How are you installing ape?

Shortlist of useful questions

Do you have GSL installed?

33

Talk Outline o


o


o


o

Evaluation

o

Conclusion

34


Key Idea How to configure path or set environment variables for installation? I'm aiming to install ape, a simple code for f or pseudopotential generation. I'm having this error message while running ./configure So I have the library but the program installation isn't finding it. Any help? help? Thanks in advance! advance!

Possible questions (a) What version of Ubuntu do you have?

!

Just right

35



Possible questions (a) What hat vers versio ion n of Ubun Ubuntu tu do you have? ve?

!

(b) What is the make of your wifi card?

!

Just right Not useful

36



Possible questions (a) What hat vers versio ion n of Ubun Ubuntu tu do you have? ve?

!

(b) What is the make of your wifi card?

!

(c) Are you running Ubuntu Ubuntu 14.10 kernel 4.4.0 4.4.0-59-59! generic on an x86 64 architecture?

Just right Not useful

Unlikely to add value

37


o

Use EVPI to identify questions that add the most value to the given post

Avriel, Avriel, Mordecai, Mordecai, and A. C. Williams. "The value of information and stochastic programming." programming." Operations Research 18.5 (1970)

38


o


o

Definition: Definit ion: Value of Perfect Information Infor mation VPI (x) How How much value value does x add add to a given information content content c?


39


o


o


o

Since we have not acquired x, we define its value in expectation


40


o


o


o

Since we have not acquired x, we define its value in expectation Likelihood of x given c

EVPI (x|c) =

P (x|c) Utility(x, Utility (x, c) x

X Value Value of updating c with x


41


EVPI formulation for our problem

42



EVPI ( qi | p

)

p

: given post

qi

: question from set of question candidates Q

43



Likelihood of a j being the answer to q i on post p

EVPI ( qi | p

)=

P(

a j |

p

, qi

)

p

: given post

qi


44




EVPI ( qi | p

)=

P(

a j |

p

, qi

) U(

p

+

a j

)

Utility of updating the post p with answer a j

p

: given post

qi


45




EVPI ( qi | p

)= a j

P(

a j |

p

, qi

) U(

p

+

a j

)

A Utility of updating the post p with answer a j

p

: given post

qi

: question from from set of question candidates Q

a j

: answer from set of answer candidates A

46


EVPI formulation for our problem We W e rank questions based on their EVPI value Likelihood of a j being the answer to q i on post p

EVPI ( qi | p

)= a j

P(

a j |

p

, qi

) U(

p

+

a j

)


p

: given post

qi


a j


47




EVPI ( qi | p

)= a j

P(

a j |

p

, qi

) U(

p

+

a j

)


p

: given post

qi


a j


1

48



2 EVPI ( qi | p


)= a j

P(

a j |

p

, qi

) U(

p

+

a j

)


p

: given post

qi


a j


1

49



2 EVPI ( qi | p

)= a j

P(

Answer Modeling Modeling

a j |

p

, qi

) U(

p

+

a j

)


p

: given post

qi


a j


1

50



2 EVPI ( qi | p

)= a j

P(


a j |

p

, qi

) U(

p

+

)

A 3

Utility of updating the post p with answer a j

p

: given post

qi


a j

a j

: answer from set of answer candidates candidat es A

1

51



2 EVPI ( qi | p

)= a j

P(


a j |

p

) U(

, qi

p

+

)

A 3

Utility Calculator

p

: given post

qi


a j

a j


1

52


1

Question & Answer Candidate Generator

53


1


Dataset of (post, question, answer)

Post as Documents

Post p as query

Lucene Search Engine

54


1



Ten posts similar to given post p p1

Post as Documents

Post p as query


p2

p j

p10

55


1



Post as Documents

Post p as query


Ten posts similar to given post p

Questions paired with those posts

p1

q1

p2

q2

p j

q j

p10

q10

56


1



Post as Documents

Post p as query




Answers paired Answers with those posts

p1

q1

a1

p2

q2

a2

p j

q j

a j

p10

q10

a10

57


1



Post as Documents

Post p as query




Answers paired Answers with those posts

p1

q1

a1

p2

q2

a2

p j

q j

a j

p10

q10

a10

58


2

EVPI ( qi | p

)= a j

Answer Modeling P(

a j |

p

, qi

) U(

p

+

a j

)

A

59


2 P(

a j |

p

, qi

Answer Modeling

)

60


2 P(

a j |

p

, qi

Answer Modeling

)

p

qi

Post LSTM

Question LSTM

Word embedding module

p

qi

61


2 P(

a j |

p

, qi

Answer Modeling

)

Embans(

p

,

qi

)

Feedforward

p

qi

Post LSTM

Question LSTM


p

qi

62


2 P(

a j |

p

, qi

Answer Modeling

)

!

cosine_sim(Embans(

p

,

qi

),

Feedforward

p

qi

Post LSTM

Question LSTM

a j

)

Averag Average e


p

qi

a j

63


2 P(

Embans(

p

a j |

, qi

)

p

, qi

Answer Modeling

)

!

cosine_sim(Embans(

p

,

qi

),

Feedforward

a j

)

Averag Average e

Training p

qi

Post LSTM

Question LSTM


p

qi

a j

64


2 P(

Embans(

p

a j |

, qi

p

)

Close to true a i paired with p ai : Ubuntu 14.04 LTS qi : Which version of

, qi

Answer Modeling

)

!

cosine_sim(Embans(

p

,

qi

),

Feedforward

a j

)

Averag Average e

Training p

qi

Post LSTM

Question LSTM

Ubuntu do you have? Word embedding module

p

qi

a j

65


2 P(

Embans(

p

a j |

, qi

p

)

Close to true a i paired with p ai : Ubuntu 14.04 LTS

, qi

Answer Modeling

)

!

cosine_sim(Embans(

p

,

qi

),

Feedforward

a j

)

Averag Average e

Training Close to ak paired with qk similar to true qi

qi : Which version of

p

qi

Post LSTM

Question LSTM

Ubuntu do you have? ak : Ubuntu 11.10 qK : What OS are you using?


p

qi

a j

66


Utility Calculator

3

EVPI ( qi | p

)= a j

P( A

a j |

p

, qi

) U(

p

+

a j

)


3

Utility Calculator U( p

+

a j

)


3

Utility Calculator Value V alue between between 0 and 1

U( p

+

a j

)

Feedforward

p

qi

Post LSTM

Question LSTM

a j

Answer Answer LSTM


p

qi

a j


3


U( p Training

+

a j

)

Feedforward

p

qi

Post LSTM

Question LSTM

a j

Answer Answer LSTM


p

qi

a j


3


U( p Training

ai : Ubuntu 14.04 LTS

a j

)

Feedforward

Label qi : Which version of Ubuntu do you have?

+

y=1 p

qi

Post LSTM

Question LSTM

a j

Answer Answer LSTM


p

qi

a j


3


U( p Training

ai : Ubuntu 14.04 LTS q j : What OS are you using? a j : Ubuntu 11.10

qk : What is the make of your wifi card? ak : TP-Link TL-WDN4800 TL-WDN480 0

a j

)

Feedforward


+

y=1 p

qi

Post LSTM

Question LSTM

a j

y=0 Answer Answer LSTM


y=0 p

qi

a j


3


U( p Training (Minimize binary cross-entropy)

ai : Ubuntu 14.04 LTS q j : What OS are you using? a j : Ubuntu 11.10

qk : What is the make of your wifi card? ak : TP-Link TL-WDN4800 TL-WDN480 0

a j

)

Feedforward


+

y=1 p

qi

Post LSTM

Question LSTM

a j

y=0 Answer Answer LSTM


y=0 p

qi

a j


1

2

3


Answer Modeling

Utility Calculator

74


1

2

3


Answer Modeling

Utility Calculator

Train time behavior: For each eac h (p, q, a) in our train trai n set 1. Generate question candidates (Q) and answer candidates (A) 2. Train Answer Model and Utility Calculator using joint loss loss function : lossans (p, q, a, Q) + lossutil (y, p, q, a)

75


1

2

3


Answer Modeling

Utility Calculator

Train time behavior: For each eac h (p, q, a) in our train trai n set 1. Generate question candidates (Q) and answer candidates (A) 2. Train Answer Model and Utility Calculator using joint loss loss function : lossans (p, q, a, Q) + lossutil (y, p, q, a)

Test time behavior: Given a post from our test set 1. Generate question candidates (Q) and answer candidates (A) 2. Calculate P(a j |p, qi) for each q 3. Calculate U(p + a j) for each a

Q using Answer Model A using Utility Calculator

4. Rank questions by EVPI (q i | p) = P(a j | p, qi) U(p + a j) a j A

76

Talk Outline o


o


o


o

Evaluation

o

Conclusion

77

Evaluation Too much disk read/write when launching an application I have Xubuntu 13.04 on an old Dell Inspiron.

Post

When I launch an application it takes a pretty long time to be launched and I see a lot of disk read/write. read/wr ite. If the system was short on memory, this would be understandable as the system would use swap. But that's not the case in my situation (i.e. I have this problem even when the RAM is almost empty).

Question Candidates 1.

How much ram do you have installed ? and what size it the swap disk partition ?

2.

If you do not have any problem with getting a little techy then may i suggest a method ?

3.

How is it slow exactly ? boot time ? hdd read/write ? cpu time ? graphics rendering ?

4.

What is the longest time you have let it run ?

5.

This may be a silly question but ... did you make your usb stick bootable ?

6.

Do your system were recently updated ?

7.

Why not have two ssds in raid 1 for redundancy ?

8.

Is that a `parted -- list` on the synology device ?

9.

Can you tell us a little about your configuration ?

10. Did you turn hardware virtualization on in bios/efi ?

78

Evaluation Too much disk read/write when launching an application I have Xubuntu 13.04 on an old Dell Inspiron.

Post


Question Candidates

Contains more than one good question!

1. How much much ram ram do you you have have installed installed ? and and what what size it the the swap swap disk partition partition ? 2.

If you do not have any problem with getting a little techy then may i suggest a method ?

3. How is it slow slow exactl exactly y ? boot time time ? hdd read/w read/write rite ? cpu time time ? graphics graphics rende rendering ring ? 4.

What is the longest time you have let it run ?

5.

This may be a silly question but ... did you make your usb stick bootable ?

6.

Do your system were recently updated ?

7.

Why not have two ssds in raid 1 for redundancy ?

8.

Is that a `parted -- list` on the synology device ?

9. Can you tell us us a little about about your your configura configuration tion ? 10. Did you you turn hardware virtualization on in bios/efi bios/efi ?

79

Evaluation

Evaluation set creation process o

We recruit 10 Unix admin experts using UpWork

o

Given a post and the set of ten question candidates "

Mark the one best question

"

Mark any other valid questions

80

Evaluation



o



"


o

We annotate a total of 500 posts from our test set

o

Each post is annotated by two experts

81

Evaluation



o



"


o

We annotate a total of 500 posts from our test set

o

Each post is annotated by two experts

o

Union of Bests: Questions Question s marked as best by by either of the annotators annotat ors

o

Intersection of Valids: Questions Questio ns marked marked as valid by both annotators

Best

Valid Valid

A1 A2 A1 A2

Union of Bests: { Q2, Q3 }

Q1 Q2

Intersection Intersec tion of Valids: { Q1, Q3, Q5 }

Q3 Q4 Q5 82

Evaluation

Baseline Models 1. Random: Randomly permute the 10 candidate questions

83

Evaluation

Baseline Models 1. Random: Randomly permute the 10 candidate questions 2. Ba Bagg-of of-n -ngr gram ams: s: Train linear classifier using bag-of-ngrams bag-of-ngrams of p, q and a

84

Evaluation

Baseline Models 1. Random: Randomly permute the 10 candidate questions 2. Ba Bagg-of of-n -ngr gram ams: s: Train linear classifier using bag-of-ngrams bag-of-ngrams of p, q and a 3. Co Comm mmun unit ity y QA (Nakov et al., 2017) : o o

o

SemEval Task: Rank comments by relevance relevance to post on Qatar Living Winning model: Logistic regression regression trained with string similarity & word embedding based features (Nandi et al., 2017) Our baseline: baseli ne:

We retrain this model on our dataset datase t

85

Evaluation


o



4. Neur Neural al (p (p,, q) Feedforward

p

qi

Post LSTM

Ques LSTM

Value V alue between between 0 and 1

Word embedding module p

qi 86

Evaluation


o



4. Neur Neural al (p (p,, q)


Feedforward

5. Neur Neural al (p (p,, q, q, a) a) pi

qi

Post LSTM

Ques LSTM

ai Ans LSTM


pi

qi

ai 87

Evaluation


o



4. Neur Neural al (p (p,, q)


Feedforward

5. Neur Neural al (p (p,, q, q, a) a) pi

qi

Post LSTM

Ques LSTM

ai Ans LSTM

Both Neural (p, q, a) and EVPI (q | p, a) have similar no. of parameters


pi

qi

ai 88

Evaluation

RESULTS Union of Best EVPI Neural (p, q, a) Neural (p, q) Community QA Bag-of-ngrams Random 0

Precision @1

89

Evaluation

RESULTS Union of Best EVPI Neural (p, q, a) Neural (p, q) Community QA Bag-of-ngrams

19.4

Random

17.5 0

5

10

15

20

25

30

35

40

Precision @1

90

Evaluation

RESULTS Union of Best EVPI Neural (p, q, a)

25.2

Neural (p, q) Non-linear vs linear Community QA Bag-of-ngrams

19.4

Random

17.5 0

5

10

15

20

25

30

35

40

Precision @1

91

Evaluation


25.2

Neural (p, q)

21.9

Explicitly modeling “answer” is useful

Community QA Bag-of-ngrams

19.4

Random

17.5 0

5

10

15

20

25

30

35

40

Precision @1

92

Evaluation


25.2

Neural (p, q)

21.9 Both use only (p, q)

Community QA

23.1

Bag-of-ngrams

19.4

Random

17.5 0

5

10

15

20

25

30

35

40

Precision @1

93

Evaluation

RESULTS Union of Best EVPI

27.7

Neural (p, q, a)

25.2

Neural (p, q)

21.9

Community QA

23.1

Bag-of-ngrams

19.4

Random

17.5 0

5

10

15

20

25

30

35

40

Precision @1 Diffe rence between EVPI and all baselines is statistically significant with p < 0.05 Note: Difference 94

Evaluation

RESULTS Union of Best EVPI

27.7

Neural (p, q, a)

Mainly differ in their loss function

25.2

Neural (p, q)

21.9

Community QA

23.1

Bag-of-ngrams

19.4

Random

17.5 0

5

10

15

20

25

30

35

40


Evaluation

RESULTS Intersection Intersec tion of Valids

Union of Best

EVPI

36.1

27.7

Neural (p, q, a)

34.4

25.2

Neural (p, q)

31.6

21.9

Community QA

33.6

23.1

Bag-of-ngrams

25.6

19.4

Random

26.4

17.5

0

5

10

15

20

25

30

35

40


Evaluation

RESULTS Intersection Intersect ion of Valids

Union of Best 32.2

EVPI

23.4 31.8

Neural (p, q, a)

22.7

Not statistically 30

Neural (p, q)

20.9

significant

30.8

Community QA

21.2 27.6

Bag-of-ngrams

19.4 26.4

Random

17.5

0

5

10

15

20

25

30

35

Precision @3

97

Talk Outline o


o


o


o

Evaluation

o

Conclusion

98

CONCLUSION #

Key Contributions "

Create a dataset of ~77K clarification questions (and answers) with context

"

Introduce novel model that integrates deep learning with classic notion of expected value of perfect information

"

Create an evaluation set of size 500 with expert human annotations

99

CONCLUSION #

Key Contributions "


"


" #


Key findings have multiple good clarification question " A context can have "

Explicitly modeling the answer helps in identifying good questions

"

EVPI formalism provides leverage over similarly expressive feedforward network

100

CONCLUSION #

Key Contributions "


"


" #


Key findings have multiple good clarification question " A context can have

#

"


"


Future work "

Sequence-to-sequence based question generation generation model

"

Multi-turn question generation

"

How How to automatically evaluate performance?

101

CONCLUSION #

Key Contributions "


"


" #


Key findings have multiple good clarification question " A context can have

#

"


"


Future work "

Sequence-to-sequence based question generation generation model

"

Multi-turn question generation

"

How How to automatically evaluate performance?

CODE + DATA: https://github.com/raosudha89/ranking_clarification_questions

102

Backup Slides

103

Evaluation

RESULTS Precision @ 1 Intersection Intersec tion of Valids

Union of Best

EVPI

36.1

27.7

Neural (p, q, a)

34.4

25.2

Neural (p, a)

32.3

24.1

Neural (p, q)

31.6

21.9

Community QA

33.6

23.1

Bag-of-ngrams

25.6

19.4

Random

26.4

17.5

0

5

10

15

20

25

30

35

40

104

Evaluation

RESULTS Precision @3 Intersection Intersec tion of Valids

Union of Best

EVPI

32.2

23.4

Neural (p, q, a)

31.8

22.7

Neural (p, a)

31.5

23.5

Neural (p, q)

20.9

Community QA

21.2

Bag-of-ngrams

26.4

17.5

0

5

10

15

30.8 27.6

19.4

Random

30

20

25

30

35

105

Evaluation

RESULTS Mean Average Precision Intersection Intersec tion of Valids

Union of Best

EVPI

49.2

43.6

Neural (p, q, a)

42.5

Neural (p, a)

46.5

41.4

Neural (p, q)

45.5

39.2

Community QA

40.2

Bag-of-ngrams

34.4

Random

35.2

0

10

20

30

47.7

47

42.7 42.1

40

50

60

106

Evaluation

RESULTS True Question EVPI

21.4

Neural (p, q, a)

20.5

Neural (p, q)

15.4

Community QA

18.5

Bag-of-ngrams

10.7

Random

10 0

5

10

15

20

25

30

35

40

Precision Diffe rence between EVPI and all baselines is statistically significant with p < 0.05 Note: Difference 107

Evaluation

RESULTS Union of Best (with true removed) EVPI

23.7

Neural (p, q, a)

22.2

Not statistically

Neural (p, a)

22.6

Neural (p, q)

significant

20.6

Community QA

22.6

Bag-of-ngrams

16.3

Random

17.4

0

5

10

15

20

25

30

Precision @1

108

SAMPLE OUTPUT Too much disk read/write when launching an application I have Xubuntu 13.04 on an old Dell Inspiron.

Post


EVPI value

Ranking of Question Candidates

Best

Valid

0.21

How much ram do you have installed? and what size is the swap disk partition

0.18

Can you tell us a little about your configuration ?

0.17

What is the longest time you have have let it run ?

0.11

How is it slow exactly ? boot time ? hdd read/write ? cpu time ?

0.00

If you do not have any problem with getting a little techy may i suggest a method ?

0.00

This may be a silly question but ... did you make your your usb stick bootable ?

0.00

Do your your system were were recently recently updated ?

0.00

Why not have two ssds in raid 1 for redundancy redundancy ?

0.00

Is that a `parted -- list` on the synology synology device ?

0.00

Did you turn hardware hardware virtualization on in bios/efi ?

109

SAMPLE OUTPUT No wifi after restart in Ubuntu 16.04 After upgrading upgrading to 16.04, 16.04, there there is no wifi whene whenever ver I restart restart the system. system. My wireless wireless interface interfa ce of Ubuntu is RT3290 Wireless 802.11n 1T/1R PCIe On iwconfig I got the following eth0 no wireless extensions …. Currently to start wifi again I have to shutdown, then boot the system again. How to fix the problem?

EVPI value

Ranking of Question Candidates

Best

Valid Valid

0.24

I doubt it, shutdown and reboot are exactly identical! are you really rebooting?

0.13

Be clear about the problem. Is Ubuntu not showing them even though they are present?

0.11

What is 4g wifi connection? connection?

0.09

Can you type `iwconfig` in terminal and paste what it returns here?

0.09

What does this tell us?

0.08

If I post it as an answer, would you kindle mark as such?

0.06

Which Which Ubuntu Ubuntu 15?

0.06

What exactly exactly do you you mean by make fails?

0.05

Welcome to ask Ubuntu! ; - ) Is the wireless lan disabled in the bios?

0.00

Is Ubuntu detecting your wireless card ? **iwconfig** does list your card?

110


LONG SHORT TERM MEMORY (LSTM)

Sepp Hochreiter and J¨urgen Schmidhuber. 1997. Long short-term memory. Neural computation , 9(8):1735–1780. Jeffrey Jeffrey Pennington, Richard Socher, Socher, and Christopher D Manning. 2014. “Glove: Global vectors for word word representation” representation” In Empirical Methods on Natural Language Processing. 111

Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information

Recommend Documents