Answer extraction when using direct prompt. #32
Description
Hi, thanks for your brilliant code, there are some issues confused me in the repo.
- How can we extract final QA answer when using direct prompt (with Flan-t5-xl model)? When utilizing direct prompt, the model tends to provide a short answer only instead of output contents like "The answer is ...", does that mean we have to run until reach the max_iter?
- For direct prompting, The model can't provide CoT outputs, that makes output sequences in different timesteps incoherently, sometimes the model just generate same words and the results are not good. Below is a simple prompt demos in different iters.
"""
First iter:
Wikipedia Title: Thomas Boylston Adams (1772–1832)
Thomas Boylston Adams (September 15, 1772 – March 13, 1832) was the third and youngest son of John and Abigail (Smith) Adams.
Wikipedia Title: Those Who Love (novel)
Those Who Love is a biographical novel of John Adams, as told from the perspective of his wife, Abigail Adams. It was written by American author Irving Stone.
Q: Answer the following question.
Who was the wife of the president when William Bainbridge started his career?
A:Abigail Adams
Second iter:
Wikipedia Title: Thomas Boylston Adams (1772–1832)
Thomas Boylston Adams (September 15, 1772 – March 13, 1832) was the third and youngest son of John and Abigail (Smith) Adams.
Wikipedia Title: Those Who Love (novel)
Those Who Love is a biographical novel of John Adams, as told from the perspective of his wife, Abigail Adams. It was written by American author Irving Stone.
Q: Answer the following question.
Who was the wife of the president when William Bainbridge started his career?
A:Abigail Adams Abigail Adams
Third iter:
Wikipedia Title: Thomas Boylston Adams (1772–1832)
Thomas Boylston Adams (September 15, 1772 – March 13, 1832) was the third and youngest son of John and Abigail (Smith) Adams.
Wikipedia Title: Those Who Love (novel)
Those Who Love is a biographical novel of John Adams, as told from the perspective of his wife, Abigail Adams. It was written by American author Irving Stone.
Q: Answer the following question.
Who was the wife of the president when William Bainbridge started his career?
A:Abigail Adams Abigail Adams Abigail Adams
...
"""