Part3 Evaluation

[bruluk5w]

Hi,
I'm currently working on part 3 and I'm a bit confused. The part is called "Extractive QA" and the first 3 bullet points ask to implement extractive qa using the transformers library with a pretrained model on Part-2/msmarco_tuples.test.tsv and Part-3/msmarco-fira-21.qrels.qa-tuples.tsv. But the last bullet point seems to ask for an evaluation of a reranking model that is actually created in Part-2. Am I understanding this correct?
So do I understand correctly that there is no evalutation of the extractive QA pipeline asked?
What content is expected in the report about the extractive QA pipeline (after all there are 30 points to gain from it).

[bruluk5w]

Anyone? From the lecture team maybe? @sebastian-hofstaetter

[sophiaalthammer]

Hi,
sorry for the late response! The last point asks for an evaluation of the extractive QA pipeline for the msmarco-fira-21.qrels.qa-answers.tsv as queries and the msmarco-fira-21.qrels.qa-tuples.tsv as labels. For the extractive QA part you should use a re-ranking model from the previous Part to retrieve a ranked list of relevant passages per query. From this ranked list take the Top1 ranked passage and do the extractive QA for the query and its top1 passage.
As there are gold labels for the extractive QA part for msmarco-fira-21.qrels.qa-tuples.tsv this is the evaluation of the QA-pipeline and yes we ask for evaluation of the QA pipeline.

[sophiaalthammer]

As content in the report we expect a brief description of the implementation, a brief description of the extractive QA model which you use and the evaluation results.

[bruluk5w]

Thank you :)

[DezignDezire]

Haha, thanks for clarifying this after we have fine tuned several BERT and Roberta models :)

[sophiaalthammer]

No worries, the exercise setting is quite open and if you do some extra work, just document it and you will get some bonus points for it!