Prompting

[bodsul]

Hello. I found that modifying the prompt to extract answer using Llama-3.1-8B-Instruct worked better (fewer -1 values returned) when modified to the slightly clearer "You are an answer extractor. When given someone's answer to some question, you will only extract the number in their answer and will respond with just the number. If there is no exact number answer, respond with -1."

I also found while trying to integrate your dataset into vlmevalkit that appending '\nPlease try to answer the question with short words or phrases if possible." to the question prompt leads to models used in the qwen2.py, molmo.py and intern.py scripts to respond with counts seemingly obviating the need to use another model to extract the answer.

Finally is it really necessary to use different prompts, in this case, for separate models?

You can take a look at the PR to add your dataset to vlmevalkit here open-compass/VLMEvalKit#974.

[atinpothiraj]

Hi, we experimented with multiple answer extractor prompts and found the one we used to be the best with zero inaccuracies over 1000 manually evaluated examples (we also ran with a different model as the extractor and found the same results). We would guess that the modified prompt doesn't actually give better scores, just more leniency to the model responses (accepting ranges of numbers, multiples numbers, etc.). If this is not the case with your tested prompt, please let us know and we will update our results accordingly.

Additionally, we found that constraining the model to respond with only numbers negatively impacted performance greatly (again, if your results differ, please let us know). Technically, different prompts are not necessary, but because models perform poorly on CAPTURe, we wanted to give each the model the best chance on CAPTURe to prove its difficulty. If you have any more questions / ideas please let us know, the feedback is greatly appreciated!