Skip to content

Add VLRMBench Support #1259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Add VLRMBench Support #1259

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5cc0289
add vlrmbench dataset
Winston-Yuan Oct 1, 2025
7ca05b6
Translate comments to English
Winston-Yuan Oct 1, 2025
86fbf9b
Fix code formatting issues
Winston-Yuan Oct 2, 2025
dbea3ff
Reconstructed the dataset and the evaluation code of vlrmbench
Winston-Yuan Oct 14, 2025
ee9f8cc
Refactor and translate comments in VLRMBench dataset code to English
Winston-Yuan Oct 15, 2025
52b88d3
Consolidate dataset imports in __init__.py
Winston-Yuan Oct 15, 2025
890434b
Merge branch 'main' into main
Winston-Yuan Oct 15, 2025
9cc6032
Add multi_solution evaluation support in VLRMBench dataset
Winston-Yuan Oct 16, 2025
10dbaea
Merge branch 'main' of github.com:Winston-Yuan/VLMEvalKit
Winston-Yuan Oct 16, 2025
29ba4bb
Add foresight evaluation support in VLRMBench dataset
Winston-Yuan Oct 23, 2025
1b7aa4c
Merge branch 'main' into main
Winston-Yuan Oct 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion vlmeval/dataset/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@
from .mmifeval import MMIFEval
from .chartmimic import ChartMimic
from .m4bench import M4Bench
from .vlrmbench import VLRMBench
from .mmhelix import MMHELIX
from .medqbench_mcq import MedqbenchMCQDataset
from .medqbench_caption import MedqbenchCaptionDataset
Expand Down Expand Up @@ -212,7 +213,8 @@ def evaluate(self, eval_file, **judge_kwargs):
MMEReasoning, GOBenchDataset, SFE, ChartMimic, MMVMBench, XLRSBench,
OmniEarthMCQBench, VisFactor, OSTDataset, OCRBench_v2, TreeBench, CVQA, M4Bench,
AyaVisionBench, TopViewRS, VLMBias, MMHELIX, MedqbenchMCQDataset,
MedqbenchPairedDescriptionDataset, MedqbenchCaptionDataset, ChartMuseum, ChartQAPro
MedqbenchPairedDescriptionDataset, MedqbenchCaptionDataset, ChartMuseum, ChartQAPro,
VLRMBench
]

VIDEO_DATASET = [
Expand Down
Loading