A bilingual vision-language multiple-choice benchmark with CircularEval over 20 ability dimensions; English and Chinese Dev and Test splits.
unassessed
| Category | multimodal |
|---|---|
| Subcategory | bilingual vision-language multiple-choice with CircularEval |
| Page status | active |
| Metric | CircularEval accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 2948 |
| Dataset licence | Apache-2.0 |
| Publisher | OpenCompass / Shanghai AI Laboratory, with CUHK, NUS and Zhejiang University |
MMBench tests whether a vision-language model can answer a multiple-choice question about an image across a taxonomy of perception and reasoning skills, rather than on one downstream task such as captioning or VQA. Questions are written in English and in a matched Chinese translation. Level-1 abilities are Perception and Reasoning; those split into six level-2 groups and twenty level-3 skills. Images plus text; not video.
Multiple-choice with a single gold option. Free-form model text is mapped to A/B/C/D first by rule matching, then by an LLM choice extractor. Paper v5 uses GPT-4 (gpt-4-0125 by default); the GitHub README still says ChatGPT. CircularEval repeats each N-option item N times with rotated choices; the item counts only if every pass is correct. OpenCompass's in-tree config wraps VLMEvalKit MMBench_DEV_EN only.
No model card in ModelSpec reports this benchmark yet.