A 11,698-question multi-image multiple-choice benchmark spanning 52 tasks and 7 image-relationship types, scored by accuracy.
unassessed
| Category | multimodal |
|---|---|
| Subcategory | multi-image multiple-choice understanding |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 11698 |
| Dataset licence | CC-BY-4.0 |
| Publisher | OpenGVLab (Shanghai AI Laboratory and collaborators) |
MMIU tests whether a vision-language model can answer a multiple-choice question about several images at once. Items cover seven relationship types, from low-level forensic detection through semantic correspondence, temporal ordering, and 2D/3D spatial tasks. Each sample is a question, an option list, a list of image paths, and a letter answer. Table 1 of arXiv:2408.02718 names five modalities (image, text, video, point cloud, depth). Language of the questions is English.
Multiple-choice with images. inspect_evals builds a user message from the question field plus the image list, uses Inspect's multiple_choice solver, and scores with choice() plus per-task accuracy. Shuffle defaults to true. A task_name filter can run one of the inspect task keys. Authors recommend VLMEvalKit as another runner.
No model card in ModelSpec reports this benchmark yet.