MIRACL

Monolingual ad hoc retrieval over Wikipedia passages in 18 languages, built from native-speaker queries and relevance judgments.

Also known as: Making a MIRACL, Multilingual Information Retrieval Across a Continuum of Languages, MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages

unassessed

What it measures

MIRACL tests whether a retrieval or embedding system can rank passages by relevance to a query within the same language, across 18 typologically diverse languages, most of them under-served by earlier retrieval benchmarks built mainly for English. For each language, the corpus is that language's Wikipedia, split into passages; native speakers of the language wrote the queries and judged which passages were relevant, rather than translating an English query set. It is monolingual retrieval (query and corpus share a language), not cross-lingual retrieval, and it is text-only, covering Arabic, Bengali, German, English, Spanish, Persian, Finnish, French, Hindi, Indonesian, Japanese, Korean, Russian, Swahili, Telugu, Thai, Yoruba and Chinese.

Task format

Given a query in one language, rank a passage corpus drawn from that language's Wikipedia by relevance to the query; relevance judgments were produced by native-speaker annotators per language, not machine-translated from a single source language.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub