{
 "body": "\n## What it measures\n\nMIRACL (Multilingual Information Retrieval Across a Continuum of Languages) tests whether a\nretrieval or embedding system can rank passages by relevance to a query within one language, across\n18 typologically diverse languages that earlier retrieval benchmarks, built mainly around English,\nleft under-served. For each language, the corpus is that language's Wikipedia, split into passages;\ncrucially, native speakers of each language wrote the queries and judged relevance themselves,\nrather than a single English query set being machine-translated outward. The task is monolingual\nretrieval \u2014 query and corpus share a language \u2014 not cross-lingual search, and it is text-only.\n\nThe 18 languages are Arabic, Bengali, German, English, Spanish, Persian, Finnish, French, Hindi,\nIndonesian, Japanese, Korean, Russian, Swahili, Telugu, Thai, Yoruba and Chinese. Coverage is\ndeliberately uneven with mainstream NLP benchmarking: languages like Swahili, Telugu and Yoruba have\ncomparatively little prior retrieval-training data, which is part of the point of the benchmark.\n\n## How it is scored\n\nSystems are ranked by nDCG@10, normalised discounted cumulative gain over the top 10 results,\ncomputed separately per language against the native-speaker relevance judgments; the original paper\nand shared task also reported Recall@100. nDCG@10 is conventionally bounded 0 to 1 in information\nretrieval, but MIRACL numbers reported through MTEB, where most current scores are read, are\ntypically shown on a 0-100 scale instead \u2014 a display difference to check before comparing two\nsources. There is no single official cross-language average; reporters choose which languages, and\nwhich split, to summarise.\n\n## Dataset and licence\n\nMIRACL provides, per language, a Wikipedia-derived passage corpus, a set of native-speaker-authored\nqueries, and relevance judgments over query-passage pairs. Train and development splits are public\nfor every language with judgments, with per-language counts ranging from roughly 2,863 to 4,683\ntraining queries and 213 to 1,271 development queries, and 1,606 to 41,358 relevance judgments. A\nfurther \"testA\" split exists only for the languages MIRACL shares with the earlier Mr. TyDi\nbenchmark. The project-miracl/miracl toolkit and repository code are licensed Apache-2.0; MTEB's\ntask metadata separately lists the dataset content itself as CC BY-SA 4.0, consistent with the\nlicence of the underlying Wikipedia text.\n\n## Who publishes it\n\nMIRACL was introduced by Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo and Jimmy\nLin at the David R. Cheriton School of Computer Science, University of Waterloo, together with\nDavid Alfonso-Hermelo, Xiaoguang Li, Qun Liu and Mehdi Rezagholizadeh at Huawei Noah's Ark Lab. An\narXiv preprint, then titled \"Making a MIRACL,\" appeared in October 2022 alongside the WSDM 2023 Cup\nshared task the dataset was built for; the peer-reviewed version, \"MIRACL: A Multilingual Retrieval\nDataset Covering 18 Diverse Languages,\" appeared in Transactions of the Association for\nComputational Linguistics in September 2023. The authors maintain the reference repository at\ngithub.com/project-miracl/miracl.\n\n## Lineage\n\nMIRACL's closest predecessor within the same research group is Mr. TyDi, an earlier, smaller\nmultilingual retrieval effort by overlapping authors \u2014 MIRACL's own \"testA\" split exists only for\nthe languages the two benchmarks share \u2014 though Mr. TyDi does not have a page in this repository.\nMIRACL's most consequential downstream relationship is with [MTEB](mteb.md) (Massive Text Embedding\nBenchmark), which implements it directly as a retrieval task, \"MIRACLRetrieval,\" inside its\nmultilingual retrieval collection (plus a hard-negatives variant built by pooling BM25 and embedding\nmodel results). In this repository's own census of reported scores, current MIRACL numbers are\ntypically read off the MTEB leaderboard rather than a standalone MIRACL-only leaderboard, which\nmakes MTEB the practical successor for day-to-day score reporting even though MIRACL's own toolkit\nstill runs standalone.\n\n## Saturation and contamination\n\nMIRACL is 18 separate per-language retrieval tasks rather than one scale, so a single saturation\nverdict is not meaningful; difficulty varies with each language's corpus size, query style and how\nmuch retrieval-training data exists for it already. This repository found no source assessing\nwhether frontier embedding models have collectively saturated MIRACL across all 18 languages, so an\noverall status is not established here. Contamination risk sits at medium: every language's corpus\nis drawn from that language's public Wikipedia, and the labelled train and development queries have\nbeen public since the October 2022 release, so both sides of the retrieval task are exposed to\nanything trained on web text since then. Whether any genuinely hidden test material still exists\nfrom the original WSDM 2023 Cup shared task was not established; current scores are read from the\npublic development split.\n\n## How to run it\n\nThe reference implementation is the project-miracl/miracl GitHub repository, built on the Pyserini\ntoolkit for indexing and evaluation. MIRACL is not one of the tasks bundled with lm-evaluation-harness,\ninspect_evals, HELM, OpenCompass or BIG-bench. In practice, most current scores for embedding models\nare produced through the mteb Python package, which implements MIRACL as the \"MIRACLRetrieval\" task\n(plus a \"MIRACLRetrievalHardNegatives\" variant) within its multilingual retrieval group, evaluating\non the development split with nDCG@10 as the headline metric. Because embedding dimensionality,\nmaximum sequence length, pooling method, and whether a task-specific instruction prefix was\nprepended to the query all affect retrieval scores and vary between reporters, MIRACL numbers from\ndifferent sources are not guaranteed to be comparable.\n\n## Reading the numbers\n\nA strong average MIRACL score suggests an embedding model retrieves well across a genuinely\ndiverse set of languages and scripts, not just the handful of high-resource languages most retrieval\nmodels are tuned on \u2014 which is the benchmark's central point. It does not say the model performs\nwell on the specific language, domain or document type a real deployment needs, since per-language\nscores vary substantially and an average can hide a model that is strong on, say, English and\nChinese but weak on Swahili or Yoruba. Check the per-language breakdown rather than a blended\naverage, note whether the score came from MIRACL's own toolkit or from MTEB's implementation, and\ntreat the two as roughly but not exactly comparable given their differing evaluation splits and\nscaling conventions.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "miracl",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge m3",
   "model_id": "baai/bge-m3",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 67.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "colnomic embed multimodal 7B",
   "model_id": "nomic/colnomic-embed-multimodal-7b",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 65.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "nomic embed multimodal 7B",
   "model_id": "nomic/nomic-embed-multimodal-7b",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 65.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "nomic embed text v2 moe",
   "model_id": "nomic/nomic-embed-text-v2-moe",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 65.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "nomic embed text v2 moe GGUF",
   "model_id": "nomic/nomic-embed-text-v2-moe-gguf",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 65.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed l v2.0",
   "model_id": "snowflake/snowflake-arctic-embed-l-v2-0",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 64.9,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina embeddings v4",
   "model_id": "jina/jina-embeddings-v4",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 63.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina embeddings v4 vllm retrieval",
   "model_id": "jina/jina-embeddings-v4-vllm-retrieval",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 63.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina embeddings v5 text nano",
   "model_id": "jina/jina-embeddings-v5-text-nano",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 63.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina embeddings v5 text small",
   "model_id": "jina/jina-embeddings-v5-text-small",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 63.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina embeddings v5 text small retrieval",
   "model_id": "jina/jina-embeddings-v5-text-small-retrieval",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 63.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina clip v2",
   "model_id": "jina/jina-clip-v2",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 61.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina code embeddings 1.5B",
   "model_id": "jina/jina-code-embeddings-1-5b",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 61.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina embeddings v3",
   "model_id": "jina/jina-embeddings-v3",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 61.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 Embedding 8B",
   "model_id": "qwen/qwen3-embedding-8b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 60.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 VL Embedding 8B",
   "model_id": "qwen/qwen3-vl-embedding-8b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 60.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed m v2.0",
   "model_id": "snowflake/snowflake-arctic-embed-m-v2-0",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 59.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge multilingual gemma2",
   "model_id": "baai/bge-multilingual-gemma2",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 59.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Voyage 3",
   "model_id": "voyage/voyage-3",
   "provider": "voyage",
   "provider_display": "Voyage AI",
   "score": 58.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multilingual e5 large instruct",
   "model_id": "intfloat/multilingual-e5-large-instruct",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 58.2,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 Embedding 4B",
   "model_id": "qwen/qwen3-embedding-4b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 57.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Voyage Multilingual 2",
   "model_id": "voyage/voyage-multilingual-2",
   "provider": "voyage",
   "provider_display": "Voyage AI",
   "score": 57.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 mistral 7B instruct",
   "model_id": "intfloat/e5-mistral-7b-instruct",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 56.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini Embedding 001",
   "model_id": "google/gemini-embedding-001",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 56.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "NV Embed v2",
   "model_id": "nvidia/nv-embed-v2",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 55.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multilingual e5 large",
   "model_id": "intfloat/multilingual-e5-large",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 55.1,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "text-embedding-3-large",
   "model_id": "openai/text-embedding-3-large",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 54.9,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite embedding 278M multilingual",
   "model_id": "ibm/granite-embedding-278m-multilingual",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 54.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multilingual e5 base",
   "model_id": "intfloat/multilingual-e5-base",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 52.5,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed l",
   "model_id": "snowflake/snowflake-arctic-embed-l",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 52.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 Embedding 0.6B",
   "model_id": "qwen/qwen3-embedding-0-6b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 51.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 VL Embedding 2B",
   "model_id": "qwen/qwen3-vl-embedding-2b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 51.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed m v1.5",
   "model_id": "snowflake/snowflake-arctic-embed-m-v1-5",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 50.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite embedding 107M multilingual",
   "model_id": "ibm/granite-embedding-107m-multilingual",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 50.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "modernbert embed base",
   "model_id": "nomic/modernbert-embed-base",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 50.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multilingual e5 small",
   "model_id": "intfloat/multilingual-e5-small",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 50.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "nomic embed text v1.5",
   "model_id": "nomic/nomic-embed-text-v1-5",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 50.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "nomic embed text v1.5 GGUF",
   "model_id": "nomic/nomic-embed-text-v1-5-gguf",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 50.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "SFR Embedding 2 R",
   "model_id": "salesforce/sfr-embedding-2-r",
   "provider": "salesforce",
   "provider_display": "Salesforce",
   "score": 50.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed m",
   "model_id": "snowflake/snowflake-arctic-embed-m",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 48.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed m long",
   "model_id": "snowflake/snowflake-arctic-embed-m-long",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 48.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Voyage 3 Lite",
   "model_id": "voyage/voyage-3-lite",
   "provider": "voyage",
   "provider_display": "Voyage AI",
   "score": 48.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "nomic embed text v1",
   "model_id": "nomic/nomic-embed-text-v1",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 47.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite embedding english r2",
   "model_id": "ibm/granite-embedding-english-r2",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 46.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite embedding small english r2",
   "model_id": "ibm/granite-embedding-small-english-r2",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 46.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Voyage Finance 2",
   "model_id": "voyage/voyage-finance-2",
   "provider": "voyage",
   "provider_display": "Voyage AI",
   "score": 45.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "LaBSE",
   "model_id": "sentence-transformers/labse",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 45.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Embed",
   "model_id": "mistral/mistral-embed",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 45.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed s",
   "model_id": "snowflake/snowflake-arctic-embed-s",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 44.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Voyage Law 2",
   "model_id": "voyage/voyage-law-2",
   "provider": "voyage",
   "provider_display": "Voyage AI",
   "score": 44.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "paraphrase multilingual mpnet base v2",
   "model_id": "sentence-transformers/paraphrase-multilingual-mpnet-base-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 43.5,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge large en v1.5",
   "model_id": "baai/bge-large-en-v1-5",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 42.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "paraphrase multilingual MiniLM L12 v2",
   "model_id": "sentence-transformers/paraphrase-multilingual-minilm-l12-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 42.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "text-embedding-3-small",
   "model_id": "openai/text-embedding-3-small",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 42.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Voyage Code 3",
   "model_id": "voyage/voyage-code-3",
   "provider": "voyage",
   "provider_display": "Voyage AI",
   "score": 42.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge large en",
   "model_id": "baai/bge-large-en",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 41.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 large v2",
   "model_id": "intfloat/e5-large-v2",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 40.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed xs",
   "model_id": "snowflake/snowflake-arctic-embed-xs",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 40.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge base en v1.5",
   "model_id": "baai/bge-base-en-v1-5",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 39.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 large",
   "model_id": "intfloat/e5-large",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 38.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "distiluse base multilingual cased v2",
   "model_id": "sentence-transformers/distiluse-base-multilingual-cased-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 38.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite embedding 125M english",
   "model_id": "ibm/granite-embedding-125m-english",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 38.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge base en",
   "model_id": "baai/bge-base-en",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 37.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 base v2",
   "model_id": "intfloat/e5-base-v2",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 37.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 large unsupervised",
   "model_id": "intfloat/e5-large-unsupervised",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 37.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "distiluse base multilingual cased v1",
   "model_id": "sentence-transformers/distiluse-base-multilingual-cased-v1",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 36.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge small en v1.5",
   "model_id": "baai/bge-small-en-v1-5",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 35.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "text-embedding-ada-002",
   "model_id": "openai/text-embedding-ada-002",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 35.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 base",
   "model_id": "intfloat/e5-base",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 35.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "SFR Embedding Code 400M R",
   "model_id": "salesforce/sfr-embedding-code-400m-r",
   "provider": "salesforce",
   "provider_display": "Salesforce",
   "score": 35.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 small v2",
   "model_id": "intfloat/e5-small-v2",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 34.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge small en",
   "model_id": "baai/bge-small-en",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 33.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 small",
   "model_id": "intfloat/e5-small",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 32.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite embedding 30M english",
   "model_id": "ibm/granite-embedding-30m-english",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 32.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite embedding 30M sparse",
   "model_id": "ibm/granite-embedding-30m-sparse",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 32.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multi qa mpnet base dot v1",
   "model_id": "sentence-transformers/multi-qa-mpnet-base-dot-v1",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 32.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multi qa mpnet base cos v1",
   "model_id": "sentence-transformers/multi-qa-mpnet-base-cos-v1",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 31.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "paraphrase mpnet base v2",
   "model_id": "sentence-transformers/paraphrase-mpnet-base-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 30.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "all mpnet base v2",
   "model_id": "sentence-transformers/all-mpnet-base-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 30.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multi qa MiniLM L6 cos v1",
   "model_id": "sentence-transformers/multi-qa-minilm-l6-cos-v1",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 30.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "all MiniLM L12 v2",
   "model_id": "sentence-transformers/all-minilm-l12-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 29.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "msmarco bert base dot v5",
   "model_id": "sentence-transformers/msmarco-bert-base-dot-v5",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 29.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "all roberta large v1",
   "model_id": "sentence-transformers/all-roberta-large-v1",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 28.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "all MiniLM L6 v2",
   "model_id": "sentence-transformers/all-minilm-l6-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 28.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "msmarco MiniLM L12 cos v5",
   "model_id": "sentence-transformers/msmarco-minilm-l12-cos-v5",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 28.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "msmarco MiniLM L6 v3",
   "model_id": "sentence-transformers/msmarco-minilm-l6-v3",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 27.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "all distilroberta v1",
   "model_id": "sentence-transformers/all-distilroberta-v1",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 27.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "paraphrase MiniLM L12 v2",
   "model_id": "sentence-transformers/paraphrase-minilm-l12-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 27.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "paraphrase MiniLM L6 v2",
   "model_id": "sentence-transformers/paraphrase-minilm-l6-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 26.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "paraphrase MiniLM L3 v2",
   "model_id": "sentence-transformers/paraphrase-minilm-l3-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 24.0,
   "source": "mteb-leaderboard, miracl"
  }
 ],
 "page": {
  "aliases": [
   "Making a MIRACL",
   "Multilingual Information Retrieval Across a Continuum of Languages",
   "MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages"
  ],
  "category": "embedding",
  "contamination": {
   "note": "Every language's corpus is built from that language's public Wikipedia, and the train and development query-judgment splits have been public since the dataset's October 2022 release, so both sides of the retrieval task (documents and labelled relevant queries) are exposed to anything trained on web text after that date. The original WSDM 2023 Cup shared task likely held out a further test set during the live competition window, but this repository could not confirm whether any such hidden test material is still withheld today; current MIRACL-derived scores, including those on the MTEB leaderboard, are read from the public development split.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "ar",
    "bn",
    "de",
    "en",
    "es",
    "fa",
    "fi",
    "fr",
    "hi",
    "id",
    "ja",
    "ko",
    "ru",
    "sw",
    "te",
    "th",
    "yo",
    "zh"
   ],
   "license": "Apache-2.0 for the project-miracl/miracl toolkit and repository code; MTEB's task metadata lists the dataset content itself as CC BY-SA 4.0, consistent with the licence of the Wikipedia text the passage collections are built from.\n",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 18,
   "size_note": "18 languages: 16 were named at the initial release (Arabic, Bengali, English, Spanish, Persian, Finnish, French, Hindi, Indonesian, Japanese, Korean, Russian, Swahili, Telugu, Thai, Chinese), with German and Yoruba added as the two initially unannounced \"surprise\" languages for the WSDM 2023 Cup shared task (per MTEB's task definition, which cites the published paper). Per-language topic counts range from roughly 2,863-4,683 training queries and 213-1,271 development queries, with 1,606-41,358 relevance judgments per language, per the Hugging Face dataset card.\n",
   "splits": "Train and development splits are public for every language with judgments. A \"testA\" split exists only for the languages MIRACL shares with the earlier Mr. TyDi benchmark. MTEB's reference MIRACLRetrieval task evaluates on the development split.\n",
   "url": "https://huggingface.co/datasets/miracl/miracl"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice P",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "The reference implementation is the project-miracl/miracl GitHub repository, built on Pyserini for indexing and evaluation. In practice, most current MIRACL scores for embedding models are read off the MTEB leaderboard, which runs it as the \"MIRACLRetrieval\" task (with a \"MIRACLRetrievalHardNegatives\" variant built by pooling top results from BM25 and two embedding models) inside its multilingual retrieval collection, scored by nDCG@10 on the development split \u2014 see mteb/tasks/retrieval/multilingual/miracl_retrieval.py in the embeddings-benchmark/mteb repository.\n"
  },
  "id": "miracl",
  "last_updated": "",
  "leaderboard_url": "https://huggingface.co/spaces/mteb/leaderboard",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "mteb"
   ],
   "variants": []
  },
  "measures": "MIRACL tests whether a retrieval or embedding system can rank passages by relevance to a query within the same language, across 18 typologically diverse languages, most of them under-served by earlier retrieval benchmarks built mainly for English. For each language, the corpus is that language's Wikipedia, split into passages; native speakers of the language wrote the queries and judged which passages were relevant, rather than translating an English query set. It is monolingual retrieval (query and corpus share a language), not cross-lingual retrieval, and it is text-only, covering Arabic, Bengali, German, English, Spanish, Persian, Finnish, French, Hindi, Indonesian, Japanese, Korean, Russian, Swahili, Telugu, Thai, Yoruba and Chinese.\n",
  "metric": {
   "baseline_note": "The original paper's shared task also reported Recall@100. nDCG@10 is bounded 0-1 by construction in standard information-retrieval usage; scores in this repository's cards, and on the MTEB leaderboard where most current MIRACL numbers are read, are typically shown on a 0-100 scale instead. No random or human baseline was established from the sources read.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "nDCG@10",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "MIRACL",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2210.09984",
   "title": "MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages",
   "url": "https://arxiv.org/abs/2210.09984",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Xinyu Zhang",
    "Nandan Thakur",
    "Odunayo Ogundepo",
    "Ehsan Kamalloo",
    "David Alfonso-Hermelo",
    "Xiaoguang Li",
    "Qun Liu",
    "Mehdi Rezagholizadeh",
    "Jimmy Lin"
   ],
   "org": "David R. Cheriton School of Computer Science, University of Waterloo, with Huawei Noah's Ark Lab",
   "url": "https://github.com/project-miracl/miracl"
  },
  "released": "2022-10",
  "repo_url": "https://github.com/project-miracl/miracl",
  "saturation": {
   "as_of": "",
   "note": "MIRACL is 18 separate per-language retrieval tasks, not one scale, so a single ceiling is not meaningful; per-language difficulty varies with corpus size and query style. This repository found no source that assesses whether frontier embedding models have collectively saturated MIRACL across its 18 languages, so an overall status is not established here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages (arXiv preprint)",
    "url": "https://arxiv.org/abs/2210.09984"
   },
   {
    "accessed": "2026-09-08",
    "title": "MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages (Transactions of the Association for Computational Linguistics, 2023, vol. 11, pp. 1114-1131)",
    "url": "https://doi.org/10.1162/tacl_a_00595"
   },
   {
    "accessed": "2026-09-08",
    "title": "project-miracl/miracl (GitHub repository, README)",
    "url": "https://github.com/project-miracl/miracl"
   },
   {
    "accessed": "2026-09-08",
    "title": "miracl/miracl dataset card (Hugging Face)",
    "url": "https://huggingface.co/datasets/miracl/miracl"
   },
   {
    "accessed": "2026-09-08",
    "title": "MTEB MIRACLRetrieval task definition (embeddings-benchmark/mteb)",
    "url": "https://github.com/embeddings-benchmark/mteb/blob/main/mteb/tasks/retrieval/multilingual/miracl_retrieval.py"
   }
  ],
  "status": "active",
  "subcategory": "multilingual retrieval",
  "summary": "Monolingual ad hoc retrieval over Wikipedia passages in 18 languages, built from native-speaker queries and relevance judgments.",
  "tags": [
   "retrieval",
   "embedding",
   "multilingual",
   "information-retrieval"
  ],
  "task_format": "Given a query in one language, rank a passage corpus drawn from that language's Wikipedia by relevance to the query; relevance judgments were produced by native-speaker annotators per language, not machine-translated from a single source language.\n"
 }
}