{
 "body": "\n## What it measures\n\nBEIR asks a single retrieval or embedding system to search 18 different collections it was not built for, covering fact-checking, question answering, biomedical literature, news, arguments, duplicate questions, citation matching, tweets and Wikipedia entities. Given a query, the system ranks a corpus of passages or documents by relevance and is scored against human relevance judgments.\n\nThe point of the suite is generalisation rather than raw retrieval quality on any one domain: a retriever tuned on one style of query, most often MS MARCO's web-search queries, is tested cold on everything else, and its BEIR score is most meaningful read alongside what it was actually trained on.\n\n## How it is scored\n\nSystems are scored per dataset, primarily by nDCG@10, normalised discounted cumulative gain over the top 10 results, which is bounded between 0 and 1 by construction. The original paper also reports Recall@100, MAP and Precision@10. There is no single official average across the suite; different reporters compute their own mean over a subset of the 18 datasets, and that subset is not standardised, so an aggregate \"BEIR score\" from two different sources is not guaranteed to cover the same datasets. The paper's own zero-shot condition trains a system only on MS MARCO, if at all, then evaluates without further tuning on the rest.\n\n## Dataset and licence\n\nBEIR wraps 18 pre-existing public datasets, among them MS MARCO, TREC-COVID, NFCorpus, NQ, HotpotQA, FiQA-2018, ArguAna, Touche-2020, Quora, CQADupStack, DBPedia, SCIDOCS, FEVER, Climate-FEVER and SciFact, behind one loading and evaluation interface. The beir-cellar/beir toolkit is released under an Apache-2.0 licence; each underlying dataset keeps its own original licence, documented per dataset in the repository. All queries, corpora and relevance judgments are public, with corpus sizes ranging from a few thousand documents to several million.\n\n## Who publishes it\n\nBEIR was introduced by Nandan Thakur, Nils Reimers, Andreas R\u00fcckl\u00e9, Abhishek Srivastava and Iryna Gurevych at UKP Lab, TU Darmstadt, appearing at the NeurIPS 2021 Datasets and Benchmarks track. The code originally lived at UKPLab/beir on GitHub; maintenance has since moved to the community-run beir-cellar/beir repository, developed together with UKP Lab, Castorini and Hugging Face.\n\n## Lineage\n\nBEIR has no predecessor of its own; it repackages 18 earlier retrieval datasets under one protocol rather than introducing new data. Its most consequential successor is [MTEB](mteb.md) (Massive Text Embedding Benchmark), which explicitly reuses BEIR's datasets and evaluation method as MTEB's Retrieval task category. In this repository's own census of reported scores, most current BEIR numbers are in fact read off the Retrieval column of the MTEB leaderboard rather than a standalone BEIR-only leaderboard, which makes MTEB the practical successor even though BEIR's own toolkit still works standalone.\n\n## Saturation and contamination\n\nNo single ceiling applies across BEIR's 18 heterogeneous datasets, so an overall saturation status is not established here. Per-dataset leaderboards on MTEB show some datasets with much higher top scores than others, which is a known property of a suite built from datasets of very different difficulty, not a recent development. Contamination risk is best treated as medium: queries, corpora and relevance judgments are fully public, and several component datasets, including MS MARCO and Natural Questions, are also standard supervised training sets for retrieval models, so training on them breaks the zero-shot premise BEIR was designed to test. The original paper itself distinguishes in-domain (MS MARCO-trained) from zero-shot results for exactly this reason.\n\n## How to run it\n\nThe reference implementation is the beir-cellar/beir Python package, which handles dataset download, indexing and scoring against BM25 and dense baselines. Because BEIR is not one dataset but 18, published numbers are hard to compare unless the reporter states which subset they averaged, which retriever architecture and index they used (sparse, dense, late-interaction or reranking), and whether the underlying embedding model saw any BEIR component's training split during its own training. The MTEB leaderboard is the most current public source for BEIR-derived retrieval scores today.\n\n## Reading the numbers\n\nA strong average BEIR or MTEB-Retrieval score suggests a model generalises across retrieval domains rather than overfitting to one style of query. It does not tell you how the model does on any single domain you care about, since a high average can hide a low score on the one dataset closest to your use case. Always check whether the reporter trained on any BEIR component, especially MS MARCO or NQ, before treating a score as genuinely zero-shot, and check which of the 18 datasets were actually averaged into the number you are reading.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "beir",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "NV Embed v2",
   "model_id": "nvidia/nv-embed-v2",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 58.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 Embedding 8B",
   "model_id": "qwen/qwen3-embedding-8b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 57.5,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 VL Embedding 8B",
   "model_id": "qwen/qwen3-vl-embedding-8b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 57.5,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina embeddings v4",
   "model_id": "jina/jina-embeddings-v4",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 57.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina embeddings v4 vllm retrieval",
   "model_id": "jina/jina-embeddings-v4-vllm-retrieval",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 57.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Voyage 3",
   "model_id": "voyage/voyage-3",
   "provider": "voyage",
   "provider_display": "Voyage AI",
   "score": 56.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 mistral 7B instruct",
   "model_id": "intfloat/e5-mistral-7b-instruct",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 56.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed l v2.0",
   "model_id": "snowflake/snowflake-arctic-embed-l-v2-0",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 56.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina embeddings v3",
   "model_id": "jina/jina-embeddings-v3",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 55.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini Embedding 001",
   "model_id": "google/gemini-embedding-001",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 55.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "text-embedding-3-large",
   "model_id": "openai/text-embedding-3-large",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 55.1,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "nomic embed text v2 moe",
   "model_id": "nomic/nomic-embed-text-v2-moe",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 55.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "nomic embed text v2 moe GGUF",
   "model_id": "nomic/nomic-embed-text-v2-moe-gguf",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 55.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 Embedding 4B",
   "model_id": "qwen/qwen3-embedding-4b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 55.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "SFR Embedding 2 R",
   "model_id": "salesforce/sfr-embedding-2-r",
   "provider": "salesforce",
   "provider_display": "Salesforce",
   "score": 55.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge m3",
   "model_id": "baai/bge-m3",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 54.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed l",
   "model_id": "snowflake/snowflake-arctic-embed-l",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 53.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite embedding english r2",
   "model_id": "ibm/granite-embedding-english-r2",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 53.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite embedding small english r2",
   "model_id": "ibm/granite-embedding-small-english-r2",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 53.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multilingual e5 large instruct",
   "model_id": "intfloat/multilingual-e5-large-instruct",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 53.5,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed m v2.0",
   "model_id": "snowflake/snowflake-arctic-embed-m-v2-0",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 53.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Voyage Code 3",
   "model_id": "voyage/voyage-code-3",
   "provider": "voyage",
   "provider_display": "Voyage AI",
   "score": 52.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "nomic embed text v1.5",
   "model_id": "nomic/nomic-embed-text-v1-5",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 52.3,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "nomic embed text v1.5 GGUF",
   "model_id": "nomic/nomic-embed-text-v1-5-gguf",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 52.3,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge large en v1.5",
   "model_id": "baai/bge-large-en-v1-5",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 52.1,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Voyage Multilingual 2",
   "model_id": "voyage/voyage-multilingual-2",
   "provider": "voyage",
   "provider_display": "Voyage AI",
   "score": 52.0,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 large v2",
   "model_id": "intfloat/e5-large-v2",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 51.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed m v1.5",
   "model_id": "snowflake/snowflake-arctic-embed-m-v1-5",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 51.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Voyage 3 Lite",
   "model_id": "voyage/voyage-3-lite",
   "provider": "voyage",
   "provider_display": "Voyage AI",
   "score": 51.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge large en",
   "model_id": "baai/bge-large-en",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 51.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed m long",
   "model_id": "snowflake/snowflake-arctic-embed-m-long",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 51.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multilingual e5 large",
   "model_id": "intfloat/multilingual-e5-large",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 50.8,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "nomic embed text v1",
   "model_id": "nomic/nomic-embed-text-v1",
   "provider": "nomic",
   "provider_display": "Nomic AI",
   "score": 50.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 large",
   "model_id": "intfloat/e5-large",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 50.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed m",
   "model_id": "snowflake/snowflake-arctic-embed-m",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 50.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge base en v1.5",
   "model_id": "baai/bge-base-en-v1-5",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 50.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 large unsupervised",
   "model_id": "intfloat/e5-large-unsupervised",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 49.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina embeddings v2 base en",
   "model_id": "jina/jina-embeddings-v2-base-en",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 49.5,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 Embedding 0.6B",
   "model_id": "qwen/qwen3-embedding-0-6b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 49.5,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 VL Embedding 2B",
   "model_id": "qwen/qwen3-vl-embedding-2b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 49.5,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Embed",
   "model_id": "mistral/mistral-embed",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 49.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge base en",
   "model_id": "baai/bge-base-en",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 49.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 base v2",
   "model_id": "intfloat/e5-base-v2",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 48.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multilingual e5 base",
   "model_id": "intfloat/multilingual-e5-base",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 48.5,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite embedding 125M english",
   "model_id": "ibm/granite-embedding-125m-english",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 48.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge small en v1.5",
   "model_id": "baai/bge-small-en-v1-5",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 47.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 base",
   "model_id": "intfloat/e5-base",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 47.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed s",
   "model_id": "snowflake/snowflake-arctic-embed-s",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 47.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "jina embeddings v2 small en",
   "model_id": "jina/jina-embeddings-v2-small-en",
   "provider": "jina",
   "provider_display": "Jina AI",
   "score": 47,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "text-embedding-3-small",
   "model_id": "openai/text-embedding-3-small",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 46.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 small v2",
   "model_id": "intfloat/e5-small-v2",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 46.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multilingual e5 small",
   "model_id": "intfloat/multilingual-e5-small",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 46.2,
   "source": "mteb-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "bge small en",
   "model_id": "baai/bge-small-en",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 45.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "e5 small",
   "model_id": "intfloat/e5-small",
   "provider": "intfloat",
   "provider_display": "intfloat",
   "score": 45.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "snowflake arctic embed xs",
   "model_id": "snowflake/snowflake-arctic-embed-xs",
   "provider": "snowflake",
   "provider_display": "Snowflake",
   "score": 44.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "all mpnet base v2",
   "model_id": "sentence-transformers/all-mpnet-base-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 44.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "text-embedding-ada-002",
   "model_id": "openai/text-embedding-ada-002",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 43.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "all MiniLM L12 v2",
   "model_id": "sentence-transformers/all-minilm-l12-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 43.2,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite embedding 30M english",
   "model_id": "ibm/granite-embedding-30m-english",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 43.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite embedding 30M sparse",
   "model_id": "ibm/granite-embedding-30m-sparse",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 43.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multi qa mpnet base dot v1",
   "model_id": "sentence-transformers/multi-qa-mpnet-base-dot-v1",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 43.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "all MiniLM L6 v2",
   "model_id": "sentence-transformers/all-minilm-l6-v2",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 42.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multi qa mpnet base cos v1",
   "model_id": "sentence-transformers/multi-qa-mpnet-base-cos-v1",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 42.8,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "all roberta large v1",
   "model_id": "sentence-transformers/all-roberta-large-v1",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 42.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "all distilroberta v1",
   "model_id": "sentence-transformers/all-distilroberta-v1",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 41.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "multi qa MiniLM L6 cos v1",
   "model_id": "sentence-transformers/multi-qa-minilm-l6-cos-v1",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 41.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "msmarco bert base dot v5",
   "model_id": "sentence-transformers/msmarco-bert-base-dot-v5",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 40.5,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "msmarco MiniLM L12 cos v5",
   "model_id": "sentence-transformers/msmarco-minilm-l12-cos-v5",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 40.0,
   "source": "mteb-leaderboard, miracl"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "msmarco MiniLM L6 v3",
   "model_id": "sentence-transformers/msmarco-minilm-l6-v3",
   "provider": "sentence-transformers",
   "provider_display": "Sentence Transformers",
   "score": 39.0,
   "source": "mteb-leaderboard, miracl"
  }
 ],
 "page": {
  "aliases": [
   "BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models"
  ],
  "category": "embedding",
  "contamination": {
   "note": "Queries, corpora and relevance judgments are fully public. Several component datasets, including MS MARCO and Natural Questions, are also standard supervised training sets for retrieval models; training on them breaks the zero-shot premise BEIR was designed to test, a distinction the original paper itself draws between in-domain and zero-shot results.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 18,
   "size_note": "18 retrieval datasets across 9 tasks (MS MARCO, TREC-COVID, NFCorpus, BioASQ, NQ, HotpotQA, FiQA-2018, Signal-1M, TREC-NEWS, Robust04, ArguAna, Touche-2020, CQADupStack, Quora, DBPedia, SCIDOCS, FEVER, Climate-FEVER, SciFact); per-dataset corpora range from a few thousand to several million documents.",
   "splits": "",
   "url": "https://github.com/beir-cellar/beir"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "beir-cellar/beir: reference Python package for loading, indexing and scoring. Most current scores are read from the MTEB leaderboard's Retrieval task, which reuses BEIR's datasets and evaluation method."
  },
  "id": "beir",
  "last_updated": "",
  "leaderboard_url": "https://huggingface.co/spaces/mteb/leaderboard",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "mteb"
   ],
   "variants": []
  },
  "measures": "BEIR gathers 18 pre-existing retrieval datasets spanning nine task types, including fact-checking, question answering, biomedical IR, news retrieval, argument retrieval, duplicate-question detection, citation prediction, tweet retrieval and entity retrieval, and evaluates one retrieval system across all of them without per-dataset fine-tuning. Given a query, a system ranks a corpus of passages or documents by relevance, scored against human relevance judgments. The premise is that a good retriever should generalise zero-shot to a new domain rather than needing supervised training data from each one, so BEIR is most informative when compared against what the system was actually trained on.",
  "metric": {
   "baseline_note": "Recall@100, MAP and Precision@10 are also reported in the original paper. There is no single official average across the 18 datasets; reporters choose their own subset to average.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "nDCG@10",
   "random_baseline": null,
   "unit": ""
  },
  "name": "BEIR",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2104.08663",
   "title": "BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models",
   "url": "https://arxiv.org/abs/2104.08663",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Nandan Thakur",
    "Nils Reimers",
    "Andreas R\u00fcckl\u00e9",
    "Abhishek Srivastava",
    "Iryna Gurevych"
   ],
   "org": "UKP Lab, TU Darmstadt",
   "url": "https://github.com/beir-cellar/beir"
  },
  "released": "2021",
  "repo_url": "https://github.com/beir-cellar/beir",
  "saturation": {
   "as_of": "",
   "note": "BEIR is 18 heterogeneous datasets, not one scale, so a single ceiling is not meaningful. Per-dataset scores on the MTEB leaderboard vary widely by design, which is a known property of the suite rather than a recent shift.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models",
    "url": "https://arxiv.org/abs/2104.08663"
   },
   {
    "accessed": "2026-09-07",
    "title": "beir-cellar/beir (GitHub repository)",
    "url": "https://github.com/beir-cellar/beir"
   },
   {
    "accessed": "2026-09-07",
    "title": "MTEB Leaderboard",
    "url": "https://huggingface.co/spaces/mteb/leaderboard"
   }
  ],
  "status": "active",
  "subcategory": "zero-shot information retrieval",
  "summary": "A suite of 18 public retrieval datasets across 9 task types used to test whether a search or embedding model generalises to new domains without fine-tuning.",
  "tags": [
   "retrieval",
   "information-retrieval",
   "embeddings",
   "zero-shot"
  ],
  "task_format": "Zero-shot passage/document retrieval and ranking across 18 datasets and 9 task types; corpora range from thousands to millions of documents"
 }
}