{
 "body": "\n## What it measures\n\nMTEB measures how useful a text embedding model's vectors are for downstream work, rather\nthan how well a model writes text. A model embeds the sentences, passages or documents in\neach task's dataset, and a fixed, non-learned scoring routine is applied to those vectors:\nnearest-neighbour classification, clustering, cosine-similarity ranking for retrieval and\nreranking, or correlation with human similarity judgements. The task types this repository\ncovers are retrieval, classification, clustering, reranking, semantic textual similarity\n(STS), summarization and pair classification. Bitext mining, and the newer image (MIEB),\naudio (MAEB) and video (MVEB) siblings, are outside this family's scope. Coverage ranges\nfrom English-only leaderboards to multilingual ones spanning over 100 languages.\n\n## How it is scored\n\nEach task type keeps its own metric: classification reports accuracy, clustering reports\nV-measure, retrieval reports nDCG@10, reranking reports MAP, STS and summarization report\nSpearman correlation between cosine similarity and gold judgements, and pair classification\nreports average precision. The public leaderboard then blends tasks two ways: a Task Mean\nthat weights every individual task equally, and a Task Type Mean that first averages within\neach task type and then across task types, so a type with many datasets cannot dominate the\nheadline number. Evaluation is zero-shot in the sense that the embedding model is not\nfine-tuned on the task, though some models are given task-specific instruction prefixes at\ninference time, which is itself a protocol difference worth checking between reporters.\n\n## Dataset and licence\n\nThe original 2022 paper drew on 58 datasets across 112 languages to evaluate 33 models. The\n2025 MMTEB expansion and the live leaderboard go well beyond that: the English v2 board runs\n41 tasks over 186 models, and the multilingual v2 board runs 131 tasks over 175 models, built\nfrom datasets contributed by dozens of research groups. There is no single MTEB licence \u2014\neach underlying dataset (Banking77, MS MARCO subsets, BIOSSES, and so on) keeps whatever\nlicence its own publisher chose \u2014 but the MTEB benchmark code and harness are Apache-2.0.\nTest labels for the classic tasks in scope here are public.\n\n## Who publishes it\n\nMTEB was introduced by Niklas Muennighoff, Nouamane Tazi, Lo\u00efc Magne and Nils Reimers, then\nat Hugging Face and Cohere, in \"MTEB: Massive Text Embedding Benchmark\" (EACL 2023; preprint\nOctober 2022). Maintenance passed to the open embeddings-benchmark GitHub organisation, and\nthe 2025 MMTEB paper credits a much larger contributor base led by Kenneth Enevoldsen and\nIsaac Chung among many others. The Hugging Face MTEB Leaderboard Space is the de facto public\nscoreboard today.\n\n## Lineage\n\nMTEB followed earlier single-purpose embedding evaluations such as SentEval; its contribution\nwas to put retrieval, classification, clustering, reranking, STS, summarization, pair\nclassification and bitext mining behind one harness and one leaderboard. In this repository,\n`mteb_overall` is the leaderboard's blended average, and `mteb_retrieval`,\n`mteb_classification`, `mteb_clustering`, `mteb_reranking`, `mteb_sts`, `mteb_summarization`\nand `mteb_pair_classification` are its task-type subsets; bitext mining has no page here yet.\nSince 2022 the project has grown sibling benchmarks \u2014 MMTEB (massively multilingual), MIEB\n(image), MAEB (audio) and MVEB (video) \u2014 and a beta private-data retrieval track called RTEB,\nnone of which are in scope for this page.\n\n## Saturation and contamination\n\nNo individual task sits at the ceiling: task-type leaderboards in 2026 still show real\nseparation between models rather than a pile-up near the maximum score. The item to watch is\ntest-set exposure rather than saturation: most classic MTEB datasets are public, some have\nbeen online since 2022, and their text can plausibly enter later models' pretraining data.\nThe project's own response has been versioning (v2 task revisions) and introducing RTEB, a\ntrack that keeps some retrieval test data private specifically to blunt this.\n\n## How to run it\n\nThe reference implementation is the `mteb` Python package (`pip install mteb` or `uv add\nmteb`), which exposes a CLI \u2014 for example `mteb run -m sentence-transformers/all-MiniLM-L6-v2\n-t Banking77Classification.v2` \u2014 and is what the public leaderboard itself runs. MTEB is not\none of the tasks bundled with lm-evaluation-harness, HELM, OpenCompass or BIG-bench. Numbers\nare hard to compare across reporters when they differ on embedding dimensionality or\ntruncation (for Matryoshka-style models), maximum sequence length, pooling method, and whether\na task-specific instruction prefix was prepended to the input.\n\n## Reading the numbers\n\nA high `mteb_overall` score says a model produces broadly useful general-purpose embeddings;\nit does not say the model is good at the one thing you need, because the average blends tasks\nas different as clustering and retrieval. Check the task-type subset that matches your use\ncase \u2014 retrieval for search and RAG, STS or pair classification for deduplication and\nmatching, classification or clustering for tagging \u2014 rather than the headline number. Note the\nmodel's embedding dimension and maximum sequence length alongside its score: both drive real\ndeployment cost and neither is captured by the benchmark.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mteb",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Massive Text Embedding Benchmark",
   "MMTEB"
  ],
  "category": "embedding",
  "contamination": {
   "note": "Most classic MTEB datasets are public with public test labels and have been online since 2022 or earlier, so their text can plausibly enter later models' pretraining data. The maintainers cite this as part of the motivation for versioned tasks and the private RTEB track.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [],
   "license": "Varies by dataset (each task keeps its source dataset's own licence); the MTEB benchmark code itself is Apache-2.0.",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "58 datasets across 8 task types and 112 languages in the original 2022 release, evaluating 33 models. The MMTEB-expanded leaderboard runs far more: 41 tasks over 186 models on MTEB(eng, v2) and 131 tasks over 175 models on MTEB(Multilingual, v2) as of access.",
   "splits": "",
   "url": "https://github.com/embeddings-benchmark/mteb"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "mteb Python package (pip install mteb; CLI: mteb run -m <model> -t <task>) is the reference implementation and what the public leaderboard runs."
  },
  "id": "mteb",
  "last_updated": "",
  "leaderboard_url": "https://huggingface.co/spaces/mteb/leaderboard",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "mteb_overall",
    "mteb_retrieval",
    "mteb_classification",
    "mteb_clustering",
    "mteb_reranking",
    "mteb_sts",
    "mteb_summarization",
    "mteb_pair_classification"
   ]
  },
  "measures": "MTEB measures how useful a text embedding model's output vectors are for downstream work, not how well the model writes text. Each task type applies a fixed, non-learned scoring routine to the embeddings a model produces: nearest-neighbour classification, clustering, cosine-similarity ranking for retrieval and reranking, or correlation with human similarity judgements. The task types in scope for this repository are retrieval, classification, clustering, reranking, semantic textual similarity, summarization and pair classification; bitext mining and the newer image, audio and video MTEB variants sit outside this family page's scope. Coverage ranges from English-only task sets to multilingual ones spanning over 100 languages.\n",
  "metric": {
   "baseline_note": "No single metric: each task type keeps its own scoring routine and its own baseline. See the subset pages.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "",
   "random_baseline": null,
   "unit": ""
  },
  "name": "MTEB (Massive Text Embedding Benchmark)",
  "page_kind": "family",
  "paper": {
   "arxiv": "2210.07316",
   "title": "MTEB: Massive Text Embedding Benchmark",
   "url": "https://aclanthology.org/2023.eacl-main.148/",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Niklas Muennighoff",
    "Nouamane Tazi",
    "Lo\u00efc Magne",
    "Nils Reimers"
   ],
   "org": "Hugging Face and Cohere (original authors); maintained today by the open embeddings-benchmark community",
   "url": "https://github.com/embeddings-benchmark"
  },
  "released": "2022-10",
  "repo_url": "https://github.com/embeddings-benchmark/mteb",
  "saturation": {
   "as_of": "",
   "note": "No individual task sits at the ceiling, but the public response to test-set exposure (versioned v2 task revisions, and a private-data RTEB track) is itself a saturation-adjacent signal worth watching.",
   "status": "watch",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "MTEB: Massive Text Embedding Benchmark (arXiv preprint)",
    "url": "https://arxiv.org/abs/2210.07316"
   },
   {
    "accessed": "2026-09-07",
    "title": "MTEB: Massive Text Embedding Benchmark (ACL Anthology, EACL 2023)",
    "url": "https://aclanthology.org/2023.eacl-main.148/"
   },
   {
    "accessed": "2026-09-07",
    "title": "embeddings-benchmark/mteb GitHub repository",
    "url": "https://github.com/embeddings-benchmark/mteb"
   },
   {
    "accessed": "2026-09-07",
    "title": "MTEB Leaderboard (Hugging Face Space)",
    "url": "https://huggingface.co/spaces/mteb/leaderboard"
   }
  ],
  "status": "active",
  "subcategory": "text embedding",
  "summary": "A multi-task suite that scores text embedding models on retrieval, classification, clustering, reranking, similarity, summarization and pair classification.",
  "tags": [
   "embedding",
   "retrieval",
   "semantic-search",
   "multilingual"
  ],
  "task_format": "Varies by task type: query-document ranking for retrieval and reranking, single-text classification into a fixed label set, unsupervised clustering of a text collection, sentence-pair scoring for similarity or entailment, and summary scoring by embedding distance to reference summaries. Every task consumes only the model's embedding vectors.\n"
 }
}