MTEB Summarization

Scores whether embedding similarity to human-written summaries predicts human quality ratings of machine summaries, on a single English dataset.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryembedding
Subcategorysummarization
Page statusactive
MetricSpearman correlation (cosine similarity)
Directionhigher_is_better
Unit%
Dataset size1
Dataset licenceVaries by dataset.
PublisherHugging Face and Cohere (original authors); maintained today by the open embeddings-benchmark community

What it measures

Machine-generated and human-written summaries of the same source articles are embedded, and each machine summary is scored by its embedding's similarity to the human references. That similarity score is then checked against human quality ratings.

Task format

Embed machine and human summaries; score each machine summary by embedding similarity to human references; correlate (Spearman) against human quality ratings.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
e5 mistral 7B instructintfloat31.52026-04
Gemini Embedding 001Google DeepMind30.52026-04
Qwen3 Embedding 8BAlibaba / Qwen Team30.52026-04
Qwen3 VL Embedding 8BAlibaba / Qwen Team30.52026-04
SFR Embedding 2 RSalesforce30.52026-04
Voyage 3Voyage AI30.52026-04
multilingual e5 large instructintfloat30.42026-04
NV Embed v2NVIDIA30.42026-04
jina embeddings v4Jina AI30.22026-04
jina embeddings v4 vllm retrievalJina AI30.22026-04
jina embeddings v5 text nanoJina AI30.22026-04
jina embeddings v5 text smallJina AI30.22026-04
jina embeddings v5 text small retrievalJina AI30.22026-04
snowflake arctic embed l v2.0Snowflake30.22026-04
text-embedding-3-largeOpenAI30.12026-04
bge large en v1.5BAAI30.02026-04
bge multilingual gemma2BAAI30.02026-04
Qwen3 Embedding 4BAlibaba / Qwen Team30.02026-04
bge m3BAAI29.82026-04
jina clip v2Jina AI29.72026-04
jina code embeddings 1.5BJina AI29.72026-04
jina embeddings v3Jina AI29.72026-04
colnomic embed multimodal 7BNomic AI29.52026-04
e5 large v2intfloat29.52026-04
granite embedding english r2IBM29.52026-04
granite embedding small english r2IBM29.52026-04
nomic embed multimodal 7BNomic AI29.52026-04
nomic embed text v2 moeNomic AI29.52026-04
nomic embed text v2 moe GGUFNomic AI29.52026-04
snowflake arctic embed lSnowflake29.52026-04
bge large enBAAI29.22026-04
multilingual e5 largeintfloat29.22026-04
e5 largeintfloat29.02026-04
e5 large unsupervisedintfloat29.02026-04
snowflake arctic embed m v2.0Snowflake29.02026-04
Voyage Code 3Voyage AI29.02026-04
Voyage Finance 2Voyage AI29.02026-04
Voyage Multilingual 2Voyage AI29.02026-04
bge base en v1.5BAAI28.82026-04
modernbert embed baseNomic AI28.82026-04
nomic embed text v1.5Nomic AI28.82026-04
nomic embed text v1.5 GGUFNomic AI28.82026-04
Voyage Law 2Voyage AI28.82026-04
e5 base v2intfloat28.52026-04
granite embedding 278M multilingualIBM28.52026-04
snowflake arctic embed mSnowflake28.52026-04
snowflake arctic embed m v1.5Snowflake28.52026-04
text-embedding-3-smallOpenAI28.52026-04
Voyage 3 LiteVoyage AI28.52026-04
nomic embed text v1Nomic AI28.22026-04
snowflake arctic embed m longSnowflake28.22026-04
bge base enBAAI28.02026-04
e5 baseintfloat28.02026-04
Mistral EmbedMistral AI28.02026-04
multilingual e5 baseintfloat28.02026-04
Qwen3 Embedding 0.6BAlibaba / Qwen Team28.02026-04
Qwen3 VL Embedding 2BAlibaba / Qwen Team28.02026-04
SFR Embedding Code 400M RSalesforce28.02026-04
bge code v1BAAI27.52026-04
CodeRankEmbedNomic AI27.52026-04
e5 small v2intfloat27.52026-04
granite embedding 125M englishIBM27.52026-04
nomic embed codeNomic AI27.52026-04
bge small en v1.5BAAI27.22026-04
all mpnet base v2Sentence Transformers27.02026-04
e5 smallintfloat27.02026-04
granite embedding 107M multilingualIBM27.02026-04
multilingual e5 smallintfloat27.02026-04
snowflake arctic embed sSnowflake27.02026-04
text-embedding-ada-002OpenAI27.02026-04
all MiniLM L12 v2Sentence Transformers26.82026-04
all MiniLM L6 v2Sentence Transformers26.52026-04
bge small enBAAI26.52026-04
multi qa mpnet base dot v1Sentence Transformers26.52026-04
multi qa mpnet base cos v1Sentence Transformers26.22026-04
paraphrase mpnet base v2Sentence Transformers26.22026-04
all roberta large v1Sentence Transformers26.02026-04
all distilroberta v1Sentence Transformers25.82026-04
paraphrase multilingual mpnet base v2Sentence Transformers25.82026-04
granite embedding 30M englishIBM25.52026-04
granite embedding 30M sparseIBM25.52026-04
msmarco bert base dot v5Sentence Transformers25.52026-04
multi qa MiniLM L6 cos v1Sentence Transformers25.52026-04
snowflake arctic embed xsSnowflake25.52026-04
msmarco MiniLM L12 cos v5Sentence Transformers25.02026-04
paraphrase MiniLM L12 v2Sentence Transformers25.02026-04
paraphrase multilingual MiniLM L12 v2Sentence Transformers25.02026-04
msmarco MiniLM L6 v3Sentence Transformers24.52026-04
paraphrase MiniLM L6 v2Sentence Transformers24.52026-04
LaBSESentence Transformers24.02026-04
distiluse base multilingual cased v2Sentence Transformers23.52026-04
paraphrase MiniLM L3 v2Sentence Transformers23.02026-04
distiluse base multilingual cased v1Sentence Transformers22.52026-04

Data

This page as JSON · Edit on GitHub