MTEB Pair Classification

Labels sentence pairs as duplicates or not from cosine similarity and scores the ranking by average precision, across 3 primarily-English MTEB datasets.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryembedding
Subcategorypair classification
Page statusactive
Metricaverage precision (cosine similarity)
Directionhigher_is_better
Unit%
Dataset size3
Dataset licenceVaries by dataset.
PublisherHugging Face and Cohere (original authors); maintained today by the open embeddings-benchmark community

What it measures

Pairs of sentences labelled as equivalent or not (paraphrases, duplicate questions) are embedded, and cosine similarity between each pair is used as a ranking score rather than a fixed threshold, evaluated by average precision.

Task format

Embed sentence pairs; rank by cosine similarity; score against binary equivalence labels with average precision.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
e5 mistral 7B instructintfloat87.02026-04
multilingual e5 large instructintfloat86.22026-04
SFR Embedding 2 RSalesforce86.02026-04
Voyage 3Voyage AI86.02026-04
Qwen3 Embedding 8BAlibaba / Qwen Team85.52026-04
Qwen3 VL Embedding 8BAlibaba / Qwen Team85.52026-04
snowflake arctic embed l v2.0Snowflake85.52026-04
text-embedding-3-largeOpenAI85.52026-04
Gemini Embedding 001Google DeepMind85.12026-04
bge large en v1.5BAAI85.02026-04
bge multilingual gemma2BAAI85.02026-04
e5 large v2intfloat85.02026-04
Qwen3 Embedding 4BAlibaba / Qwen Team85.02026-04
multilingual e5 largeintfloat84.82026-04
bge m3BAAI84.52026-04
e5 largeintfloat84.52026-04
e5 large unsupervisedintfloat84.52026-04
granite embedding english r2IBM84.52026-04
granite embedding small english r2IBM84.52026-04
jina embeddings v4Jina AI84.52026-04
jina embeddings v4 vllm retrievalJina AI84.52026-04
jina embeddings v5 text nanoJina AI84.52026-04
jina embeddings v5 text smallJina AI84.52026-04
jina embeddings v5 text small retrievalJina AI84.52026-04
snowflake arctic embed lSnowflake84.52026-04
NV Embed v2NVIDIA84.12026-04
bge large enBAAI84.02026-04
colnomic embed multimodal 7BNomic AI84.02026-04
jina clip v2Jina AI84.02026-04
jina code embeddings 1.5BJina AI84.02026-04
jina embeddings v3Jina AI84.02026-04
nomic embed multimodal 7BNomic AI84.02026-04
nomic embed text v2 moeNomic AI84.02026-04
nomic embed text v2 moe GGUFNomic AI84.02026-04
snowflake arctic embed m v2.0Snowflake84.02026-04
e5 base v2intfloat83.52026-04
Voyage Code 3Voyage AI83.52026-04
Voyage Multilingual 2Voyage AI83.52026-04
bge base en v1.5BAAI83.22026-04
e5 baseintfloat83.02026-04
granite embedding 278M multilingualIBM83.02026-04
modernbert embed baseNomic AI83.02026-04
multilingual e5 baseintfloat83.02026-04
nomic embed text v1.5Nomic AI83.02026-04
nomic embed text v1.5 GGUFNomic AI83.02026-04
snowflake arctic embed mSnowflake83.02026-04
snowflake arctic embed m v1.5Snowflake83.02026-04
Voyage 3 LiteVoyage AI83.02026-04
Voyage Finance 2Voyage AI83.02026-04
snowflake arctic embed m longSnowflake82.82026-04
Voyage Law 2Voyage AI82.82026-04
bge base enBAAI82.52026-04
nomic embed text v1Nomic AI82.52026-04
SFR Embedding Code 400M RSalesforce82.52026-04
bge code v1BAAI82.02026-04
e5 small v2intfloat82.02026-04
Mistral EmbedMistral AI82.02026-04
Qwen3 Embedding 0.6BAlibaba / Qwen Team82.02026-04
Qwen3 VL Embedding 2BAlibaba / Qwen Team82.02026-04
text-embedding-3-smallOpenAI82.02026-04
all mpnet base v2Sentence Transformers81.52026-04
granite embedding 125M englishIBM81.52026-04
multilingual e5 smallintfloat81.52026-04
snowflake arctic embed sSnowflake81.52026-04
all MiniLM L12 v2Sentence Transformers81.02026-04
bge small en v1.5BAAI81.02026-04
CodeRankEmbedNomic AI81.02026-04
e5 smallintfloat81.02026-04
nomic embed codeNomic AI81.02026-04
all MiniLM L6 v2Sentence Transformers80.52026-04
bge small enBAAI80.52026-04
granite embedding 107M multilingualIBM80.52026-04
text-embedding-ada-002OpenAI80.22026-04
multi qa mpnet base dot v1Sentence Transformers80.02026-04
paraphrase mpnet base v2Sentence Transformers80.02026-04
multi qa mpnet base cos v1Sentence Transformers79.82026-04
all roberta large v1Sentence Transformers79.52026-04
snowflake arctic embed xsSnowflake79.52026-04
all distilroberta v1Sentence Transformers79.02026-04
paraphrase multilingual mpnet base v2Sentence Transformers79.02026-04
granite embedding 30M englishIBM78.52026-04
granite embedding 30M sparseIBM78.52026-04
msmarco bert base dot v5Sentence Transformers78.52026-04
multi qa MiniLM L6 cos v1Sentence Transformers78.52026-04
paraphrase MiniLM L12 v2Sentence Transformers78.02026-04
paraphrase multilingual MiniLM L12 v2Sentence Transformers78.02026-04
msmarco MiniLM L12 cos v5Sentence Transformers77.52026-04
paraphrase MiniLM L6 v2Sentence Transformers77.52026-04
msmarco MiniLM L6 v3Sentence Transformers77.02026-04
LaBSESentence Transformers76.02026-04
paraphrase MiniLM L3 v2Sentence Transformers75.52026-04
distiluse base multilingual cased v2Sentence Transformers75.02026-04
distiluse base multilingual cased v1Sentence Transformers73.52026-04

Data

This page as JSON · Edit on GitHub