BEIR

A suite of 18 public retrieval datasets across 9 task types used to test whether a search or embedding model generalises to new domains without fine-tuning.

Also known as: BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models

unassessed

What it measures

BEIR gathers 18 pre-existing retrieval datasets spanning nine task types, including fact-checking, question answering, biomedical IR, news retrieval, argument retrieval, duplicate-question detection, citation prediction, tweet retrieval and entity retrieval, and evaluates one retrieval system across all of them without per-dataset fine-tuning. Given a query, a system ranks a corpus of passages or documents by relevance, scored against human relevance judgments. The premise is that a good retriever should generalise zero-shot to a new domain rather than needing supervised training data from each one, so BEIR is most informative when compared against what the system was actually trained on.

Task format

Zero-shot passage/document retrieval and ranking across 18 datasets and 9 task types; corpora range from thousands to millions of documents

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub