HELM's MS MARCO passage-ranking wrap: an LLM says Yes or No whether a passage answers a Bing query, then HELM ranks those decisions.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | HELM binary LLM ranker on MS MARCO passage tracks (regular and TREC) |
| Page status | unknown |
| Metric | RR@10 (regular track); NDCG@10 (TREC track) |
| Direction | higher_is_better |
| Dataset licence | non-commercial research only; no IP licence granted (MS MARCO terms) |
| Publisher | Microsoft (dataset); HELM wrap by Stanford CRFM |
This id is HELM's `msmarco` scenario, not the original 2016 reading- comprehension leaderboard and not an embedding nDCG run from [beir](beir.md). Microsoft MS MARCO pairs Bing queries with web passages. HELM follows Nogueira and Jiang (2020) and turns ranking into binary classification: given one passage and one query, the model must answer whether the passage answers the query. HELM then sorts passages by the Yes/No token and its log-probability. Two passage tracks exist: regular (dev.small qrels) and TREC (2019 deep-learning qrels). English text.
Binary ranking adapter. Prompt block is Passage / Query / "Does the passage answer the query?" / Answer: Yes or No. Default four in-context training instances (two blocks each: one gold, one non-gold). Stop at newline. Run spec name msmarco:track={regular|trec},valid_topk={int}. Regular gold relation is 1. TREC gold relations are 2 and 3 (0 and 1 count as non-gold).
No model card in ModelSpec reports this benchmark yet.