HELM's Vietnamese information-retrieval track: rank passages for a query on translated mMARCO (RR@10) and mRobust (NDCG@10).
unassessed
| Category | knowledge |
|---|---|
| Subcategory | Vietnamese passage ranking with a binary LLM ranker |
| Page status | unknown |
| Metric | RR@10 on Vietnamese mMARCO; NDCG@10 on Vietnamese mRobust (no combined score) |
| Direction | higher_is_better |
| Dataset licence | mMARCO README states Apache-2.0; the dataset repo also ships a Creative Commons Attribution 4.0 licence text. Both were opened. mRobust's card citation block states Creative Commons Attribution 4.0 International for the mRobust04 paper; a separate licence file was not opened for mRobust. |
| Publisher | Stanford CRFM (HELM MELT scenarios); mMARCO/mRobust from unicamp-dl |
MELT IR is not one item set. HELM file `melt_ir_scenario.py` implements two Vietnamese ranking scenarios that share a binary ranker: mMARCO (translated MS MARCO passage ranking) and mRobust (translated TREC Robust 2004). For each query the model is asked, in Vietnamese, whether a passage answers the query. HELM then turns those yes/no decisions into ranking metrics. There is no official average of the two tracks. HELM's schema labels the parent group "MELT Scenarios"; the same file's one-line description says "medical domain", which does not match these retrieval tasks.
Binary ranking: query plus candidate passage, Vietnamese prompt "Đoạn văn này có trả lời được câu hỏi không?", stop at newline. Default run entries set valid_topk=30. Training uses 1,000 query pairs with one positive and one negative passage each.
No model card in ModelSpec reports this benchmark yet.