DeepMind MRCR v2

Google DeepMind's MRCR v2: reproduce the i-th matching assistant turn in a long dialogue, after emitting a 12-character hash.

Also known as: MRCR v2, multi-round coreference resolution, DeepMindMRCRV2Scenario, mrcr_v2p1

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	long-context
Subcategory	count-and-reproduce the i-th assistant writing in a long synthetic dialogue
Page status	active
Metric	deepmind_mrcr_v2_score
Direction	higher_is_better
Dataset licence	Apache-2.0
Publisher	Google DeepMind (eval_hub); HELM wrap by Stanford CRFM

What it measures

deepmind_mrcr_v2 is HELM's wrap of Google DeepMind eval_hub MRCR v2 (multi-round coreference resolution). The model reads a long English user–assistant dialogue in which the user asks for writing that matches a format/style/topic triple and the assistant replies. Several asks share a key; replies to the same ask are distinct. At the end the model must emit a unique 12-character string from the target, then reproduce the i-th matching assistant reply. Needle counts in the public files are 2, 4, and 8. Context bins run from 4K–8K tokens through 4M–8M, plus an upto_128K cumulative file.

Task format

Open-ended generation over a single concatenated prompt (HELM: temperature 0, max_tokens 1000, no chat wrapper). DeepMind's example script uses the Gemini API at temperature 1.0 with no max_tokens cap. HELM parameters: needles and tokens.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub