OpenAI's MRCR: reproduce the i-th matching assistant writing in a long synthetic chat, after a hash prefix.
unassessed
| Category | long-context |
|---|---|
| Subcategory | multi-needle writing retrieval in a synthetic dialogue |
| Page status | active |
| Metric | openai_mrcr_accuracy |
| Direction | higher_is_better |
| Dataset size | 2400 |
| Dataset licence | MIT |
| Publisher | OpenAI (dataset); Stanford CRFM (HELM wrap) |
openai_mrcr is OpenAI's public Multi-round co-reference resolution set, wrapped by Stanford HELM. The model reads a long English user–assistant chat in which the user repeatedly asks for a piece of writing (poem, blog post, and similar). Two, four, or eight of those asks share a topic/format key. The last user turn asks for the i-th matching assistant reply and requires a short alphanumeric hash at the front of the answer. Distractors are other GPT-4o writings, not random tokens. Inspired by Gemini MRCR in Michelangelo; it is not [deepmind_mrcr_v2](deepmind_mrcr_v2.md).
Chat generation. HELM AdapterSpec method ADAPT_CHAT, temperature 0, max_tokens 2000, one output. Scenario parameters: needles in {2,4,8} and optional max_num_words (default 131,072). Loads openai/mrcr train split, data_files {needles}needle.parquet, revision 204b0d4e8d9ca5c0a90bf942fdb2a5969094adc0. Extra data carries random_string_to_prepend.
No model card in ModelSpec reports this benchmark yet.