A synthetic long-context suite of 13 tasks across four categories, built to show a model's effective context length is usually shorter than its claimed maximum.
unassessed
| Category | long-context |
|---|---|
| Subcategory | synthetic long-context retrieval, tracing and aggregation |
| Page status | active |
| Metric | accuracy, per task and averaged across all 13 |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | Apache-2.0 |
| Publisher | NVIDIA |
RULER procedurally generates long documents at a chosen sequence length and asks a model to complete one of 13 synthetic tasks over them, grouped into four categories: retrieval (needle-in-a-haystack variants with different needle/haystack types and multiple needles or queries at once), multi-hop tracing (following chains of variable-binding across the document), aggregation (extracting the most or least frequent words from a long list), and question answering (real SQuAD or HotpotQA questions embedded in a padded-out long context). It goes beyond a plain needle-in-a-haystack test specifically because that simpler test can be solved with a single read-through; RULER's multi-hop and aggregation tasks require combining information scattered across the whole context, which single-pass retrieval cannot do.
Long synthetic or padded-real-text document plus an instruction in; a short answer (a value, a word list, or an entity name) out, at a chosen target sequence length from 4K to 1M+ tokens.
No model card in ModelSpec reports this benchmark yet.