A rolling benchmark turning freshly-published arXiv text into sentence-ordering, cloze and next-fragment multiple-choice tasks every six months, built to measure how much contamination inflates benchmark scores.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | rolling, contamination-auditing sentence-ordering, cloze and continuation tasks built from recent arXiv text |
| Page status | active |
| Metric | accuracy (per SCP-type, per domain, per release); the paper's own headline metric is Rugged Score (RS), a derived overestimation measure, not this page's per-item accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | Not established. No LICENSE file was detected by GitHub's own licence API for the liangzid/ArxivRoll source repository, and the sampled Hugging Face dataset configs carry no licence tag. The underlying text is drawn from recent arXiv papers, each subject to arXiv's own non-exclusive distribution terms or the author's chosen licence; the repository's own README explicitly instructs contributors not to redistribute paper source files beyond what each paper's licence permits. |
| Publisher | The Hong Kong Polytechnic University |
ArxivRollBench is the evaluation-task component of ArxivRoll, a framework whose actual goal is not to test topical arXiv knowledge but to measure how much a model's public-benchmark score is inflated by contamination or uneven training exposure. It does this with SCP tasks -- Sequencing (reorder three shuffled sentences, choosing among four candidate orderings), Cloze (fill three masked sentences in a paragraph from labelled candidates, again choosing among four candidate combinations), and Prediction (choose the correct next fragment of a passage from four options) -- built automatically from arXiv article text collected after each task set is constructed, across eight domains (cs, econ, eess, math, physics, q-bio, q-fin, stat). Because the source text postdates every model's training cutoff at the time a round is built, and each round is evaluated once and then either kept private or later published as "expired," the authors use performance on ArxivRollBench relative to performance on long-public benchmarks to compute a "Rugged Score," a measure of public-benchmark overestimation -- inspired explicitly by one-time-pad encryption, where reusing the same key (here, the same test items) twice defeats the scheme's guarantees.
Four-way multiple choice per item. Sequencing and Cloze items ask the model to answer "Selection 1" through "Selection 4"; Prediction items ask for a letter A-D. All are zero-shot, generation-based (the model's free text is parsed by regex for a selection number or letter), scored per domain, per release and per SCP type.
No model card in ModelSpec reports this benchmark yet.