Rolling perplexity on monthly snapshots of new Wikipedia, GitHub, BBC, arXiv, bioRxiv and AO3 text, meant to limit train-set leakage.
unassessed
| Category | generation |
|---|---|
| Subcategory | rolling log-likelihood / bits-per-byte on newly published web documents |
| Page status | active |
| Metric | bits_per_byte (also word_perplexity and byte_perplexity) |
| Direction | lower_is_better |
| Unit | bpb |
| Dataset size | 7500 |
| Dataset licence | MIT on the evaluation code (Jellyfish042/uncheatable_eval). The 2026-07 Hub card states no dataset licence. |
| Publisher | Jellyfish042 |
Uncheatable Eval does not ask questions. It scores how well a causal language model assigns probability to documents published after typical training cutoffs: new Wikipedia pages, GitHub files, BBC news, arXiv and bioRxiv papers, and AO3 fiction. The publisher argues that a public static QA set can leak into training, while last-week's crawl cannot already be in a released checkpoint. lm-eval implements the upstream rolling log-likelihood over a pinned Hub snapshot. The README says the method is for base models, not chat-tuned ones.
output_type loglikelihood_rolling. doc_to_text is empty; doc_to_target is the document content field. Fifteen category tasks filter the snapshot on the category column. Group uncheatable_eval averages word_perplexity, byte_perplexity and bits_per_byte weighted by size. Lower is better for all three.
No model card in ModelSpec reports this benchmark yet.