Allen AI fit benchmark over 546 English and code domains from 16 sources, scored as perplexity and bits per byte rather than task accuracy.
unassessed
| Category | generation |
|---|---|
| Subcategory | domain-stratified language-modelling perplexity and bits per byte |
| Page status | active |
| Metric | bits_per_byte (also word_perplexity, byte_perplexity) |
| Direction | lower_is_better |
| Unit | bpb |
| Dataset licence | AI2 ImpACT License – Low Risk Artifacts (WikiText-103 CC BY-SA; M2D2 CC BY-NC; TwitterAAE research-only; RedPajama per source) |
| Publisher | Allen Institute for AI |
Paloma (Perplexity Analysis for Language Model Assessment) measures how well a language model's next-token probabilities fit many held-out domains, instead of one mixed validation dump. Magnusson et al. sample 16 sources (C4, mC4-en, WikiText-103, Penn Treebank, RedPajama, Falcon RefinedWeb, Dolma v1.5, M2D2 Wikipedia and S2ORC, C4-100 URL domains, Dolma top-100 subreddits and programming languages, TwitterAAE, Manosphere, Gab, 4chan). Those sources further split into 546 English and code domains. There are no questions. A score says how surprising the domain's text is under the model, not whether it answers items.
Rolling loglikelihood. lm-eval sets output_type loglikelihood_rolling, empty doc_to_text, and doc_to_target as the document text. Splits are val and test. should_decontaminate is true. The group is the paloma tag: 16 tasks named paloma_<config>. Hugging Face models need logits_cache=False for the full group or paloma_dolma_100_programing_languages.
No model card in ModelSpec reports this benchmark yet.