lm-eval group that scores bits-per-byte and perplexity on 22 Pile component streams as a language-modelling eval, not a QA task.
unassessed
| Category | generation |
|---|---|
| Subcategory | 22-component language-modelling bits-per-byte / perplexity group |
| Page status | unknown |
| Metric | bits_per_byte (also word_perplexity, byte_perplexity) |
| Direction | lower_is_better |
| Unit | bpb |
| Dataset licence | other (Hub card); compilation/replication repo MIT; constituents keep their own licences |
| Publisher | EleutherAI |
The Pile is an 825 GiB English-targeted pretraining mix of 22 sources (Pile-CC, PubMed Central, Books3, OpenWebText2, ArXiv, GitHub, and others). Gao et al. also treat held-out Pile text as a language-modelling benchmark: the model assigns probabilities to documents, and the preferred figure is bits per UTF-8 byte (bpb), which does not depend on a tokenizer the way word perplexity does. lm-evaluation-harness exposes that eval as group `pile` with one rolling-loglikelihood task per component. There are no questions or labels. A score says how well the model predicts that domain's text, not whether it answers items correctly.
Rolling loglikelihood (`output_type: loglikelihood_rolling`). `doc_to_text` is empty; `doc_to_target` is the document `text`. Metrics: word_perplexity and byte_perplexity (weighted_perplexity, lower_is_better) and bits_per_byte (lower_is_better). `should_decontaminate: true` on the arXiv YAML. Each component YAML sets `test_split: train` on `dataset_path: EleutherAI/pile`.
No model card in ModelSpec reports this benchmark yet.