Rolling loglikelihood on the first 10,000 Pile documents; a debug sample, not the official Pile test split.
unassessed
| Category | generation |
|---|---|
| Subcategory | 10k-document Pile sample, rolling perplexity / bits-per-byte |
| Page status | unknown |
| Metric | bits_per_byte (also word_perplexity, byte_perplexity) |
| Direction | lower_is_better |
| Unit | bpb |
| Dataset size | 10000 |
| Dataset licence | bigscience-bloom-rail-1.0 |
| Publisher | Neel Nanda (Hub packager); underlying corpus EleutherAI |
pile_10k asks a language model to assign probabilities to the first 10,000 documents of The Pile, as packaged by Neel Nanda on Hugging Face (`NeelNanda/pile-10k`). Each row is a `text` field plus `meta.pile_set_name`. lm-evaluation-harness scores rolling loglikelihood over that stream. The Hub card states the intended use: debugging models trained on The Pile, in the same spirit as `stas/openwebtext-10k`. It is not a question set and not Gao et al.'s published Pile test split.
`output_type: loglikelihood_rolling`. `test_split: train` (the dataset has only a train split). `doc_to_text` is empty; `doc_to_target` is `text`. Metrics: word_perplexity, byte_perplexity (weighted_perplexity, lower_is_better) and bits_per_byte (lower_is_better). YAML metadata version 1.0. The lm-eval README says the task is not in a group.
No model card in ModelSpec reports this benchmark yet.