Bits-per-character compression of three raw corpora, used as an unsupervised linear proxy for knowledge, coding, and math benchmarks.
unassessed
| Category | generation |
|---|---|
| Subcategory | unsupervised bits-per-character on Common Crawl, GitHub Python, and arXiv-math |
| Page status | active |
| Metric | bits per character (BPC), also averaged across the three corpora |
| Direction | lower_is_better |
| Unit | BPC |
| Dataset size | 32344 |
| Dataset licence | CC-BY-NC-SA-4.0 |
| Publisher | Hong Kong University of Science and Technology, with Tencent |
LLM Compression does not ask questions. It scores how many bits a base language model needs, on average, to encode held-out raw text. Huang, Zhang, Shan, and He collect three external corpora aimed at three abilities: Common Crawl snapshots for knowledge and commonsense, GitHub Python for coding, and arXiv mathematics papers for math. They report bits per character (BPC) rather than bits per token so tokenizers can be compared. The claim in the COLM 2024 paper is that average BPC is almost linearly correlated with average scores on 12 downstream benchmarks. The HTML body says 30 public LLMs; the arXiv v2 abstract says 31.
No prompt and no target string. OpenCompass feeds the content field through SWCELossInferencer with a sliding window (paper reproduction: block_size 1900, stride 512), sums cross-entropy, and converts to BPC. ZeroRetriever; no in-context examples. Intended for base models, not chat-tuned models.
No model card in ModelSpec reports this benchmark yet.