{ "body": "\n## What it measures\n\nLLM Compression scores a base language model as a lossless compressor of raw text. The model is not asked to answer questions. OpenCompass and the authors' script concatenate corpus documents, run a sliding window of 1,900 tokens with stride 512, and convert total cross-entropy into bits per character. Three corpora stand in for three abilities: Common Crawl for knowledge and commonsense, GitHub Python for coding, and arXiv math papers for mathematical reasoning. The COLM 2024 paper reports that this unsupervised BPC tracks average scores on 12 downstream benchmarks almost linearly for the base models they tested.\n\n## How it is scored\n\nBPC is total cross-entropy divided by (character count \u00d7 ln 2), matching PyTorch's natural-log loss. Lower is better. The authors unify the context window at 1,900 tokens so a long-context model cannot buy a better compressor score than the short-context downstream tasks would reflect. They average the three corpora for a headline figure and also report each domain. The GitHub README table lists Llama-3-70b at 0.359 average BPC as the best of the printed models. OpenCompass's config README prints a shorter table headed by qwen1.5-32b-hf at 0.4191. Those are different model sets, not a disagreement about the same run. Fine-tuned chat models are out of scope in the paper's main claim, because they no longer model arbitrary next tokens.\n\n## Dataset and licence\n\nHugging Face `hkust-nlp/llm-compression` has 32,344 public test rows (cc 20,159, python 10,977, arxiv_math 1,208) under CC BY-NC-SA 4.0. The Hub API lastModified stamp is 16 April 2024. The GitHub repository's code is MIT; the README says stricter upstream terms are honoured when a source requires them. The paper quotes character totals (131M / 98M / 101M) and crawl windows in 2023 rather than row counts. OpenCompass expects local jsonl files under `./data/llm-compression`, with the crawl file named `commoncraw.jsonl` even though the Hub config is `cc`. There is no hidden test split.\n\n## Who publishes it\n\nYuzhen Huang, Jinghan Zhang, and Junxian He at HKUST, with Zifei Shan at Tencent. The paper was posted on arXiv on 15 April 2024 (2404.09937). The GitHub README records OpenCompass support on 1 May 2024 and COLM 2024 acceptance on 20 September 2024, and its citation block points at OpenReview forum SHMj84U5SH. There is no separate live leaderboard URL beyond the static GitHub table.\n\n## Lineage\n\nThis page is the compression corpora and BPC protocol from that paper, as wired in OpenCompass under `llm_compression`. It is not [Inference-PPL](inference_ppl.md), which averages token NLL on labeled spans of a local reasoning jsonl. It is also not WikiText perplexity ([wikitext](wikitext.md)). The authors present BPC as a refreshable unsupervised stand-in for MMLU-style, HumanEval-style, and GSM8K/MATH-style averages, not as a replacement for those pages. No successor id is recorded here.\n\n## Saturation and contamination\n\nBPC has no 100% ceiling. Among models printed on the GitHub table, Llama-3-70b is best at 0.359 average, but later models were not read here. Because the metric is just language-model loss on public crawls, a model trained on those crawls can look strong without being a better reasoner. The authors chose 2023 windows to limit overlap with older dumps and argue that new crawls can be swapped in. Treat a low BPC as compression skill on these files, then check the matching downstream suite before claiming knowledge, code, or math gains.\n\n## How to run it\n\nIn OpenCompass, `--datasets llm_compression` builds three dataset abbrs with `SWCELossInferencer` (block_size 1900, stride 512) and `BPCEvaluator`. Download the Hub configs into `./data/llm-compression` first; the loader reads local json/jsonl only. The authors' `code/evaluation/main.py` is the reference outside OpenCompass. Keep flash-attention and GPU settings explicit. Do not score chat checkpoints with this protocol if you want numbers comparable to the paper: the paper's main plots are base models.\n\n## Reading the numbers\n\nA lower average BPC means the model assigned higher probability to these three crawls under a 1,900-token window. It does not mean the model is better at GSM8K or HumanEval on its own, only that the authors saw a near-linear fit for their 2024 base-model pool. Always name the subset (cc vs python vs arxiv_math): a code specialist can look strong on python and weak on crawl. Do not mix OpenCompass's short example table with the GitHub leaderboard, and do not compare BPC to [Inference-PPL](inference_ppl.md) NLL. If two models use very different tokenizers, BPC is the quantity that is supposed to remain comparable; bits-per-token is not.\n", "build": { "built_at": "2026-09-09T16:56:50+00:00", "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11", "eligibility_as_of": "2026-09-09" }, "disposition": { "canonical_id": "llm_compression", "reasons": [], "status": "unassessed", "verified_results": [] }, "models_covered": [], "page": { "aliases": [ "llm-compression", "Compression Represents Intelligence Linearly", "BPC compression corpora" ], "category": "generation", "contamination": { "note": "Corpora are public raw text from Common Crawl, GitHub, and arXiv, collected in mid-to-late 2023 to reduce overlap with older pretraining dumps. The paper argues the metric can be refreshed with new crawls. No source opened here measured leakage of these exact jsonl files into a named model.\n", "risk": "medium" }, "dataset": { "languages": [ "en" ], "license": "CC-BY-NC-SA-4.0", "modalities": [ "text" ], "public_test_set": true, "size": 32344, "size_note": "Hugging Face datasets-server for hkust-nlp/llm-compression reports 32,344 test rows: cc 20,159, python 10,977, arxiv_math 1,208. The paper Table 1 instead quotes character counts and crawl windows: Common Crawl 131M characters (September\u2013October 2023), GitHub 98M (June\u2013September 2023), ArXiv 101M (September\u2013October 2023). The Hub card's size_categories field is the coarse bucket 10K