OpenCompass long-context suite that plants synthetic needles in English and Chinese haystacks at chosen lengths and depths, plus an Ancestral Trace Challenge.
unassessed
| Category | long-context |
|---|---|
| Subcategory | bilingual synthetic needle retrieval, multi-needle reasoning, and ancestral trace |
| Page status | superseded |
| Metric | OpenCompass v1: keyword hit 100 else 0.2 x Levenshtein similarity; v1 paper overall weights 0.4/0.3/0.3 on S-RT/M-RT/M-RS |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | MIT |
| Publisher | Shanghai AI Laboratory and Tsinghua University |
NeedleBench builds a long prompt and asks a model to recover or reason over planted facts ("needles"). Information-sparse tasks put one or more synthetic facts into a haystack of Paul Graham essays (English) or ChineseDomainModelingEval passages (Chinese), at a chosen token length and depth. Single-needle retrieval (S-RT) asks for one fact. Multi-needle retrieval (M-RT) asks for several. Multi-needle reasoning (M-RS) asks the model to combine two to five facts. The Ancestral Trace Challenge (ATC) is information-dense: every sentence is a kinship fact and there is no filler. English and Chinese. Text only. OpenCompass v1 length packs are 4k, 8k, 32k, 128k, 200k and 1000k (the 256k pack is in needlebench_v2, not this tree).
Long generated prompt in; short free-text answer out. OpenCompass v1 sparse scoring (NeedleBenchOriginEvaluator) is 100 if a core keyword appears, else 0.2 times Levenshtein similarity to the reference. ATC in the later paper uses boxed exact match; v1 configs do not. OpenCompass v1 uses NeedleBenchOriginDataset, a GPT-4 tokenizer length target, and a length buffer.
No model card in ModelSpec reports this benchmark yet.