Inspect Evals NIAH: plant English needles in long haystacks and score recall with a 1–10 LLM judge across a length-by-depth grid.
unassessed
| Category | long-context |
|---|---|
| Subcategory | Inspect Evals in-context retrieval over NeedleBench English haystacks |
| Page status | active |
| Metric | overall_accuracy (mean of numeric LLM-judge grades; also per-length and per-position means) |
| Direction | higher_is_better |
| Dataset size | 225 |
| Dataset licence | MIT |
| Publisher | UK AI Security Institute (Inspect Evals); NeedleBench assets from OpenCompass |
Inspect Evals niah tests whether a long-context model can retrieve a planted fact (the needle) from a long English passage (the haystack). The implementation, contributed by Owen Parsons, builds a length-by-depth grid at evaluation time. Haystacks and needle phrases come from Hugging Face opencompass/NeedleBench (en_haystack_texts and retrieval_needles, English needles only). This is the Inspect Evals task named niah, not NVIDIA [ruler](ruler.md) niah_* tasks, not OpenCompass [needlebench](needlebench.md) keyword scoring, and not Greg Kamradt's original notebook as a harness.
Generated long prompt in; short free-text answer out; second model call grades the answer. Default grid: min_context 10000, max_context 120000, n_contexts 15, n_positions 15, n_needles 1, sample_method fixed, n_runs 1 (eval.yaml dataset_samples 225). Solver inserts the needle then applies MAIN_PROMPT. Scorer is model_graded_qa with Kamradt-style 1/3/5/7/10 instructions, grade_pattern r"(\\d+)", and history limited to the needle question so the judge does not reread the haystack.
No model card in ModelSpec reports this benchmark yet.