{
 "body": "\n## What it measures\n\nInspect Evals niah hides one English fact in a long English passage and asks the model to retrieve it. Context length and needle depth vary on a grid. Owen Parsons contributed the Inspect task. Haystacks and needles are loaded from Hugging Face `opencompass/NeedleBench`, then trimmed and inserted with tiktoken. Needles that are not labelled English are dropped. The test is retrieval under length, not multi-needle reasoning and not OpenCompass [needlebench](needlebench.md) keyword scoring.\n\n## How it is scored\n\nA task model answers from the full haystack. A judge model then grades only the question, the gold, and that answer on a 1\u201310 rubric that the Inspect README attributes to Kamradt. The Kamradt main README opened here is a v2 CLI and does not reprint that list. The haystack is not passed to the judge. Code averages the numeric grades as `overall_accuracy` and as per-length and per-position means. Version 3-A (2026-08-20) treats a digit-free judge completion as unscored (NaN), not as 0. If no judge is set, scoring uses the active model; a different judge can be passed.\n\n## Dataset and licence\n\nThere is no frozen test file. `eval.yaml` records 225 samples for the default 15 lengths \u00d7 15 depths \u00d7 1 run. Changing `-T` knobs changes the count. Assets are `en_haystack_texts` and `retrieval_needles` at revision `651d7c8f4eae047b3f47ca24e92e09f2acf64af5`. inspect_evals is MIT. Hugging Face tags NeedleBench MIT as well. English only in this filter.\n\n## Who publishes it\n\nUK AI Security Institute ships the task in inspect_evals (`inspect_evals/niah`). NeedleBench data come from OpenCompass (paper arXiv:2407.11963). The README scoring write-up points at Kamradt's GitHub, not at a new NIAH paper. `eval.yaml` sets `arxiv` to 2407.01437, which is IBM's Larimar memory paper, not this Inspect grid.\n\n## Lineage\n\nKamradt's Needle in a Haystack notebook popularised the heatmap. OpenCompass [needlebench](needlebench.md) packaged essays and needles; [needlebench_v2](needlebench_v2.md) follows. NVIDIA [ruler](ruler.md) adds niah_single_* and multi-key variants with exact-match scoring. This id is the Inspect Evals wrapper: NeedleBench English assets, Kamradt-style LLM judge, default 10k\u2013120k grid. It is not an alias of those pages.\n\n## Saturation and contamination\n\nNo numeric overall_accuracy table was opened here. README plots cover Mistral small and medium on a shorter 2k\u201327k grid and warn that the judge sometimes marks a correct fictional needle as unrelated. Needles and haystacks are public. Placement is generated, so the prompt is not a single leaked JSON, but the facts are.\n\n## How to run it\n\nInstall `inspect-evals[niah]` (pandas). Run `inspect eval inspect_evals/niah`. Defaults are 10k\u2013120k tokens, 15 lengths, 15 positions, one fixed needle, one run (225 samples). Override with `-T min_context=...` and related knobs. Tokenisation uses tiktoken; unknown model names fall back to a default encoder. Compare only runs that share grid, needle sampling, judge model, and eval version (3-A changed unscored handling).\n\n## Reading the numbers\n\nA high `overall_accuracy` means the judge assigned high 1\u201310 grades on that grid, not that the model matched a string. The README shows judges clustering at 1 and 10, and sometimes penalising a correct needle because it is fictional. Do not compare this number to [needlebench](needlebench.md) keyword scores or to [ruler](ruler.md) `niah_single_1`. Quote context length, depth grid, `n_runs`, judge model, and eval version. The default 225-sample 10k\u2013120k run is not the Mistral heatmap protocol in the README.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "niah",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "inspect_evals/niah",
   "Needle in a Haystack (Inspect Evals)"
  ],
  "category": "long-context",
  "contamination": {
   "note": "Needles, questions, and haystacks are public in opencompass/NeedleBench. Placement is generated at run time, so the exact prompt is not a fixed dump, but the facts and essays are. Scoring is a second model call (default: the active task model). Judge errors (treating fictional needles as world-knowledge fails) can look like retrieval fails.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 225,
   "size_note": "eval.yaml dataset_samples is 225, matching the default 15\u00d715\u00d71 grid, not a frozen item file. Size changes with min_context, max_context, n_contexts, n_positions, n_needles, and n_runs. README experimental plots used a different grid (9 lengths from 2k to 27k tokens, 10 positions, n_runs 10 or 5) and are not the default 225. Hugging Face assets are pinned at revision 651d7c8f4eae047b3f47ca24e92e09f2acf64af5.\n",
   "splits": "generated at eval time from NeedleBench test configs; no train split",
   "url": "https://huggingface.co/datasets/opencompass/NeedleBench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-062 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-062"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "niah",
   "lm_eval": "",
   "opencompass": "",
   "other": "inspect eval inspect_evals/niah; extra inspect-evals[niah] (pandas). eval.yaml version 3-A, group Reasoning, dataset_samples 225. eval.yaml arxiv field points at 2407.01437 (Larimar), which is not this task's method paper.\n"
  },
  "id": "niah",
  "last_updated": "2026-08",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Inspect Evals niah tests whether a long-context model can retrieve a planted fact (the needle) from a long English passage (the haystack). The implementation, contributed by Owen Parsons, builds a length-by-depth grid at evaluation time. Haystacks and needle phrases come from Hugging Face opencompass/NeedleBench (en_haystack_texts and retrieval_needles, English needles only). This is the Inspect Evals task named niah, not NVIDIA [ruler](ruler.md) niah_* tasks, not OpenCompass [needlebench](needlebench.md) keyword scoring, and not Greg Kamradt's original notebook as a harness.\n",
  "metric": {
   "baseline_note": "The judge prompt asks for 1, 3, 5, 7, or 10 only. README heatmaps treat those integers as the plotted scores and note that judges often emit 1 or 10 (mistral-small sometimes emitted 0). subset_accuracy_combinations averages the numeric Score.value as overall_accuracy. Whether inspect_ai rescales that capture to 0\u20131 was not confirmed from inspect_ai source opened here; the rubric maximum is 10. No human-rater figure is published for this Inspect grid.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 10.0,
   "name": "overall_accuracy (mean of numeric LLM-judge grades; also per-length and per-position means)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "NIAH (Inspect Evals Needle in a Haystack)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [
    "Owen Parsons"
   ],
   "org": "UK AI Security Institute (Inspect Evals); NeedleBench assets from OpenCompass",
   "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/niah"
  },
  "released": "2025-12",
  "repo_url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/niah",
  "saturation": {
   "as_of": "",
   "note": "README heatmaps for mistral-small-latest and mistral-medium-latest are images without a published overall_accuracy table. The text says the medium model scored worse than the small model, partly from judge \"unrelated\" refusals. No numeric top cell was transcribed from those plots. Default 10k\u2013120k grid results were not published in the README.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Inspect Evals niah README (grid defaults, Kamradt rubric, changelog through 3-A)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/niah/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "eval.yaml (version 3-A, dataset_samples 225, NeedleBench pin, arxiv 2407.01437)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/niah/eval.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "niah.py (defaults 10k\u2013120k, 15\u00d715, TOKEN_BUFFER 100)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/niah/niah.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "dataset_generation.py (NeedleBench configs, English needle filter, revision pin)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/niah/utils/dataset_generation.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "scoring.py (1\u201310 rubric, overall_accuracy, unscored parse misses in 3-A)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/niah/utils/scoring.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "prompting.py (MAIN_PROMPT and question wrapper)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/niah/utils/prompting.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals MIT License (UK AI Security Institute, 2024)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "opencompass/NeedleBench dataset card (MIT; created 2024-07-21)",
    "url": "https://huggingface.co/datasets/opencompass/NeedleBench"
   },
   {
    "accessed": "2026-09-08",
    "title": "NeedleBench paper (dataset source; not cited in niah README)",
    "url": "https://arxiv.org/abs/2407.11963"
   },
   {
    "accessed": "2026-09-08",
    "title": "Larimar NIAH paper (eval.yaml arxiv field; different work)",
    "url": "https://arxiv.org/abs/2407.01437"
   },
   {
    "accessed": "2026-09-08",
    "title": "Kamradt Needle in a Haystack (current main README is v2 CLI; 1-10 rubric not on that page)",
    "url": "https://github.com/gkamradt/LLMTest_NeedleInAHaystack"
   }
  ],
  "status": "active",
  "subcategory": "Inspect Evals in-context retrieval over NeedleBench English haystacks",
  "summary": "Inspect Evals NIAH: plant English needles in long haystacks and score recall with a 1\u201310 LLM judge across a length-by-depth grid.",
  "tags": [
   "long-context",
   "retrieval",
   "needle-in-a-haystack",
   "inspect-evals",
   "llm-judge"
  ],
  "task_format": "Generated long prompt in; short free-text answer out; second model call grades the answer. Default grid: min_context 10000, max_context 120000, n_contexts 15, n_positions 15, n_needles 1, sample_method fixed, n_runs 1 (eval.yaml dataset_samples 225). Solver inserts the needle then applies MAIN_PROMPT. Scorer is model_graded_qa with Kamradt-style 1/3/5/7/10 instructions, grade_pattern r\"(\\\\d+)\", and history limited to the needle question so the judge does not reread the haystack.\n"
 }
}