{
 "body": "\n## What it measures\n\nGraphWalks fills a model's context window with a directed graph -- nodes named with hexadecimal hashes,\nedges listed one per line -- then asks it to perform one of two operations starting from a random node:\na breadth-first search that returns the nodes reachable at an exact depth, or a \"parents\" query that\nreturns every node with a direct edge into a given target. OpenAI introduced it alongside OpenAI-MRCR (a\nneedle-disambiguation benchmark) when launching GPT-4.1, specifically because a model could in principle\nsolve a needle-retrieval or MRCR-style problem with a single read-through of the prompt. GraphWalks is\nbuilt so a single pass is not enough: correctly tracing even a shallow search requires jumping between\nscattered positions in the context and combining what is found there, targeting multi-hop reasoning\nrather than single-pass retrieval.\n\n## How it is scored\n\nModels are graded on F1 over the returned node set against the ground-truth set: recall is the fraction\nof correct nodes returned, precision is the fraction of returned nodes that are correct, and F1 is their\nharmonic mean. Answers must appear on the prompt's final line as \"Final Answer: [node1, node2, ...]\";\nresponses that do not match this format score as empty. Because the underlying dataset is fixed rather\nthan regenerated per run, and because vendors have found and fixed genuine scoring bugs after\npublication (see Lineage and How to run it), an F1 score's exact value depends on which version of the\ngrading logic produced it, not solely on the model being tested.\n\n## Dataset and licence\n\nThe Hugging Face dataset `openai/graphwalks` holds about 1,150 rows in a single MIT-licensed \"train\"\nsplit, each row a prompt (three worked examples plus one graph-and-operation problem), its correct\nanswer node set, the prompt's character length, and whether the problem is a `bfs` or `parents` task.\nRows are split across two parquet files by prompt length -- `graphwalks_128k_and_shorter.parquet` and\n`graphwalks_256k_to_1mil.parquet` -- rather than by train and test; the dataset card does not publish a\nrow count broken down by problem type within each file. There is no separate held-out test split: the\nsame published rows are the entire evaluation set.\n\n## Who publishes it\n\nOpenAI released GraphWalks on Hugging Face on 11 April 2025 and announced it three days later in the\n\"Introducing GPT-4.1 in the API\" blog post, alongside OpenAI-MRCR, as one of two new long-context evals\nit was open-sourcing. No accompanying academic paper was found; the dataset card and blog post are the\nprimary documentation, and OpenAI continues to maintain the dataset, including a February 2026 bugfix\ncredited to an external system card (see Lineage).\n\n## Lineage\n\nGraphWalks has no formal predecessor; it was released as a companion to OpenAI-MRCR, a different\nlong-context benchmark testing disambiguation between similar requests rather than graph traversal. Two\ncontext-length-specific benchmark pages exist in this repository as `graphwalks_bfs_256k_1m` and\n`graphwalks_parents_256k_1m`, both `lineage.family: graphwalks`; a shorter-context bucket (128K tokens\nor less) and a full-range \"1M\" cut used in at least one vendor system card are documented in the\nunderlying dataset but do not yet have their own pages here. The dataset's own changelog credits a\ncorrection to Anthropic's Claude Opus 4.6 system card, an unusual instance of a benchmark's maintainer\nfixing a bug a downstream evaluator found.\n\n## Saturation and contamination\n\nAt launch, OpenAI reported GPT-4.1 scoring 61.7% aggregate accuracy, \"matching the performance of o1 and\nbeating GPT-4o handily,\" without breaking the number down by context length. At the harder, longest\ncontext band that this repository's two variant pages track, scores split sharply by task: \"parents\"\nqueries are close to solved for the strongest model (Claude Opus 4.6 above 95%) but leave a real gap to\nother models (Claude Sonnet 4.5 near 81%), while BFS queries leave substantial headroom across the board\n(Claude Opus 4.6 near 61%, Claude Sonnet 4.5 near 45%). Contamination risk is medium: the graphs are\nsynthetic and not drawn from any pre-existing public corpus, but the fixed set of published prompts and\nanswers has been downloadable since April 2025, so direct memorisation cannot be ruled out for models\ntrained after that date.\n\n## How to run it\n\nNo major third-party harness carries GraphWalks. The grading code lives in the Hugging Face dataset\ncard's README. Anthropic has published specific, documented departures from that reference code: a fix\nto how the F1 formula handles an empty ground truth, a clarified BFS prompt requiring nodes at exactly\nthe target depth rather than \"reachable\" nodes more loosely, and a correction to 24 of 400 mislabeled\n\"parents\" ground-truth rows in the shorter context file. Scores computed before and after these fixes\nare not guaranteed to be comparable, and the benchmark's short public history means this kind of\ncorrection may recur.\n\n## Reading the numbers\n\nA high GraphWalks score shows a model can correctly trace a multi-hop path through a graph scattered\nacross a long context, not merely retrieve one fact from it -- a different skill from\nneedle-in-a-haystack tests. Because BFS and parents queries behave very differently, and scores depend\non which context-length bucket was tested, do not read one aggregate GraphWalks number as\nrepresentative; check the specific task and context length, and prefer this repository's\n`graphwalks_bfs_256k_1m` and `graphwalks_parents_256k_1m` pages for numbers tied to a stated context\nband. Given the documented scoring-bug history, treat a score reported before 2026 with some caution\nrelative to one reported after.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "graphwalks",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Graphwalks"
  ],
  "category": "long-context",
  "contamination": {
   "note": "Graphs and node names are synthetically generated, not drawn from an existing public corpus, so the classic risk of an old, indexed answer key leaking into training data does not apply the way it does for a scraped-text benchmark. But the dataset itself, prompts and correct answers included, has been openly downloadable from Hugging Face since April 2025, so a model trained directly on these exact published rows could match answers by memorising this fixed set rather than by tracing the graph fresh, since GraphWalks reuses the same published examples across evaluation runs rather than generating new graphs each time.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 1150,
   "size_note": "About 1,150 rows in the Hugging Face dataset's single \"train\" split (used purely for evaluation, not training), each with a `prompt`, an `answer_nodes` list, a `prompt_chars` length and a `problem_type` of either `bfs` or `parents`. The data is split across two parquet files by prompt length: `graphwalks_128k_and_shorter.parquet` and `graphwalks_256k_to_1mil.parquet`; the dataset card does not break out how many rows of each problem type fall in each file, so an exact per-bucket, per-task count is not established here.\n",
   "splits": "single 'train' split (~1,150 rows), bucketed into two files by prompt length rather than by train/test",
   "url": "https://huggingface.co/datasets/openai/graphwalks"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice K",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "No major third-party harness was found to carry GraphWalks. The grading code is published in the Hugging Face dataset card's README. Anthropic's Claude Opus 4.6 system card documents three specific departures from that reference code when it ran the eval: an edge-case fix to the F1 formula for empty ground truth, a prompt clarification that BFS answers must include only nodes at exactly the requested depth (the original public prompt was ambiguous about this), and a fix for 24 of 400 \"parents\" rows in the shorter, `128k_and_shorter` file whose ground truth wrongly included the target node itself due to graph self-loops -- a fix OpenAI's dataset changelog credits to that same system card.\n"
  },
  "id": "graphwalks",
  "last_updated": "2026-03",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "graphwalks_bfs_256k_1m",
    "graphwalks_parents_256k_1m"
   ]
  },
  "measures": "GraphWalks fills a model's context window with a directed graph, written out as an edge list of hexadecimal-hash node names, then asks the model to perform one of two operations starting from a random node: a breadth-first search (BFS) that returns the nodes reachable at an exact depth, or a \"parents\" query that returns every node with a direct edge into a given target node. OpenAI built it specifically because simpler long-context tests -- finding one \"needle\" fact, or its own OpenAI-MRCR benchmark, which disambiguates between several similar requests -- can in principle be solved by one read-through of the prompt. GraphWalks cannot: correctly tracing even a shallow BFS requires jumping between multiple, scattered positions in the context and combining them, so it specifically tests multi-hop reasoning over a long context rather than single-pass retrieval.\n",
  "metric": {
   "baseline_note": "F1 is computed from the overlap between the model's returned node set and the ground-truth set: recall = overlap / |ground truth|, precision = overlap / |model's set|, F1 = the harmonic mean of the two. OpenAI's original scoring scores an empty ground truth against any non-empty prediction as an F1 of 1.0 by the standard formula's typical convention (0 when precision+recall is 0); Anthropic's Claude Opus 4.6 system card documents changing this specific edge case so that an empty prediction against an empty ground truth scores 1.0 instead of 0, which is not the same fix and makes scores computed under the two conventions not directly comparable. No fixed random-guess or human baseline is established, since both depend on the size of the (randomly generated) graph in a given prompt.\n",
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "F1 score (node-set overlap)",
   "unit": "F1 x100"
  },
  "name": "GraphWalks",
  "page_kind": "family",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "OpenAI",
   "url": "https://huggingface.co/datasets/openai/graphwalks"
  },
  "released": "2025-04",
  "repo_url": "https://huggingface.co/datasets/openai/graphwalks",
  "saturation": {
   "as_of": "",
   "note": "OpenAI reported GPT-4.1 at 61.7% aggregate accuracy at the benchmark's April 2025 launch, \"matching the performance of o1 and beating GPT-4o handily,\" without a context-length breakdown. At the harder, longest context band, Anthropic's Claude Opus 4.6 system card (2026-02) reports a wide spread by task: parents queries are close to solved for the best model (Claude Opus 4.6 at 95.1-95.4%, Claude Sonnet 4.5 at 81.0%), while BFS queries still leave real headroom (Claude Opus 4.6 at 61.1-61.5%, Claude Sonnet 4.5 at 44.9%) -- see the two variant pages for the specific numbers this family page is averaging over.\n",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Introducing GPT-4.1 in the API, OpenAI (2025-04-14)",
    "url": "https://openai.com/index/gpt-4-1/"
   },
   {
    "accessed": "2026-09-08",
    "title": "openai/graphwalks dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/openai/graphwalks"
   },
   {
    "accessed": "2026-09-08",
    "title": "Claude Opus 4.6 System Card, section 2.18.2 GraphWalks (Anthropic, 2026-02)",
    "url": "https://www-cdn.anthropic.com/6a5fa276ac68b9aeb0c8b6af5fa36326e0e166dd/Claude%20Opus%204.6%20System%20Card.pdf"
   }
  ],
  "status": "active",
  "subcategory": "multi-hop graph traversal in long context",
  "summary": "OpenAI's long-context eval that hides a directed graph of hashed node names in the prompt and asks the model to run a breadth-first search or list a node's parents.",
  "tags": [
   "long-context",
   "graph-traversal",
   "multi-hop-reasoning",
   "openai",
   "synthetic"
  ],
  "task_format": "A prompt containing three worked examples, then a large directed graph as an edge list of hex-hash node ids, then an instruction to perform a BFS from a node at a given depth or to find a node's parents. The model must reply with the resulting node set on the prompt's final line, formatted as \"Final Answer: [node1, node2, ...]\".\n"
 }
}