OpenAI's long-context eval that hides a directed graph of hashed node names in the prompt and asks the model to run a breadth-first search or list a node's parents.
unassessed
| Category | long-context |
|---|---|
| Subcategory | multi-hop graph traversal in long context |
| Page status | active |
| Metric | F1 score (node-set overlap) |
| Direction | higher_is_better |
| Unit | F1 x100 |
| Dataset size | 1150 |
| Dataset licence | MIT |
| Publisher | OpenAI |
GraphWalks fills a model's context window with a directed graph, written out as an edge list of hexadecimal-hash node names, then asks the model to perform one of two operations starting from a random node: a breadth-first search (BFS) that returns the nodes reachable at an exact depth, or a "parents" query that returns every node with a direct edge into a given target node. OpenAI built it specifically because simpler long-context tests -- finding one "needle" fact, or its own OpenAI-MRCR benchmark, which disambiguates between several similar requests -- can in principle be solved by one read-through of the prompt. GraphWalks cannot: correctly tracing even a shallow BFS requires jumping between multiple, scattered positions in the context and combining them, so it specifically tests multi-hop reasoning over a long context rather than single-pass retrieval.
A prompt containing three worked examples, then a large directed graph as an edge list of hex-hash node ids, then an instruction to perform a BFS from a node at a given depth or to find a node's parents. The model must reply with the resulting node set on the prompt's final line, formatted as "Final Answer: [node1, node2, ...]".
No model card in ModelSpec reports this benchmark yet.