A synthetic SQL-execution benchmark that generates unlimited, contamination-resistant tables and queries to probe reasoning and long-context comprehension from 200 tokens to 200K.
unassessed
| Category | long-context |
|---|---|
| Subcategory | synthetic SQL-execution suite for scalable long-context and reasoning evaluation |
| Page status | unknown |
| Metric | Exact Match (EM) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1168 |
| Dataset licence | Not established: no licence file was found in the GitHub repository (confirmed via the GitHub API) and no licence tag is present on the Hugging Face dataset card. |
S3Eval tests whether a model can execute a SQL query against a table it is shown, exactly and losslessly, as a proxy for two harder-to-measure capabilities at once: multi-step symbolic reasoning (each SQL keyword -- WHERE, GROUP BY, HAVING, ORDER BY and so on -- implies a distinct reasoning operation) and long-context comprehension, because tables can be generated at any length, from roughly 200 tokens up to 200K. Every table and query is synthetically generated: table headers are sampled from a list of English nouns, cell values are randomly generated integers, dates or strings, and the SQL itself is built from a context-free grammar with configurable nesting depth, keyword mix and numeric complexity, so no table or query in S3Eval corresponds to any real-world data. The authors also provide an alternate "multi-step instruction" framing that converts the same SQL query into a natural-language sequence of table operations, meant to isolate reasoning ability from a model's specific familiarity with SQL syntax.
Given a markdown-formatted synthetic table and a SQL query (or an equivalent natural-language multi-step instruction), generate the exact execution result as a table; zero-shot and few-shot variants exist, English by default with multilingual query support added later. Difficulty, table size and context length are all independently configurable by the generator.
No model card in ModelSpec reports this benchmark yet.