{
 "body": "\n## What it measures\n\njsonschema_bench is a constrained-generation task rather than a knowledge test: given a real-world JSON\nSchema, the model has to produce a JSON object that satisfies it. The underlying JSONSchemaBench project\ncollects roughly 10,000 such schemas across ten domains -- five GitHub-sourced difficulty tiers from\ntrivial to ultra, plus Kubernetes configuration schemas, GlaiveAI function-call schemas, the JSON Schema\nStore, Snowplow event schemas and Washington Post resource schemas -- assembled to evaluate both\nconstrained-decoding frameworks and a model's native, unconstrained ability to hit a schema without any\ndecoding-time enforcement. lm-evaluation-harness implements three of those ten domains: the GitHub \"easy,\"\n\"medium\" and \"hard\" difficulty tiers, grouped together under the shared tag `jsonschema_bench`.\n\n## How it is scored\n\nThe model is shown a JSON Schema as raw text, preceded by a fixed two-shot demonstration built from two\nunrelated example schemas and their correct JSON objects, and generates until a blank line; the harness\nstrips code-fence markers before scoring. Two binary metrics are computed per sample and averaged:\njson_validity, which checks only that the output parses as JSON, and schema_compliance, which additionally\nvalidates the parsed object against the schema under the JSON Schema Draft 2020-12 specification (with\nextra checks for the ipv4, ipv6 and uuid string formats). schema_compliance can never exceed json_validity,\nsince syntactically valid JSON that breaks the schema's constraints is a common failure mode on its own.\n\n## Dataset and licence\n\nThe three tiers this harness runs hold 577 (easy), 586 (medium) and 368 (hard) test-split schemas -- 1,531\nin total -- under an MIT licence, confirmed directly against the released Hugging Face dataset. The full\nJSONSchemaBench project is larger: ten domains totalling 9,558 schemas across its combined\ntrain/validation/test splits, which the dataset's own card rounds to \"approximately 10,000.\" Seven domains\nin that fuller release -- Github_trivial, Github_ultra, Glaiveai2K, JsonSchemaStore, Kubernetes, Snowplow\nand WashingtonPost -- are not implemented by this harness task, so a jsonschema_bench score reflects only\nthe three GitHub difficulty tiers, not the published benchmark's full domain coverage.\n\n## Who publishes it\n\n\"JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models\" was posted to arXiv in\nJanuary 2025 by Saibo Geng, Hudson Cooper, Micha\u0142 Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin,\nRobert West, Eric Horvitz and Harsha Nori, last revised in February 2025. The project is hosted under the\nepfl-dlab organisation on both GitHub and Hugging Face; a separate `guidance-ai` GitHub organisation hosts a\nrelated repository that points back to epfl-dlab's codebase as the canonical implementation and also hosts\nMaskBench, a distinct, purely performance-oriented benchmark for constrained-decoding mask computation time\nthat this page does not otherwise describe.\n\n## Lineage\n\nNo predecessor or successor is tracked for this id in this repository, and no other page here yet covers\nJSONSchemaBench's other seven domains or the constrained-decoding frameworks (Guidance, Outlines, XGrammar\nand others) the paper evaluates alongside native model generation. The paper itself pairs the benchmark with\nthe pre-existing official JSON Schema Test Suite for spec-conformance testing, a different, framework-level\ntest not implemented as part of this harness task.\n\n## Saturation and contamination\n\nThis benchmark separates models by difficulty tier rather than sitting at one ceiling. The project's own\ncommunity leaderboard -- a contributor-submitted spreadsheet linked from its README, not part of the\npaper's own controlled evaluation -- reads GPT-4o at roughly 96.9% schema compliance on the easy tier\nagainst about 87.8% on hard, and Qwen2.5-32B-Instruct at roughly 94.3% easy against 74.7% hard, with weaker\nmodels dropping much further on hard schemas. No date is attached to those entries. That pattern -- easy\nclose to a ceiling for capable models, hard still spreading scores by tens of points -- is why this page\nmarks the benchmark `watch` rather than `open` or `saturated`. Contamination risk is low: most items have no\nsingle correct output to memorise, since any schema-valid JSON object counts and there are infinitely many\nof them, though a model may have seen the public schemas themselves (and typical conforming data) during\npretraining.\n\n## How to run it\n\n`lm_eval --tasks jsonschema_bench` runs all three implemented tiers via their shared tag and reports each\ntier's json_validity and schema_compliance separately -- there is no blended score across tiers. Each tier\nis also runnable alone (for example `--tasks jsonschema_bench_hard`). The task requires the `jsonschema`\nPython package with its `format` extra installed. Because the reference implementation measures a model's\nnative generation rather than output produced under grammar-constrained decoding, scores from this harness\ntask are not directly comparable to a constrained-decoding framework's numbers from the paper's own\nframework comparison, which is a different evaluation setting over the same underlying schemas.\n\n## Reading the numbers\n\nA high schema_compliance score shows a model can turn a novel, real-world JSON Schema into syntactically\nvalid, spec-conformant JSON without any decoding-time enforcement -- useful for anticipating how well an\nAPI-integration or tool-calling pipeline will behave without constrained decoding switched on. It says\nnothing about whether the generated data is factually sensible, only that its shape and types match the\nschema. Because scores drop sharply from the easy to the hard tier for every model class, and the harness\nimplements only three of the published benchmark's ten domains, a single reported percentage is only\ninformative once it is clear which tier, and how much of the full JSONSchemaBench, it actually covers.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "jsonschema_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "JSON Schema Bench"
  ],
  "category": "generation",
  "contamination": {
   "note": "Unlike a fixed-answer benchmark, most items here have no single correct output to leak: any JSON object that validates against the schema counts, and there are infinitely many such objects, so memorising one specific \"gold\" response is not the obvious way to game this benchmark. The schemas themselves are sourced from public repositories (GitHub, Kubernetes configs, JSON Schema Store, Snowplow, Washington Post), so a model may have seen the schema and typical conforming instances during pretraining, which could make compliant generation easier through familiarity -- but that is a weaker and less direct risk than the answer-key leakage a fixed-answer benchmark faces.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [],
   "license": "MIT",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 1531,
   "size_note": "lm-evaluation-harness implements three of the published benchmark's ten domains -- the GitHub-sourced \"easy,\" \"medium\" and \"hard\" difficulty tiers -- whose test splits hold 577, 586 and 368 schemas respectively (1,531 total), confirmed directly against the Hugging Face dataset's per-config split sizes. The full JSONSchemaBench release covers all ten domains and totals 9,558 schemas across its combined train/validation/test splits (5,754 / 937 / 2,867), which its own dataset card rounds to \"approximately 10,000.\" The other seven domains -- Github_trivial, Github_ultra, Glaiveai2K, JsonSchemaStore, Kubernetes, Snowplow and WashingtonPost -- are part of the published dataset but are not wired into this harness task.\n",
   "splits": "each of the three implemented tiers has its own train/val/test split on Hugging Face; the harness scores against 'test' and uses six hardcoded few-shot exemplars in its config rather than sampling from 'train'",
   "url": "https://huggingface.co/datasets/epfl-dlab/JSONSchemaBench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "jsonschema_bench",
   "opencompass": "",
   "other": "The runnable component task names are jsonschema_bench_easy, jsonschema_bench_medium and jsonschema_bench_hard (Hugging Face configs Github_easy, Github_medium, Github_hard respectively). They share only a `tag: jsonschema_bench` in the harness, not a `group:` with an aggregate metric, so `--tasks jsonschema_bench` runs all three and reports each tier's own json_validity and schema_compliance separately rather than blending them into one number. Requires `pip install \"jsonschema[format]\"`.\n"
  },
  "id": "jsonschema_bench",
  "last_updated": "2025-02",
  "leaderboard_url": "https://docs.google.com/spreadsheets/d/1gloUwsKiiOgrBmxbNluh-2_-vV_6RgaPji6KAD52pv0/edit?usp=sharing",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "jsonschema_bench is a constrained-generation task, not a knowledge test: given a real-world JSON Schema, the model must produce a JSON object that satisfies it. The underlying JSONSchemaBench project collects roughly 10,000 real schemas across ten domains -- five GitHub difficulty tiers (trivial through ultra) plus Kubernetes configuration schemas, GlaiveAI function-call schemas, the JSON Schema Store, Snowplow event schemas and Washington Post resource schemas -- built to evaluate both constrained-decoding frameworks (the paper tests Guidance, Outlines, Llamacpp, XGrammar, OpenAI and Gemini) and a model's native, unconstrained ability to produce schema-conformant output. lm-evaluation-harness implements three of those ten domains: the Github_easy, Github_medium and Github_hard difficulty tiers, grouped under the shared tag `jsonschema_bench`.\n",
  "metric": {
   "baseline_note": "json_validity checks only that the output parses as JSON at all. schema_compliance additionally validates the parsed object against the schema under the JSON Schema Draft 2020-12 specification, with added format-keyword checks for ipv4, ipv6 and uuid; a per-sample validation timeout treats a schema that cannot be checked in time as non-compliant rather than hanging the run. schema_compliance is always less than or equal to json_validity, since valid JSON that violates the schema is common. No random or human baseline applies to open-ended structured generation.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "json_validity and schema_compliance, both binary per-sample metrics averaged across each tier",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "JSONSchemaBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2501.10868",
   "title": "JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models",
   "url": "https://arxiv.org/abs/2501.10868",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Saibo Geng",
    "Hudson Cooper",
    "Micha\u0142 Moskal",
    "Samuel Jenkins",
    "Julian Berman",
    "Nathan Ranchin",
    "Robert West",
    "Eric Horvitz",
    "Harsha Nori"
   ],
   "org": "Hosted under the epfl-dlab (EPFL Data Science Lab) organisation on GitHub and Hugging Face; individual authors' institutional affiliations were not confirmed from a source read for this page",
   "url": "https://github.com/epfl-dlab/jsonschemabench"
  },
  "released": "2025-01",
  "repo_url": "https://github.com/epfl-dlab/jsonschemabench",
  "saturation": {
   "as_of": "",
   "note": "The project's own community leaderboard (a contributor-submitted Google Sheet linked from its README, not an audited part of the paper) reads GPT-4o at roughly 96.9% schema compliance on the easy tier and 87.8% on the hard tier, with Qwen2.5-32B-Instruct at roughly 94.3% easy and 74.7% hard, and smaller or older models dropping much further on hard schemas (one 7B-class model reads around 5% hard-tier schema compliance). No exact date is attached to that spreadsheet's entries, so no `as_of` is recorded for the top_score above beyond the fact that it was read from that leaderboard rather than a controlled, dated evaluation. The pattern across tiers -- easy close to a ceiling for strong models, hard still separating them by tens of points -- is why this page records `watch` rather than `open` or `saturated`.\n",
   "status": "watch",
   "top_score": 87.8
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "jsonschema_bench task README, lm-evaluation-harness",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/jsonschema_bench/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "jsonschema_bench_easy.yaml: 2-shot config, dataset_name Github_easy",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/jsonschema_bench/jsonschema_bench_easy.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "metrics.py: json_validity and schema_compliance implementations",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/jsonschema_bench/metrics.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models",
    "url": "https://arxiv.org/abs/2501.10868"
   },
   {
    "accessed": "2026-09-08",
    "title": "epfl-dlab/JSONSchemaBench metadata and per-config split sizes, Hugging Face API",
    "url": "https://huggingface.co/api/datasets/epfl-dlab/JSONSchemaBench"
   },
   {
    "accessed": "2026-09-08",
    "title": "guidance-ai/jsonschemabench README: ten-domain dataset table and totals",
    "url": "https://raw.githubusercontent.com/guidance-ai/jsonschemabench/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "JSONSchemaBench community leaderboard (Google Sheet, contributor-submitted)",
    "url": "https://docs.google.com/spreadsheets/d/1gloUwsKiiOgrBmxbNluh-2_-vV_6RgaPji6KAD52pv0/edit?usp=sharing"
   }
  ],
  "status": "active",
  "subcategory": "constrained structured-output generation: producing JSON that validates against a supplied JSON Schema",
  "summary": "JSONSchemaBench tests whether a model can generate JSON that both parses and validates against a supplied JSON Schema, drawn from real-world schemas across ten source domains.",
  "tags": [
   "generation",
   "constrained-decoding",
   "json",
   "structured-output",
   "real-world-schemas"
  ],
  "task_format": "Given a JSON Schema shown as raw text and a two-shot prompt demonstrating the expected input/output format with schemas unrelated to the one being tested, the model generates free text until a blank line; the harness strips code-fence and language-tag markers before scoring. The easy tier expects roughly a 2K-token context window, medium roughly 3K, and hard roughly 10K, since harder schemas are themselves larger and more deeply nested.\n"
 }
}