JSONSchemaBench tests whether a model can generate JSON that both parses and validates against a supplied JSON Schema, drawn from real-world schemas across ten source domains.
unassessed
| Category | generation |
|---|---|
| Subcategory | constrained structured-output generation: producing JSON that validates against a supplied JSON Schema |
| Page status | active |
| Metric | json_validity and schema_compliance, both binary per-sample metrics averaged across each tier |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1531 |
| Dataset licence | MIT |
| Publisher | Hosted under the epfl-dlab (EPFL Data Science Lab) organisation on GitHub and Hugging Face; individual authors' institutional affiliations were not confirmed from a source read for this page |
jsonschema_bench is a constrained-generation task, not a knowledge test: given a real-world JSON Schema, the model must produce a JSON object that satisfies it. The underlying JSONSchemaBench project collects roughly 10,000 real schemas across ten domains -- five GitHub difficulty tiers (trivial through ultra) plus Kubernetes configuration schemas, GlaiveAI function-call schemas, the JSON Schema Store, Snowplow event schemas and Washington Post resource schemas -- built to evaluate both constrained-decoding frameworks (the paper tests Guidance, Outlines, Llamacpp, XGrammar, OpenAI and Gemini) and a model's native, unconstrained ability to produce schema-conformant output. lm-evaluation-harness implements three of those ten domains: the Github_easy, Github_medium and Github_hard difficulty tiers, grouped under the shared tag `jsonschema_bench`.
Given a JSON Schema shown as raw text and a two-shot prompt demonstrating the expected input/output format with schemas unrelated to the one being tested, the model generates free text until a blank line; the harness strips code-fence and language-tag markers before scoring. The easy tier expects roughly a 2K-token context window, medium roughly 3K, and hard roughly 10K, since harder schemas are themselves larger and more deeply nested.
No model card in ModelSpec reports this benchmark yet.