{
 "body": "\n## What it measures\n\nMatbench asks a model to predict a physical property of an inorganic bulk material -- a formation\nenergy, a band gap, an elastic modulus, whether a composition is a metal, whether it forms a glass,\nor the yield strength of a steel -- from either its chemical composition alone or its full crystal\nstructure. It was built for classical and graph-based machine learning models, not language models:\n9 of its 13 tasks require a crystal structure as input, which has no natural plain-text encoding.\nOpenCompass's LLM evaluation harness works around this by running only the 4 tasks that take a\ncomposition string alone (matbench_steels, matbench_expt_gap, matbench_expt_is_metal,\nmatbench_glass) as free-text prompts, so an LLM's \"Matbench\" score, where one exists at all, covers\nunder a third of the full suite and none of its structure-dependent tasks.\n\n## How it is scored\n\nMetrics are task-specific and not uniform: the two composition-only regression tasks\n(matbench_steels, predicting yield strength in MPa; matbench_expt_gap, predicting an experimental\nband gap in eV) are scored by mean absolute error, while the two composition-only classification\ntasks (matbench_expt_is_metal, matbench_glass) are scored by accuracy, precision, recall and F1. The\nmaintainers publish each regression task's mean absolute deviation as a naive baseline -- 229.4 MPa\nfor matbench_steels, 1.143 eV for matbench_expt_gap -- and each classification task's positive-class\nrate as a base rate (49.8% and 71.0% respectively). OpenCompass ships two different ways to grade an\nLLM's free-text answer for the same 4 tasks: a regex-based extractor, or a second LLM call acting as\njudge, so two \"Matbench\" numbers are not comparable unless both used the same grading path.\n\n## Dataset and licence\n\nThe full suite is 13 tasks ranging from 312 to 132,752 samples, drawn from 10 density-functional-theory\nand experimental sources. Every task is organised as 5-fold cross-validation (fold_0 through\nfold_4), confirmed from the maintainers' own validation-split file, rather than a single train/test\npartition. OpenCompass's LLM harness reads only fold_0's test split for each of its 4 tasks, not the\n5-fold mean the official leaderboard reports -- a material difference in protocol, not just in task\ncount. The reference repository (materialsproject/matbench) is MIT licensed; composition, structure\nand target values are public for all tasks.\n\n## Who publishes it\n\nMatbench comes from Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp and Anubhav Jain at the\nHacking Materials Research Group, part of the Materials Project at Lawrence Berkeley National\nLaboratory, published in npj Computational Materials in 2020. The group maintains the `matbench`\nPython package, the reference `Automatminer` algorithm, and the public leaderboard at\nmatbench.materialsproject.org, which tracks classical and graph-based ML submissions, not LLMs.\n\n## Lineage\n\nMatbench has no predecessor or successor tracked in this repository; it predates the LLM benchmark\nera by several years and was not originally designed as one. Its relevance to LLM evaluation is\nentirely downstream, through OpenCompass's own text-prompt adaptation of 4 of its 13 tasks -- a\nrepurposing, not an official variant, and not maintained by Matbench's own authors. No other\nharness's adaptation of Matbench for language models was found in sources opened for this page.\n\n## Saturation and contamination\n\nThe official leaderboard ranks classical and graph-neural-network algorithms against each other, not\nlanguage models, so it gives no signal about LLM saturation. No source opened for this page reported\ncurrent LLM scores on OpenCompass's 4-task adaptation, so an LLM-specific saturation call is not\nestablished here. Contamination risk sits at medium: composition, structure and property data have\nbeen public since 2020 and much of it overlaps with the broader Materials Project database that\ncirculates widely in materials-science literature and code, making memorisation of common compounds'\nproperties plausible, though this looks more like general domain-knowledge overlap than benchmark-\nspecific leakage. No contamination study targeting LLM use of Matbench was found.\n\n## How to run it\n\nThe original 13-task, 5-fold protocol runs through the pip-installable `matbench` package\n(`pip install matbench`), using the `MatbenchBenchmark` class to iterate folds and record\npredictions per task. For LLMs, OpenCompass registers the 4 composition-only tasks under the\n`matbench` config directory, with both a regex-based evaluator (`MatbenchEvaluator_regression`,\n`MatbenchEvaluator_classification`) and an LLM-judge evaluator (`GenericLLMEvaluator` with separate\ncorrectness and numeric-extraction prompt templates) available for the same 4 tasks. A score is only\ncomparable to another if both used the same task subset, the same fold, and the same grading path.\n\n## Reading the numbers\n\nBecause OpenCompass's LLM adaptation covers only 4 of 13 tasks, all composition-only, a \"Matbench\"\nfigure attached to a language model says nothing about the 9 structure-dependent tasks that make up\nmost of the original suite. Because it reads a single fold rather than the 5-fold mean the official\nclassical-ML leaderboard reports, it is not directly comparable to a matbench.materialsproject.org\nranking even on the same 4 tasks. Treat any LLM \"Matbench\" score as a narrow, non-standard read on a\nbenchmark built for a different kind of model, and check which grading path (regex or LLM judge)\nproduced it before comparing two such scores.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "matbench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "domain",
  "contamination": {
   "note": "Matbench's compositions, structures, property values and fold assignments have all been public since 2020, several years before the LLM benchmark era, and much of the underlying property data overlaps with the broader Materials Project database that circulates widely in the materials science literature. That makes exact memorisation of specific numeric answers plausible for very common compounds, though this is closer to general domain-knowledge overlap than to the kind of targeted benchmark leakage seen in exam-style datasets. No contamination study specific to LLM use of Matbench was found.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text",
    "tabular"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "The full suite spans 13 tasks ranging from 312 to 132,752 samples each (summed from the maintainers' own matbench_v0.1_dataset_metadata.json): 9 take a crystal structure as input (636-132,752 samples) and 4 take only a composition string (matbench_steels 312, matbench_expt_gap 4,604, matbench_expt_is_metal 4,921, matbench_glass 5,680). Those 4 composition-only tasks are the ones OpenCompass's LLM harness runs. Every task is organised as 5-fold cross-validation (fold_0 through fold_4, confirmed from the maintainers' matbench_v0.1_validation.json); OpenCompass's loader reads only fold_0's test split per task, not the 5-fold mean the official leaderboard reports.\n",
   "splits": "5-fold cross-validation per task (fold_0-fold_4); OpenCompass's LLM harness uses only fold_0's test split",
   "url": "https://github.com/materialsproject/matbench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "matbench (4 tasks: matbench_steels, matbench_expt_gap, matbench_expt_is_metal, matbench_glass; both a regex/rule-based eval_cfg and an LLM-judge eval_cfg exist for the same tasks)",
   "other": "The reference implementation for the original suite is the pip-installable `matbench` Python package (materialsproject/matbench), which runs the full 5-fold protocol against classical ML pipelines via the MatbenchBenchmark class. No LLM-oriented reference harness was found outside OpenCompass's adaptation.\n"
  },
  "id": "matbench",
  "last_updated": "",
  "leaderboard_url": "https://matbench.materialsproject.org",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Matbench asks a model to predict a physical property of an inorganic bulk material -- for example a formation energy, a band gap, an elastic modulus, whether a composition is a metal, whether a composition forms a glass, or the yield strength of a steel -- given either the material's chemical composition alone or its full crystal structure. It was built for classical and graph-based machine learning models (random forests, descriptor-based pipelines, crystal graph neural networks), not for language models: 9 of its 13 tasks require a crystal structure as input, which has no natural plain-text encoding. OpenCompass's LLM evaluation harness sidesteps this by running only the 4 tasks that take a composition string alone -- matbench_steels, matbench_expt_gap, matbench_expt_is_metal and matbench_glass -- as free-text prompts, which is why an LLM's \"Matbench\" score, where one exists, covers less than a third of the full suite.\n",
  "metric": {
   "baseline_note": "Metrics are not uniform across the 4 LLM-relevant tasks, so no single direction or unit applies: MAE is lower-is-better while accuracy/F1 are higher-is-better. The maintainers publish each regression task's mean absolute deviation (MAD) as a naive \"always predict the training mean\" baseline -- 229.4 MPa for matbench_steels, 1.143 eV for matbench_expt_gap -- and each classification task's positive-class fraction as a base rate: 49.8% for matbench_expt_is_metal, 71.0% for matbench_glass. No baseline specific to LLM text-prompt evaluation was found.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "task-specific: mean absolute error (MAE) for the regression tasks (matbench_steels, matbench_expt_gap); accuracy, precision, recall and F1 for the classification tasks (matbench_expt_is_metal, matbench_glass). The full 13-task suite also reports RMSE, MAPE, max error, balanced accuracy and ROC-AUC depending on task.",
   "random_baseline": null,
   "unit": ""
  },
  "name": "Matbench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2005.00707",
   "title": "Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm",
   "url": "https://arxiv.org/abs/2005.00707",
   "year": 2020
  },
  "publisher": {
   "authors": [
    "Alexander Dunn",
    "Qi Wang",
    "Alex Ganose",
    "Daniel Dopp",
    "Anubhav Jain"
   ],
   "org": "Hacking Materials Research Group (Lawrence Berkeley National Laboratory); part of the Materials Project",
   "url": "https://matbench.materialsproject.org"
  },
  "released": "2020-05",
  "repo_url": "https://github.com/materialsproject/matbench",
  "saturation": {
   "as_of": "",
   "note": "The official matbench.materialsproject.org leaderboard tracks classical and graph-based ML algorithms (Automatminer, CGCNN, MODNet, coNGN and similar), not language models, so it says nothing about LLM saturation. No source opened for this page reported current LLM scores on OpenCompass's 4-task text adaptation, so a saturation call for LLMs specifically is not established here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm",
    "url": "https://arxiv.org/abs/2005.00707"
   },
   {
    "accessed": "2026-09-08",
    "title": "materialsproject/matbench repository (README, LICENSE)",
    "url": "https://github.com/materialsproject/matbench"
   },
   {
    "accessed": "2026-09-08",
    "title": "matbench_v0.1_dataset_metadata.json (per-task size, input type, MAD, target, unit)",
    "url": "https://raw.githubusercontent.com/materialsproject/matbench/main/matbench/matbench_v0.1_dataset_metadata.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "matbench_v0.1_validation.json (5-fold split assignments per task)",
    "url": "https://raw.githubusercontent.com/materialsproject/matbench/main/matbench/matbench_v0.1_validation.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "matbench/constants.py (package version 0.6, regression/classification metric lists)",
    "url": "https://raw.githubusercontent.com/materialsproject/matbench/main/matbench/constants.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "Matbench leaderboard and documentation site",
    "url": "https://matbench.materialsproject.org"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass matbench_llm_judge_gen_0e9276.py (4-task LLM adaptation, judge templates)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/matbench/matbench_llm_judge_gen_0e9276.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass MatbenchDataset loader and regex-based evaluators",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/matbench/matbench.py"
   }
  ],
  "status": "active",
  "subcategory": "materials property prediction from composition and/or crystal structure",
  "summary": "A 13-task materials-property-prediction suite built for classical ML; OpenCompass's LLM harness adapts its 4 composition-only tasks into text prompts graded by regex or an LLM judge.",
  "tags": [
   "domain",
   "materials-science",
   "regression",
   "classification",
   "tabular",
   "cross-validation"
  ],
  "task_format": "Originally a structured regression or classification problem over composition/structure features. OpenCompass's LLM adaptation turns each of the 4 composition-only tasks into a free-text prompt asking for a predicted number (matbench_steels, matbench_expt_gap) or a yes/no classification (matbench_expt_is_metal, matbench_glass), extracted from the model's response either by regex or by a separate LLM-judge call.\n"
 }
}