A 13-task materials-property-prediction suite built for classical ML; OpenCompass's LLM harness adapts its 4 composition-only tasks into text prompts graded by regex or an LLM judge.
unassessed
| Category | domain |
|---|---|
| Subcategory | materials property prediction from composition and/or crystal structure |
| Page status | active |
| Metric | task-specific: mean absolute error (MAE) for the regression tasks (matbench_steels, matbench_expt_gap); accuracy, precision, recall and F1 for the classification tasks (matbench_expt_is_metal, matbench_glass). The full 13-task suite also reports RMSE, MAPE, max error, balanced accuracy and ROC-AUC depending on task. |
| Direction | higher_is_better |
| Dataset licence | MIT |
| Publisher | Hacking Materials Research Group (Lawrence Berkeley National Laboratory); part of the Materials Project |
Matbench asks a model to predict a physical property of an inorganic bulk material -- for example a formation energy, a band gap, an elastic modulus, whether a composition is a metal, whether a composition forms a glass, or the yield strength of a steel -- given either the material's chemical composition alone or its full crystal structure. It was built for classical and graph-based machine learning models (random forests, descriptor-based pipelines, crystal graph neural networks), not for language models: 9 of its 13 tasks require a crystal structure as input, which has no natural plain-text encoding. OpenCompass's LLM evaluation harness sidesteps this by running only the 4 tasks that take a composition string alone -- matbench_steels, matbench_expt_gap, matbench_expt_is_metal and matbench_glass -- as free-text prompts, which is why an LLM's "Matbench" score, where one exists, covers less than a third of the full suite.
Originally a structured regression or classification problem over composition/structure features. OpenCompass's LLM adaptation turns each of the 4 composition-only tasks into a free-text prompt asking for a predicted number (matbench_steels, matbench_expt_gap) or a yes/no classification (matbench_expt_is_metal, matbench_glass), extracted from the model's response either by regex or by a separate LLM-judge call.
No model card in ModelSpec reports this benchmark yet.