{
 "body": "\n## What it measures\n\nOpenCompass mp20 is a text-to-crystal task. The model is given a system prompt and a user prompt and must emit lattice constants plus atomic sites. The scorer rebuilds a crystal and asks whether it matches a stored ground-truth cell with the same elements. The crystals come from MP-20: Materials Project structures with at most 20 atoms, filtered to ICSD-origin cells that sit close to the hull (CDVAE, arXiv:2110.06197).\n\nThis is reconstruction against a reference, not a band-gap quiz. It is not [matbench](matbench.md), which predicts properties from composition. It is not a graph generator such as CDVAE or DiffCSP running on CIF files.\n\n## How it is scored\n\n`MP20Evaluator` parses JSON from `<material>` tags, a ```json fence, or the last large brace block. It converts lattice a, b, c, alpha, beta, gamma into a matrix and reads `atomic_sites`. pymatgen `StructureMatcher` (ltol 0.3, stol 0.5, angle 10\u00b0) returns an RMS or nothing. `match_rate` is the percent of items with a numeric RMS. `mean_rms_dist` averages those RMS values only. A parse failure counts as unmatched. Those two numbers move in opposite directions; do not quote RMS as if higher were better. No random or human baseline is in the OpenCompass files.\n\n## Dataset and licence\n\nXie et al. state MP-20 has 45,231 crystals, 89 elements, and a 60/20/20 split. The CDVAE dump is MIT-licensed code plus `train.csv` / `val.csv` / `test.csv`. OpenCompass does not read those CSVs. It reads a JSONL named `opencompass/mp20` with `id_ddm`, `system_prompt`, `user_input`, and `ground_truth`. Hugging Face returned 401 for that dataset in this session, so the prompt-file size and its licence are not established. Do not treat 45,231 as the OpenCompass item count. Ground-truth coordinates are in the JSONL, so the test answers are not held out from someone who has the file.\n\n## Who publishes it\n\nThe LLM wrap is OpenCompass, added 2026-06-24 and patched 2026-06-25 (`#2482`, `#2488`). MP-20 itself was curated by Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola from the Materials Project (Jain et al., 2013) and ICSD. There is no separate OpenCompass mp20 leaderboard in the files opened here.\n\n## Lineage\n\nCDVAE (2021) introduced MP-20 as a generative-materials split. Later graph and diffusion papers reuse that split; those scores are not this page. OpenCompass turns the reconstruction idea into chat JSON. [matbench](matbench.md) is a different Materials Project-derived suite for property prediction. No `diffcsp` or `mattergen` page is in this repository.\n\n## Saturation and contamination\n\nNo LLM match_rate table was opened, so saturation is unknown. MP-20 has been a public training corpus for crystal generators since 2021. An LLM that memorised CIFs can still fail the JSON format the matcher needs. Failed parses look like zeros on match_rate, which can hide a model that almost got the geometry.\n\n## How to run it\n\nIn OpenCompass, import `mp20_datasets` from `opencompass.configs.datasets.mp20.mp20_gen` (that file only re-exports `mp20_rawprompt_gen.py`). Install pymatgen. smact is imported only inside optional validity helpers, not the headline matcher. Do not compare an OpenCompass match_rate to a CDVAE coverage or RMS number from 2021 without checking that both used StructureMatcher on the same split.\n\n## Reading the numbers\n\nA high match_rate means the parsed cells matched the stored references often enough to return an RMS. It does not mean the model invented a new stable material. mean_rms_dist is only defined on matches, so a tiny matched set can look accurate. Report both, plus how many rows ran. Check whether the reporter used the OpenCompass JSON wrap or a CIF-based CDVAE script before lining the number up with a materials paper.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mp20",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "MP-20",
   "MP20",
   "opencompass/mp20"
  ],
  "category": "domain",
  "contamination": {
   "note": "MP-20 crystals have been public in txie-93/cdvae since 2021 and are a standard train set for later generators. Ground-truth lattices sit in the OpenCompass JSONL. Hugging Face opencompass/mp20 was not readable here (401), so prompt leakage for the LLM wrap is plausible but not measured.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "CDVAE MP-20 is 45,231 inorganic crystals, 89 elements, 1\u201320 atoms per cell, 60/20/20 random split (arXiv:2110.06197; txie-93/cdvae data/mp_20). OpenCompass loads a JSONL of system_prompt, user_input, and ground_truth (lattice plus sites). Hugging Face datasets/opencompass/mp20 returned HTTP 401 on 2026-09-08, so the JSONL row count is not established and is not assumed to be 45,231.\n",
   "splits": "OpenCompass JSONL split not established; CDVAE dump has train.csv, val.csv, test.csv",
   "url": "https://github.com/txie-93/cdvae/tree/main/data/mp_20"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-060 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-060"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "mp20",
   "other": "OpenCompass configs mp20_rawprompt_gen.py (re-exported by mp20_gen.py). Needs pymatgen and optionally smact."
  },
  "id": "mp20",
  "last_updated": "2026-06",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "OpenCompass mp20 asks a language model to write a crystal as JSON: lattice parameters and atomic-site coordinates. The evaluator parses that JSON, builds a pymatgen Structure, and matches it to a ground-truth structure with the same element list. The underlying crystals are MP-20, the Materials Project slice with at most 20 atoms per cell, curated for CDVAE. This is structure reconstruction as text, not [matbench](matbench.md) property prediction, and not unconstrained de novo generation without a reference cell.\n",
  "metric": {
   "baseline_note": "match_rate is 100 times the fraction of items where StructureMatcher.get_rms_dist returns a number (ltol 0.3, stol 0.5, angle_tol 10, 10-second worker timeout). mean_rms_dist is the mean of those RMS values and is lower-is-better; failed parses score as unmatched, not as a large RMS. No OpenCompass leaderboard cell was read. CDVAE paper metrics on the same crystals are a different stack (graph models, not LLM JSON).\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "match_rate",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "MP-20 (OpenCompass)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2110.06197",
   "title": "Crystal Diffusion Variational Autoencoder for Periodic Material Generation",
   "url": "https://arxiv.org/abs/2110.06197",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Tian Xie",
    "Xiang Fu",
    "Octavian-Eugen Ganea",
    "Regina Barzilay",
    "Tommi Jaakkola"
   ],
   "org": "OpenCompass (LLM wrap); MP-20 curated by Xie et al. from the Materials Project",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/mp20"
  },
  "released": "2026-06",
  "repo_url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/mp20",
  "saturation": {
   "as_of": "",
   "note": "No OpenCompass mp20 score table was opened. CDVAE-era graph-model numbers on MP-20 are not this JSON generation protocol.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass mp20_rawprompt_gen.py (abbr mp20, path opencompass/mp20, MP20Evaluator)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/mp20/mp20_rawprompt_gen.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "mp20_gen.py re-exports mp20_rawprompt_gen",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/mp20/mp20_gen.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "MP20Dataset and MP20Evaluator (JSON parse, StructureMatcher, match_rate)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/mp20/mp20.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass mp20.py commits (added 2026-06-24, metrics fix 2026-06-25)",
    "url": "https://api.github.com/repos/open-compass/opencompass/commits?path=opencompass/datasets/mp20/mp20.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "CDVAE paper (arXiv:2110.06197; MP-20 45,231 crystals, 60/20/20 split)",
    "url": "https://arxiv.org/abs/2110.06197"
   },
   {
    "accessed": "2026-09-08",
    "title": "CDVAE HTML (MP-20 definition, 89 elements, ICSD filter, hull cutoff)",
    "url": "https://ar5iv.labs.arxiv.org/html/2110.06197"
   },
   {
    "accessed": "2026-09-08",
    "title": "CDVAE data/mp_20 README (45,231 synthesizable crystals)",
    "url": "https://raw.githubusercontent.com/txie-93/cdvae/main/data/mp_20/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "CDVAE MIT License (copyright 2021 Tian Xie, Xiang Fu)",
    "url": "https://raw.githubusercontent.com/txie-93/cdvae/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass Apache License 2.0",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/LICENSE"
   }
  ],
  "status": "unknown",
  "subcategory": "OpenCompass LLM crystal-structure reconstruction on MP-20, scored with StructureMatcher",
  "summary": "OpenCompass LLM wrap of MP-20: emit lattice and atomic sites as JSON, scored by StructureMatcher match rate and RMS.",
  "tags": [
   "materials",
   "crystal-structure",
   "opencompass",
   "reconstruction"
  ],
  "task_format": "Zero-shot GenInferencer with RawPromptTemplate. Messages are a system role from {system_prompt} and a user role from {user_input}. ZeroRetriever. abbr mp20. Dataset class MP20Dataset reads a JSONL via get_data_path('opencompass/mp20'). Evaluator MP20Evaluator, pred_role BOT. JSON may be wrapped in <material> tags, a markdown fence, or a trailing large JSON object.\n"
 }
}