{
 "body": "\n## What it measures\n\nSciReasoner 1.5 in OpenCompass is a local test pack. The model reads a prompt and returns either a material property number, Gene Ontology biological-process terms, a TM-score, or a DUD-E-style pair score.\n\nMaterial abbrs cover OQMD band gap and formation energy plus many JARVIS-DFT properties (gaps, moduli, Seebeck, piezo, and others). GO-BP is multi-label term prediction. TM-score is a scalar similarity. DUD-E-count is a pair classification/score. This is not the full [scireasoner](scireasoner.md) concatenation.\n\n## How it is scored\n\nEach abbr has its own evaluator in `SciReasoner1_5.py`. Materials and TM-score: parse a float (JSON property or first number) and report MAE, RMSE, correlations, and valid_rate. GO-BP: split labels on semicolons or commas and mean F1 (`score` is F1\u00d7100). DUD-E: parse a 0/1 or [0,1] score, then ROC-AUC. `<think>` blocks are stripped. Mini_set draws 150 rows with seed 1024. MAE/RMSE are lower-is-better even when other columns are not.\n\n## Dataset and licence\n\nThe loader expects five files in `opencompass/SciReasoner1.5`. Those files are not in the OpenCompass git tree, so row counts are unknown. Twenty-five full abbrs plus twenty-five mini twins. OpenCompass code is Apache-2.0. Licence of the local JSON/JSONL was not published in the files opened here. OpenCompass still cites arXiv 2509.21320 for this id.\n\n## Who publishes it\n\nOpenCompass maintainers added `configs/datasets/scireasoner1_5`. The cited paper is the 2025 SciReasoner report (Yizhou Wang, Chen Tang, Lei Bai, and co-authors). The 1.5 local files themselves have no separate paper in dataset-index.yml.\n\n## Lineage\n\nPredecessor is [scireasoner](scireasoner.md), the broader OpenCompass suite. This id is not an alias of that suite: different files, metrics, and abbrs. A 2026 arXiv paper (2607.07708) also names its model SciReasoner; OpenCompass does not cite that paper here.\n\n## Saturation and contamination\n\nNo leaderboard for these abbrs was opened. OQMD, JARVIS-DFT, GO, TM-score, and DUD-E labels have been public for years, so a model that trained on those databases can look strong without solving a new exam. Mini 150-item draws are deterministic.\n\n## How to run it\n\nPlace the five test files, then import `scireasoner1_5_datasets` or `mini_scireasoner1_5_datasets` from `opencompass.configs.datasets.scireasoner1_5.scireasoner1_5_gen`. Zero-shot generation. Compare MAE only to MAE, F1 only to F1. State mini_set if you used it.\n\n## Reading the numbers\n\nA low MAE on JARVIS-DFT is a regression result on that property, not a general materials-science grade. GO F1 can be high if the model dumps many terms. Do not average these abbrs with [scireasoner](scireasoner.md) SMILES-match cells. If the local files differ across machines, the scores are not comparable.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "scireasoner1_5",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SciReasoner1.5",
   "SciReasoner1_5",
   "scireasoner1_5"
  ],
  "category": "domain",
  "contamination": {
   "note": "OQMD, JARVIS-DFT, Gene Ontology, TM-score, and DUD-E are long-public scientific resources. The OpenCompass JSON wrappers are local; if they are later uploaded, labels would be public. Mini_set uses a fixed seed, so the 150-item draws are reproducible and leakable.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "Files expected: oqmd_test.json, jarvis_dft_test.json, go_test_bp.json, tmscore_test.json, dude_count.jsonl. Twenty-two material (name, property) pairs plus GO-BP, TMScore, and DUDE-count (25 OpenCompass abbrs, each with a -mini twin). Row counts were not readable because the files are local, not in the git tree. Mini sample_size is 150.\n",
   "splits": "local test files; optional mini_set of 150 per abbr (seed 1024)",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/scireasoner1_5"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-070 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-070"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "scireasoner1_5",
   "other": "dataset-index.yml name SciReasoner1.5, paper 2509.21320; lists scireasoner1_5_datasets and mini_scireasoner1_5_datasets. Abbrs SciReasoner1_5-{task} and SciReasoner1_5-{task}-mini."
  },
  "id": "scireasoner1_5",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "scireasoner",
   "successors": [],
   "variants": []
  },
  "measures": "scireasoner1_5 is the OpenCompass config for a local SciReasoner 1.5 test directory, not the 103-task [scireasoner](scireasoner.md) suite. The loader reads user-provided files under `opencompass/SciReasoner1.5`. Material tasks ask for a numeric property of a structure from OQMD or JARVIS-DFT. Other tasks ask for Gene Ontology biological-process terms, a TM-score in [0, 1], or a DUD-E-like pair score. English prompts. Mini runs subsample 150 items per abbr with seed 1024.\n",
  "metric": {
   "baseline_note": "Material and TM-score evaluators report MAE, RMSE, MAD, MAD/MAE, Pearson, Spearman, and valid_rate. GO-BP reports Precision, Recall, F1 (also copied to `score` as F1\u00d7100), and exact_match. DUD-E uses a binary score parse and ROC-AUC. Several of those columns are lower-is-better (MAE, RMSE). No human baseline is in the OpenCompass module. Mini_set is 150 rows, not the full local files.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "task-dependent (MAE/RMSE/Pearson/Spearman for materials and TM-score; F1 for GO-BP; AUC for DUD-E)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "SciReasoner 1.5 (OpenCompass)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2509.21320",
   "title": "SciReasoner: Laying the Scientific Reasoning Ground Across Disciplines",
   "url": "https://arxiv.org/abs/2509.21320",
   "year": 2025
  },
  "publisher": {
   "authors": [],
   "org": "Open Science Lab (SciReason); OpenCompass integration",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/scireasoner1_5"
  },
  "released": "2025-09",
  "repo_url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/scireasoner1_5",
  "saturation": {
   "as_of": "",
   "note": "No public score table for these 25 abbrs was opened. Paper 2509.21320 tables are for the broader suite, not this local file set.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "scireasoner1_5_gen.py (25 tasks, mini 150, evaluators, output instructions)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/scireasoner1_5/scireasoner1_5_gen.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "SciReasoner15Dataset and evaluators (local files, metrics, think-tag strip)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/SciReasoner1_5.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "dataset-index.yml scireasoner1_5 (SciReasoner1.5, paper 2509.21320)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/dataset-index.yml"
   },
   {
    "accessed": "2026-09-08",
    "title": "SciReasoner paper cited by OpenCompass for this config",
    "url": "https://arxiv.org/abs/2509.21320"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass scireasoner1_5 config directory",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/scireasoner1_5"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass Apache License 2.0",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "arXiv API listing 2509.21320 and later 2607.07708 namesake",
    "url": "https://export.arxiv.org/api/query?search_query=all:SciReasoner&start=0&max_results=15"
   }
  ],
  "status": "active",
  "subcategory": "local OpenCompass SciReasoner 1.5 tests: materials regression, GO-BP, TM-score, DUD-E pairs",
  "summary": "OpenCompass local SciReasoner 1.5 slice: OQMD and JARVIS-DFT material regression plus GO biological-process, TM-score, and DUD-E pair tasks.",
  "tags": [
   "science",
   "materials",
   "biology",
   "opencompass",
   "regression"
  ],
  "task_format": "Zero-shot GenInferencer with RawPromptTemplate (user role, `{prompt}`). GO-BP, TM-score, and DUD-E append a short output instruction (semicolon GO terms, one float, or one score in [0, 1]). Material tasks use SciReasoner15MaterialEvaluator; others use GO, TM-score, or DUD-E evaluators. Reasoning in `<think>` tags is stripped before parse.\n"
 }
}