OpenCompass local SciReasoner 1.5 slice: OQMD and JARVIS-DFT material regression plus GO biological-process, TM-score, and DUD-E pair tasks.
unassessed
| Category | domain |
|---|---|
| Subcategory | local OpenCompass SciReasoner 1.5 tests: materials regression, GO-BP, TM-score, DUD-E pairs |
| Page status | active |
| Metric | task-dependent (MAE/RMSE/Pearson/Spearman for materials and TM-score; F1 for GO-BP; AUC for DUD-E) |
| Direction | higher_is_better |
| Publisher | Open Science Lab (SciReason); OpenCompass integration |
scireasoner1_5 is the OpenCompass config for a local SciReasoner 1.5 test directory, not the 103-task [scireasoner](scireasoner.md) suite. The loader reads user-provided files under `opencompass/SciReasoner1.5`. Material tasks ask for a numeric property of a structure from OQMD or JARVIS-DFT. Other tasks ask for Gene Ontology biological-process terms, a TM-score in [0, 1], or a DUD-E-like pair score. English prompts. Mini runs subsample 150 items per abbr with seed 1024.
Zero-shot GenInferencer with RawPromptTemplate (user role, `{prompt}`). GO-BP, TM-score, and DUD-E append a short output instruction (semicolon GO terms, one float, or one score in [0, 1]). Material tasks use SciReasoner15MaterialEvaluator; others use GO, TM-score, or DUD-E evaluators. Reasoning in `<think>` tags is stripped before parse.
No model card in ModelSpec reports this benchmark yet.