944 manually checked multiple-choice questions on proteins, used to score LLMs on sequence and function understanding rather than to train them.
unassessed
| Category | domain |
|---|---|
| Subcategory | multiple-choice protein sequence and function understanding |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 944 |
| Dataset licence | Apache-2.0 |
| Publisher | Toursun Synbio, with Johns Hopkins University, University of Cambridge, Shanghai AI Laboratory, Shanghai Institute for Biomedical and Pharmaceutical Technologies, and Shanghai Jiao Tong University |
ProteinLMBench is a 944-item multiple-choice test of whether a language model understands proteins from sequence and text. Questions mix English protein facts with sequence-bearing items. They were drafted with retrieval-augmented generation and GPT-4, then passed through a two-round machine check and human verification; inconsistent items were dropped. The same paper also releases ProteinLMDataset (pretraining tokens and SFT instructions). That training set is not this benchmark. OpenCompass loads Hugging Face config `evaluation` only.
Six-choice multiple choice. The paper states 944 questions with six lettered options and an explanation of the correct answer. OpenCompass's default generator prompt asks for a final line `Answer: $LETTER` between the first and last option letters (the loader is generic over option count). A second OpenCompass config swaps in an LLM judge (GenericLLMEvaluator) instead of rule extraction.
No model card in ModelSpec reports this benchmark yet.