An OpenCompass-maintained, free-response QA benchmark graded like OpenAI's SimpleQA and refreshed through periodic dated dataset versions; its underlying question set is not publicly downloadable.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | free-response general QA/reasoning, periodically re-versioned, SimpleQA-style grading |
| Page status | active |
| Metric | accuracy_given_attempted (precision on attempted answers) and F1, following SimpleQA's own metric design |
| Direction | higher_is_better |
| Unit | % |
| Publisher | Shanghai Artificial Intelligence Laboratory (OpenCompass project) |
LiveReasonBench poses a single free-response question and compares the model's answer against a gold target, with no answer options offered. Its harness code is adapted directly from OpenAI's SimpleQA grading rubric, down to reusing SimpleQA's own worked examples (the Barack Obama's children example, the Jason Wei height example) inside its LLM-judge prompt. What exact topic or difficulty range the questions cover could not be independently confirmed for this page: the dataset (nominally hosted at `opencompass/LiveReasonBench` on Hugging Face) returns an authentication error to an anonymous request rather than a public dataset card, so its content is not publicly inspectable even though the evaluation code that runs against it is. The name and the question/gold-target/free-response structure suggest general reasoning or knowledge QA rather than a narrow task, but this is inferred from the harness mechanism, not confirmed from example items.
Single-turn free-response question answering: the model receives only "Question: {question}" with no options, and its answer is graded by a separate LLM-judge call comparing it against a gold target, following the CORRECT / INCORRECT / NOT_ATTEMPTED rubric from OpenAI's SimpleQA.
No model card in ModelSpec reports this benchmark yet.