MuSR

Algorithmically generated murder mysteries, object placement puzzles and team allocation problems needing long-range narrative reasoning.

Also known as: Multistep Soft Reasoning

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryreasoning
Subcategorymultistep narrative and long-context reasoning
Page statusactive
Metricaccuracy (normalized)
Directionhigher_is_better
Unit%
Dataset size756
Dataset licenceMIT (authors' GitHub repository); a Hugging Face mirror separately labels it CC BY 4.0
PublisherUniversity of Texas at Austin

What it measures

MuSR (Multistep Soft Reasoning) tests whether a model can answer a question that requires combining facts scattered across a long narrative, rather than reasoning that is stated in one place. Each item is an algorithmically generated story of roughly 1,000 words in one of three domains: a murder mystery (identify the culprit), an object placement puzzle (determine where an object ended up after being moved), or a team allocation problem (assign people to tasks under stated constraints). The stories are produced by a "neurosymbolic synthetic-to-natural generation algorithm," which builds a structured reasoning tree first and then renders it into free-form prose, so the correct answer depends on connecting details spread through the narrative rather than pattern-matching a single sentence. It is a single-turn, English-language, text-only task.

Task format

A narrative of roughly 1,000 words followed by a multiple-choice question; the model must integrate facts scattered through the text to select the correct option.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
Falcon3 7B InstructTII48.32025-03
Falcon3 7B BaseTII47.02025-03
deepseek llm 7B chatDeepSeek46.72024-07
Nous Hermes 2 Mixtral 8x7B DPONous Research46.02024-07
Yi 6B 200K01.AI45.92024-07
Infinity Instruct 3M 0625 Yi 1.5 9BBAAI45.82025-03
Qwen2.5 Coder 32B Instruct AWQAlibaba / Qwen Team45.32026-04
Meta Llama 3 70BMeta45.22024-07
OpenHermes 2 Mistral 7BTeknium45.22025-03
Yi 1.5 34B 32K01.AI44.02024-07
Yi 1.5 34B Chat 16K01.AI44.02024-07
glm 4 9BZhipu AI43.92025-03
GLM 4 9B 0414Zhipu AI43.92025-03
Qwen2.5 Coder 32B InstructAlibaba / Qwen Team43.92026-04
Yi 1.5 6B Chat01.AI43.92024-07
Qwen2 VL 7B InstructAlibaba / Qwen Team43.82025-03
Qwen2 VL 7B Instruct AWQAlibaba / Qwen Team43.82025-03
Hermes 3 Llama 3.1 8BNous Research43.72025-03
Hermes 3 Llama 3.1 8B GGUFNous Research43.72025-03
Nous Hermes 2 SOLAR 10.7BNous Research43.72024-07
Yi 1.5 6B01.AI43.72024-07
Yi 1.5 9B01.AI43.32024-07
DialoGPT mediumMicrosoft42.92025-03
Qwen2 1.5B InstructAlibaba / Qwen Team42.92024-07
Yi 9B 200K01.AI42.92025-03
Phi 3 mini 4K instructMicrosoft42.82024-07
gemma 7B itGoogle DeepMind42.72024-07
Hermes 2 Pro Llama 3 8BNous Research42.62024-07
Nous Hermes llama 2 7BNous Research42.62024-07
Yi 1.5 9B Chat01.AI42.62024-07
gemma 2 2B itGoogle DeepMind42.42025-03
OpenHermes 2.5 Mistral 7BTeknium42.42025-03
flan t5 xlGoogle DeepMind42.22025-03
gemma 2 2BGoogle DeepMind42.22025-03
falcon mamba 7BTII42.12025-03
falcon mamba 7B instructTII42.12025-03
falcon mamba 7B instruct Q4 K M GGUFTII42.12025-03
Falcon3 1B InstructTII41.92025-03
Yi 1.5 9B 32K01.AI41.92024-07
stablelm zephyr 3BStability AI41.82025-03
Falcon3 1B BaseTII41.52025-03
Meta Llama 3 70B InstructMeta41.52026-04
Meta Llama 3 70B InstructNous Research41.52026-04
Falcon3 3B InstructTII41.42025-03
Ministral 8B Instruct 2410Mistral AI41.42025-03
Mistral 7B v0.1Mistral AI41.42024-07
flan t5 smallGoogle DeepMind41.22025-03
Llama 2 70B hfMeta41.22024-07
Yi 34B01.AI41.22024-07
OLMo 1B hfAllen AI41.02024-07
phi 2Microsoft41.02024-07
Qwen2.5 Coder 7B InstructAlibaba / Qwen Team41.02026-04
Yi 1.5 9B Chat 16K01.AI41.02024-07
flan t5 largeGoogle DeepMind40.82025-03
Llama 2 7B 32K InstructTogether AI40.62025-03
Yi 9B01.AI40.52024-07
OpenHermes 13BTeknium40.42025-03
Hermes 3 Llama 3.2 3BNous Research40.32025-03
Mistral 7B v0.3Mistral AI40.32024-07
mistral 7B v0.3 bnb 4bitUnsloth40.32024-07
Llama 2 13B chat hfMeta40.12024-07
falcon 11BTII39.92024-07
glm 4 9B chatZhipu AI39.92025-03
Yi Coder 9B01.AI39.92025-03
Yi Coder 9B Chat01.AI39.92025-03
gemma 2BGoogle DeepMind39.82024-07
Yi 34B Chat01.AI39.82024-07
Mistral 7B Instruct v0.2Mistral AI39.72024-07
Qwen2.5 3B InstructAlibaba / Qwen Team39.72025-03
Hermes 2 Theta Llama 3 8BNous Research39.52024-07
rwkv raven 14BRWKV Foundation39.52025-03
Phi 3 mini 128K instructMicrosoft39.42024-07
Yi 6B01.AI39.42024-07
Llama 3.1 8BCerebras39.32025-03
Qwen2.5 Coder 14B InstructAlibaba / Qwen Team39.12026-04
granite 3.0 8B instructIBM39.02025-03
stablelm 2 1 6BStability AI38.82025-03
Falcon3 Mamba 7B InstructTII38.72025-03
Phi 4 mini instructMicrosoft38.72026-04
mt5 smallGoogle DeepMind38.62025-03
Mistral 7B Instruct v0.1Mistral AI38.52024-07
Yi 34B 200K01.AI38.22024-07
Meta Llama 3 8B InstructMeta38.12024-07
Meta Llama 3 8B InstructNous Research38.12024-07
glm 4 9B chat 1MZhipu AI37.92025-03
falcon 7BTII37.82024-07
stablelm 3B 4e1tStability AI37.82025-03
falcon 40B instructTII37.62024-07
Falcon3 3B BaseTII37.52025-03
LLaMA 2 7B 32KTogether AI37.52025-03
deepseek llm 7B baseDeepSeek37.42024-07
GPT JT 6B v1Together AI37.42025-03
Mistral 7B Instruct v0.3Mistral AI37.42025-03
RedPajama INCITE Base 3B v1Together AI37.42025-03
Infinity Instruct 3M 0625 Llama3 8BBAAI37.12025-03
Llama 2 7B hfMeta37.02024-07
Llama 2 7B hfNous Research37.02024-07
Llama 2 70B chat hfMeta36.92024-07
Qwen2.5 Math 1.5BAlibaba / Qwen Team36.92025-03
RedPajama INCITE 7B InstructTogether AI36.92025-03
Yi 6B Chat01.AI36.92024-07
Yi 6B Chat 4bits01.AI36.92025-03
Llama 2 7B chat hfMeta36.82024-07
Llama 2 7B chat hfNous Research36.82024-07
RedPajama INCITE Chat 3B v1Together AI36.82025-03
flan t5 baseGoogle DeepMind36.72025-03
mt5 baseGoogle DeepMind36.72025-03
deepseek moe 16B baseDeepSeek36.62025-03
Qwen2.5 1.5B InstructAlibaba / Qwen Team36.62025-03
OLMoE 1B 7B 0125Allen AI36.42025-03
OLMoE 1B 7B 0125 InstructAllen AI36.42025-03
DeepSeek R1 Distill Qwen 1.5BDeepSeek36.32026-04
falcon 40BTII36.32024-07
falcon 7B instructTII36.32024-07
RedPajama INCITE 7B BaseTogether AI36.22025-03
granite 3.1 2B instructIBM36.12025-03
Meta Llama 3 8BMeta36.12024-07
Meta Llama 3 8BNous Research36.12024-07
glm 4 9B chat hfZhipu AI35.92025-03
Jamba v0.1AI21 Labs35.92025-03
Infinity Instruct 7M Gen Llama3 1 8BBAAI35.82025-03
Llama 3.2 3BMeta35.82026-04
Qwen2.5 1.5BAlibaba / Qwen Team35.82025-03
Qwen2.5 1.5B Instruct AWQAlibaba / Qwen Team35.82025-03
Llama 2 13B hfMeta35.42024-07
Llama 2 13B hfNous Research35.42024-07
Llama 3.2 3B InstructMeta35.32026-04
OLMo 2 1124 7BAllen AI35.12025-03
OLMoE 1B 7B 0924Allen AI34.92025-03
GPT NeoXT Chat Base 20BTogether AI34.62025-03
Llama 3.2 1BMeta34.52025-03
Llama 3.2 1BNous Research34.52025-03
Qwen2.5 Coder 7B Instruct GPTQ Int4Alibaba / Qwen Team34.52026-04
RedPajama INCITE 7B ChatTogether AI34.52025-03
Qwen2.5 0.5BAlibaba / Qwen Team34.32025-03
gemma 1.1 2B itGoogle DeepMind33.92025-03
Ministral 3B (latest)Mistral AI33.82025-03
granite 3.0 1B a400m baseIBM33.72025-03
DeepSeek R1DeepSeek33.52026-04
DeepSeek R1 0528DeepSeek33.52026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA33.52026-04
DeepSeek ReasonerDeepSeek33.52026-04
Qwen2 0.5B InstructAlibaba / Qwen Team33.52024-07
gemma 2B itGoogle DeepMind33.42024-07
Qwen2.5 0.5B InstructAlibaba / Qwen Team33.42025-03
granite 3.1 1B a400m instructIBM33.02025-03
Llama 3.2 1B InstructMeta32.02025-03
Gemma 4 31BGoogle DeepMind30.42026-04
gemma 4 31B itGoogle DeepMind30.42026-04
gemma 4 31B it GGUFUnsloth30.42026-04
Gemma 4 31B IT NVFP4NVIDIA30.42026-04
Qwen3 32BAlibaba / Qwen Team30.12026-04
Qwen3 32B AWQAlibaba / Qwen Team30.12026-04
Qwen3 32B NVFP4NVIDIA30.12026-04
DeepSeek ChatDeepSeek29.82026-04
DeepSeek V2DeepSeek29.82026-04
DeepSeek V2 LiteDeepSeek29.82026-04
DeepSeek V2 Lite ChatDeepSeek29.82026-04
DeepSeek V3DeepSeek29.82026-04
DeepSeek V3 0324DeepSeek29.82026-04
DeepSeek V3.1DeepSeek29.82026-04
DeepSeek V3.2DeepSeek29.82026-04
DeepSeek V3.2 ExpDeepSeek29.82026-04
Gemma 4 26BGoogle DeepMind28.72026-04
Qwen2.5 72B InstructAlibaba / Qwen Team28.32026-04
Mistral Large (latest)Mistral AI26.32026-04
Mistral Large 2.1Mistral AI26.32026-04
Mistral Large 3Mistral AI26.32026-04
DeepSeek R1 Distill Llama 70BDeepSeek25.82026-04
Gemma 3 27BGoogle DeepMind25.22026-04
Qwen3 14BAlibaba / Qwen Team24.82026-04
Qwen3 14B AWQAlibaba / Qwen Team24.82026-04
Qwen3 14B NVFP4NVIDIA24.82026-04
Llama 3.3 70B Instruct NVFP4NVIDIA24.52026-04
Llama-3.3-70B-InstructMeta24.52026-04
phi 4Microsoft23.52026-04
Phi 4 multimodal instructMicrosoft23.52026-04
Llama 3.1 70BMeta22.72026-04
Llama 3.1 70B InstructMeta22.72026-04
Qwen2.5 32B InstructAlibaba / Qwen Team22.12026-04
Qwen2.5 32B Instruct AWQAlibaba / Qwen Team22.12026-04
Qwen3 30B A3B Instruct 2507Alibaba / Qwen Team21.32026-04
Qwen3 30B A3B NVFP4NVIDIA21.32026-04
Qwen3 30B-A3BAlibaba / Qwen Team21.32026-04
DeepSeek R1 Distill Qwen 32BDeepSeek20.82026-04
Gemma 3 12BGoogle DeepMind20.82026-04
gemma 2 27B itGoogle DeepMind20.12026-04
Qwen3 8BAlibaba / Qwen Team19.42026-04
Qwen3 8B AWQAlibaba / Qwen Team19.42026-04
Qwen3 8B BaseAlibaba / Qwen Team19.42026-04
Qwen2.5 14B InstructAlibaba / Qwen Team18.62026-04
Qwen2.5 14B Instruct AWQAlibaba / Qwen Team18.62026-04
Mixtral 8x22BMistral AI18.52026-04
Mixtral 8x22B Instruct v0.1Mistral AI18.52026-04
Yi 1.5 34B01.AI18.22026-04
Yi 1.5 34B Chat01.AI18.22026-04
Command R+Cohere17.32026-04
DeepSeek R1 Distill Qwen 14BDeepSeek16.52026-04
Mistral NemoMistral AI16.12026-04
Mistral Nemo Base 2407Mistral AI16.12026-04

Showing the top 200 of 221.

Data

This page as JSON · Edit on GitHub