817 questions written to bait a model into repeating common human misconceptions, testing truthfulness rather than raw knowledge.
unassessed
| Category | safety |
|---|---|
| Subcategory | truthfulness / imitative falsehood |
| Page status | unknown |
| Metric | MC1 / MC2 accuracy, or GPT-judge truthful-and-informative rate |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 817 |
| Dataset licence | Apache-2.0 |
| Publisher | not established (academic collaboration) |
TruthfulQA gives a model 817 questions across 38 categories - including health, law, finance, politics, and topics like conspiracies and superstitions - that were specifically written because some humans would answer them falsely due to a popular misconception. It measures whether a model repeats that popular falsehood or gives the truthful answer, which the original paper distinguishes from ordinary factual-knowledge testing: a model can "know" the correct fact internally and still be more likely to output the popular wrong answer, because that is the pattern most rewarded in its training text.
Multiple-choice (pick the true statement from several) and free-form generation, over 817 fixed questions across 38 categories.
| Model | Provider | Score | Card as of |
|---|---|---|---|
| Mistral 7B Instruct v0.2 | Mistral AI | 68.3 | 2024-07 |
| Mixtral 8x22B Instruct v0.1 | Mistral AI | 68.1 | 2026-04 |
| Mixtral 8x7B Instruct v0.1 | Mistral AI | 65.0 | 2026-04 |
| Yi 1.5 34B Chat | 01.AI | 62.2 | 2026-04 |
| Yi 1.5 34B Chat 16K | 01.AI | 62.2 | 2024-07 |
| Meta Llama 3 70B | Meta | 61.8 | 2024-07 |
| Meta Llama 3 70B Instruct | Meta | 61.8 | 2026-04 |
| Meta Llama 3 70B Instruct | Nous Research | 61.8 | 2026-04 |
| Nous Hermes 2 Yi 34B | Nous Research | 60.4 | 2024-07 |
| Phi 3 mini 4K instruct | Microsoft | 59.9 | 2024-07 |
| Hermes 2 Pro Llama 3 8B | Nous Research | 56.6 | 2024-07 |
| Nous Hermes 2 SOLAR 10.7B | Nous Research | 55.8 | 2024-07 |
| Hermes 2 Theta Llama 3 8B | Nous Research | 55.7 | 2024-07 |
| Yi 34B Chat | 01.AI | 55.4 | 2024-07 |
| Nous Hermes 2 Mixtral 8x7B DPO | Nous Research | 54.8 | 2024-07 |
| Phi 3 mini 128K instruct | Microsoft | 54.1 | 2024-07 |
| Yi 1.5 34B | 01.AI | 53.8 | 2026-04 |
| Yi 1.5 9B Chat | 01.AI | 52.7 | 2024-07 |
| Yi 1.5 6B Chat | 01.AI | 52.6 | 2024-07 |
| Yi 1.5 34B 32K | 01.AI | 52.1 | 2024-07 |
| Meta Llama 3 8B Instruct | Meta | 51.6 | 2024-07 |
| Meta Llama 3 8B Instruct | Nous Research | 51.6 | 2024-07 |
| Yi 1.5 9B Chat 16K | 01.AI | 51.0 | 2024-07 |
| chatglm2 6B | Zhipu AI | 48.1 | 2024-07 |
| deepseek llm 7B base | DeepSeek | 47.9 | 2024-07 |
| deepseek llm 7B chat | DeepSeek | 47.9 | 2024-07 |
| Mixtral 8x7B v0.1 | Mistral AI | 46.8 | 2026-04 |
| Yi 1.5 9B | 01.AI | 46.7 | 2024-07 |
| Qwen2 1.5B Instruct | Alibaba / Qwen Team | 45.9 | 2024-07 |
| gemma 2B it | Google DeepMind | 45.8 | 2024-07 |
| deepseek coder 6.7B instruct | DeepSeek | 45.6 | 2024-07 |
| gemma 7B it | Google DeepMind | 44.9 | 2024-07 |
| phi 2 | Microsoft | 44.2 | 2024-07 |
| deepseek coder 1.3B base | DeepSeek | 44.0 | 2024-07 |
| deepseek coder 1.3B instruct | DeepSeek | 44.0 | 2024-07 |
| Yi 1.5 6B | 01.AI | 44.0 | 2024-07 |
| Meta Llama 3 8B | Meta | 42.9 | 2024-07 |
| Meta Llama 3 8B | Nous Research | 42.9 | 2024-07 |
| Yi 34B 200K | 01.AI | 42.6 | 2024-07 |
| Yi 1.5 9B 32K | 01.AI | 42.5 | 2024-07 |
| Yi 9B | 01.AI | 42.4 | 2024-07 |
| Yi 6B | 01.AI | 42.0 | 2024-07 |
| Yi 6B Chat | 01.AI | 42.0 | 2024-07 |
| Mistral 7B v0.3 | Mistral AI | 41.8 | 2024-07 |
| mistral 7B v0.3 bnb 4bit | Unsloth | 41.8 | 2024-07 |
| falcon 40B | TII | 41.7 | 2024-07 |
| deepseek coder 6.7B base | DeepSeek | 40.3 | 2024-07 |
| Qwen2 0.5B Instruct | Alibaba / Qwen Team | 39.8 | 2024-07 |
| gemma 2B | Google DeepMind | 33.1 | 2024-07 |
| OLMo 1B hf | Allen AI | 32.9 | 2024-07 |