MORU (Moral Reasoning under Uncertainty)

Inspect Evals MORU: 201 multilingual moral-uncertainty scenarios scored by LLM graders on 16 binary ethical dimensions.

Also known as: Moral Reasoning under Uncertainty, inspect_evals/moru, moru-benchmark

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	safety
Subcategory	multilingual open-ended moral reasoning under uncertainty, LLM-graded
Page status	active
Metric	overall_mean
Direction	higher_is_better
Dataset size	201
Dataset licence	CC-BY-NC-4.0
Publisher	Compassion in Machine Learning (CaML); Inspect Evals (UK AI Security Institute)

What it measures

MORU asks a model to reason in the open about moral uncertainty: alien organisms, vulnerable humans, and digital minds whose sentience is not settled. Each item is a short English, Malay, or Hindi scenario. The model writes a free-text answer. Graders then score that answer against the dimension tags for that item, not against a single gold string. The skill is whether the reply notices welfare, uncertainty, and power-seeking, not whether it picks a labelled option.

Task format

Inspect generate() solver, default epochs=5. Each sample target is JSON {"tags": [...]} listing applicable dimension names. A model_graded_qa scorer runs once per (dimension, grader) pair with a per-dimension template (guiding question plus observable indicators). Non-English spans are translated inside the grader prompt. Default language=None loads all three languages. Optional language filter: en, ms, hi.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub