A 168,093-sample role-play benchmark over 100 characters; OpenCompass scores English and Chinese splits with ROUGE against RoleGPT-style references.
unassessed
| Category | generation |
|---|---|
| Subcategory | character-level role-play: instruction and role generalization, English and Chinese |
| Page status | active |
| Metric | ROUGE-L (OpenCompass RougeEvaluator / JiebaRougeEvaluator); paper also reports GPT-3.5 judgments |
| Direction | higher_is_better |
| Dataset size | 168093 |
| Dataset licence | Apache-2.0 |
| Publisher | InteractiveNLP-Team (RoleLLM) |
RoleBench tests whether a model can stay in character. Each item names a role, gives a short profile, and asks a question. The model must answer in that voice without breaking character. The paper splits evaluation two ways: instruction generalization (held-out questions for seen roles) and role generalization (held-out English roles). Profiles cover 95 English characters and 5 Chinese ones (100 roles). References come from RoleGPT (GPT-4 role prompting) and Context-Instruct, not from human scripts of the same replies. OpenCompass runs three of those splits as generation tasks.
Chat-style generation: a system prompt that assigns the role and description, then a user question. OpenCompass is zero-shot with max_out_len 512. English uses RougeEvaluator; Chinese uses JiebaRougeEvaluator.
No model card in ModelSpec reports this benchmark yet.