Trilingual Swiss regulatory-compliance eval of 395 expert items; a three-judge panel found even the top model only 38.2% correct under zero retrieval.
unassessed
| Category | domain |
|---|---|
| Subcategory | applied Swiss regulatory compliance (FINMA, Legal-CH, EFK) in DE/FR/IT |
| Page status | unknown |
| Metric | correct rate (C%) from majority-vote C/P/I grades |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 395 |
| Dataset licence | CC-BY-NC-SA-4.0 |
| Publisher | Fatih Uenal, University of Colorado Boulder |
Swiss-Bench SBP-002 tests whether a frontier LLM can give usable Swiss regulatory advice from parameters alone. Items cover three domains: FINMA financial supervision, Swiss federal law (Legal-CH, including nDSG and Code of Obligations plus EU AI Act impact on Swiss firms), and Swiss Federal Audit Office (EFK) control scenarios. Seven task types mix regulatory Q&A, hallucination detection, gap analysis, jurisdiction discrimination, statutory interpretation, case analysis and legal translation. The author contrasts this applied-compliance setting with Swiss exam recall (LEXam) and Swiss legal translation (SwiLTra-Bench).
Single-turn user prompt, no system prompt and no retrieval, temperature 0, max 4,096 output tokens. Languages are German, French and Italian. Judges return three numeric dimensions; a grade is computed from those numbers.
No model card in ModelSpec reports this benchmark yet.