Tests whether a language agent can propose and implement a genuinely novel ML method across 7 real research-competition tasks, scored against each competition's own baseline and top human result.
unassessed
| Category | agentic |
|---|---|
| Subcategory | machine-learning research-competition agent benchmark (novel methods vs. baseline and top human) |
| Page status | active |
| Metric | relative_improvement_to_human (primary), with absolute_improvement_to_baseline reported alongside |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 7 |
| Dataset licence | CC-BY-4.0 (Hugging Face dataset card and the companion Space's cardData both state this) |
| Publisher | University of Michigan; LG AI Research; University of Illinois at Chicago |
MLRC-Bench gives an agent an open machine-learning research problem adapted from a real competition at a recent ML conference (NeurIPS, KDD, ECCV and similar venues, spanning areas such as LLM safety, computer vision and AI for science): a baseline code repository, background literature, and dev-set data to iterate against. The agent must propose a new method, implement it in code, and run it, with its final version evaluated once on a held test split. Unlike end-to-end agent benchmarks that grade a full pipeline with an LLM-as-judge, MLRC-Bench isolates the specific steps of proposing and implementing a novel method and grades the result with each competition's own objective metric, explicitly to avoid rewarding an agent for sounding innovative rather than being effective.
The agent works inside a per-task sandboxed environment (its own conda/Docker setup, described in a README the agent can read) with shell and code-execution access, iterating on a development split for a time or message budget. Its method is then run once via the task's own `main.py -m <method_name> -p test` command on the held test split; some tasks (llm-merging, machine-unlearning, product-rec) require submitting a run to Kaggle or AIcrowd to obtain the final score rather than scoring locally.
No model card in ModelSpec reports this benchmark yet.