Tests whether an AI agent can act as a machine learning engineer on 75 real Kaggle competitions, graded against the competitions' own medal thresholds.
unassessed
| Category | agentic |
|---|---|
| Subcategory | machine-learning-engineering agent benchmark (Kaggle competitions) |
| Page status | active |
| Metric | % of competitions earning at least a bronze medal ("Any Medal") |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 75 |
| Publisher | OpenAI |
MLE-bench evaluates whether an AI agent can do the job of a machine learning engineer end to end: given a real Kaggle competition's description, starter files and training data, the agent must explore the data, choose and implement a modelling approach, train it, and produce a submission file, typically over many hours of autonomous work with shell and code-execution access. This is a full workflow rather than a single coding or math problem — the agent makes its own decisions about what to try, debugs its own failures, and manages a time budget, closer to how a human competitor works a Kaggle competition than to a single-turn benchmark question.
Given a Kaggle competition's public description, starter code and training data, an agent with shell and code-execution access must produce a submission file, graded against the competition's own scoring metric and converted into a medal outcome.
No model card in ModelSpec reports this benchmark yet.