{
 "body": "\n## What it measures\n\nMLE-bench evaluates whether an AI agent can do the job of a machine learning engineer end to end: given a\nreal Kaggle competition's description, starter files and training data, the agent must explore the data,\nchoose and implement a modelling approach, train it, and produce a submission file, typically over many\nhours of autonomous work with shell and code-execution access. This is a full workflow rather than a\nsingle coding or math problem \u2014 the agent makes its own decisions about what to try, debugs its own\nfailures, and manages a time budget, rather than answering a single-turn question.\n\n## How it is scored\n\nEach competition's submission is graded against Kaggle's own scoring metric for that competition and\nconverted into a medal outcome \u2014 bronze, silver, gold or none \u2014 using threshold rules taken from Kaggle's\nreal competition rules, which scale with the number of competing teams (on a competition with 1,000 or\nmore teams, for example, bronze is the top 10%, silver the top 5%, and gold roughly the top 10 teams plus\n0.2%). The paper's headline metric is \"Any Medal\": the percentage of\ncompetitions in which an agent earns at least a bronze. Because Kaggle does not release official\nheld-out test labels, MLE-bench's own preparation scripts re-split each competition's public training\ndata to create a new held-out test portion rather than using Kaggle's real, non-public test set.\n\n## Dataset and licence\n\nThe full benchmark spans 75 real, historical Kaggle competitions, grouped by the authors into 22\nlow-complexity, 38 medium-complexity and 15 high-complexity competitions; a separate \"Lite\" subset uses\nthe 22 low-complexity competitions for faster iteration. Aggregate competition data totals roughly 3.3TB,\nwith individual competition training sets ranging from 144 rows to well over one hundred billion. The\nbenchmark's own code is released on GitHub under the MIT licence, but the underlying competition data is\nnot redistributed with it: each competition carries its own Kaggle rules and terms, and users must\ndownload it themselves through the Kaggle API using their own account credentials after accepting each\ncompetition's rules, which is why this page leaves the dataset licence and URL fields blank rather than\nassign one blanket value.\n\n## Who publishes it\n\nMLE-bench comes from OpenAI: Chan Jun Shern, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan\nMays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng and Aleksander M\u0105dry, posted\nto arXiv in October 2024. OpenAI maintains the reference implementation at github.com/openai/mle-bench and\nhas continued to cite it in subsequent evaluations of agentic and research capability.\n\n## Lineage\n\nMLE-bench names no single predecessor benchmark, though its evaluation design draws on, and directly\ncompares against, two earlier open-source agent scaffolds built for adjacent tasks: MLAgentBench's\nMLAB/ResearchAgent scaffold and OpenHands' CodeActAgent, alongside AIDE, a scaffold purpose-built for\nKaggle-style competitions that the paper's own experiments found strongest. It has no official successor\nand no variant pages in this repository.\n\n## Saturation and contamination\n\nAt release, OpenAI's own best-performing combination \u2014 o1-preview scaffolded with AIDE \u2014 earned at least\na bronze medal in 16.9% of the full 75-competition set, the strongest of the scaffolds tested. That\nleaves most competitions unsolved to a medal-worthy standard, so this page reads the benchmark as open\nrather than saturated; no later, independently reported full-benchmark number was found in this research\nto say how much that figure has moved since. Contamination risk sits at medium: the paper's own authors\nraise the concern directly, since every competition is real and publicly documented, often with public\nwinning solutions and notebooks a model could have seen during pretraining. They built plagiarism-detection\nand log-analysis tooling to check for it and found no systematic evidence of inflated GPT-4o performance,\nbut state plainly that this does not rule out \"subtler effects of contamination.\"\n\n## How to run it\n\nThe reference harness lives in OpenAI's own GitHub repository and grades submissions inside Docker\ncontainers built per competition. inspect_evals provides an independent implementation with three\nregistered tasks \u2014 `mle_bench` (a single default competition), `mle_bench_lite` (20 competitions) and\n`mle_bench_full` (73 competitions) \u2014 counts that run a little below the paper's own 22-competition Lite\nand 75-competition Full groupings, worth checking before comparing a harness-reported score directly\nagainst the paper's numbers. Running either implementation requires Docker (inspect_evals' image is\nroughly 25GB and takes about 30 minutes to build on first run) and the evaluator's own Kaggle account\ncredentials, since competition data is pulled live from Kaggle rather than shipped with the benchmark.\n\n## Reading the numbers\n\nA high MLE-bench score shows an agent can independently execute a substantial share of the\nmachine-learning-engineering workflow \u2014 data exploration, modelling, debugging and submission \u2014 well\nenough to place competitively against real Kaggle competitors. It does not\nshow the agent can do novel ML research without a well-defined competition structure and scoring metric\nalready in place, and because different scaffolds (AIDE-style vs. MLAgentBench-style vs. CodeAct-style\nagents) produce very different results on the same underlying model, a score is at least as much a\nproperty of the scaffold as of the model. Always check which subset (Lite vs. the full 75) and which\nscaffold a reported number used before comparing it with another.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mle_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "mle-bench"
  ],
  "category": "agentic",
  "contamination": {
   "note": "The paper's own authors raise this directly: \"it's possible that models have trained on all public Kaggle material including competition details, solutions, and even the datasets including our test set,\" since every competition is real and publicly documented, often with public winning solutions and notebooks. They built plagiarism-detection and log-analysis tooling to check for it and found no systematic evidence of inflated GPT-4o performance, but state plainly that this does not rule out \"subtler effects of contamination.\"\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "tabular",
    "text",
    "image",
    "code"
   ],
   "public_test_set": false,
   "size": 75,
   "size_note": "75 Kaggle competitions in the full set, grouped by the authors into 22 low-complexity, 38 medium-complexity and 15 high-complexity; a separate 22-competition \"Lite\" subset uses the low-complexity group specifically. Aggregate competition data totals roughly 3.3TB; individual competition training sets range from 144 rows to well over one hundred billion. Kaggle does not release official held-out test labels, so MLE-bench's own preparation scripts re-split each competition's public training data to create a new held-out test portion.\n",
   "splits": "75 competitions total (22 low / 38 medium / 15 high complexity); a 22-competition Lite subset; each competition internally re-split into train/test by MLE-bench's own scripts since Kaggle's real test labels are not public",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice A",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "mle_bench",
   "lm_eval": "",
   "opencompass": "",
   "other": "inspect_evals registers three task variants: mle_bench (a single default competition, spaceship-titanic), mle_bench_lite (20 competitions) and mle_bench_full (73 competitions) \u2014 counts a little below the paper's own 22-competition Lite and 75-competition Full groupings, a small but real discrepancy worth checking before comparing harness-reported scores to the paper's own numbers. Running it requires Docker (roughly a 25GB image, about 30 minutes to build on first run) and the evaluator's own Kaggle account credentials (kaggle.json), since competition data is pulled live from Kaggle rather than shipped with the benchmark.\n"
  },
  "id": "mle_bench",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "MLE-bench evaluates whether an AI agent can do the job of a machine learning engineer end to end: given a real Kaggle competition's description, starter files and training data, the agent must explore the data, choose and implement a modelling approach, train it, and produce a submission file, typically over many hours of autonomous work with shell and code-execution access. This is a full workflow rather than a single coding or math problem \u2014 the agent makes its own decisions about what to try, debugs its own failures, and manages a time budget, closer to how a human competitor works a Kaggle competition than to a single-turn benchmark question.\n",
  "metric": {
   "baseline_note": "Medal thresholds are themselves defined relative to the original human Kaggle competitors' final leaderboard placement (for example, bronze is the top 10% of teams on a competition with 1,000 or more teams), so \"any medal\" already means beating a meaningful share of the original human field rather than a separately measured average-human score.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "% of competitions earning at least a bronze medal (\"Any Medal\")",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "MLE-bench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2410.07095",
   "title": "MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering",
   "url": "https://arxiv.org/abs/2410.07095",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Chan Jun Shern",
    "Neil Chowdhury",
    "Oliver Jaffe",
    "James Aung",
    "Dane Sherburn",
    "Evan Mays",
    "Giulio Starace",
    "Kevin Liu",
    "Leon Maksin",
    "Tejal Patwardhan",
    "Lilian Weng",
    "Aleksander M\u0105dry"
   ],
   "org": "OpenAI",
   "url": "https://github.com/openai/mle-bench"
  },
  "released": "2024-10",
  "repo_url": "https://github.com/openai/mle-bench",
  "saturation": {
   "as_of": "2024-10",
   "note": "OpenAI's own headline number at release: o1-preview scaffolded with AIDE reached at least a bronze medal in 16.9% of the full 75-competition set, the strongest of the scaffolds the paper tested (AIDE, the MLAB/ResearchAgent scaffold from MLAgentBench, and CodeActAgent from OpenHands). That leaves most competitions unsolved to a medal-worthy standard. No later, independently reported full-benchmark number was found in this research, so how much that figure has moved since is not established here.\n",
   "status": "open",
   "top_score": 16.9
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering",
    "url": "https://arxiv.org/abs/2410.07095"
   },
   {
    "accessed": "2026-09-08",
    "title": "openai/mle-bench repository (README)",
    "url": "https://github.com/openai/mle-bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "openai/mle-bench LICENSE (MIT)",
    "url": "https://raw.githubusercontent.com/openai/mle-bench/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals mle_bench task (mle_bench / mle_bench_lite / mle_bench_full)",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/mle_bench"
   }
  ],
  "status": "active",
  "subcategory": "machine-learning-engineering agent benchmark (Kaggle competitions)",
  "summary": "Tests whether an AI agent can act as a machine learning engineer on 75 real Kaggle competitions, graded against the competitions' own medal thresholds.",
  "tags": [
   "agentic",
   "machine-learning-engineering",
   "kaggle",
   "code-execution",
   "autonomous-agent",
   "docker"
  ],
  "task_format": "Given a Kaggle competition's public description, starter code and training data, an agent with shell and code-execution access must produce a submission file, graded against the competition's own scoring metric and converted into a medal outcome.\n"
 }
}