---
title: "Adaptive Proxy Evaluation for AI Research Agents"
summary: "Cheap ML evaluations make agent search possible, but their scores can mislead and candidate-defined tests are hard to compare. This article explains our adaptive proxy method, its cross-family ranking results, and the Ventilator run that reached 0.1354 MAE in 12 hours."
date: 2026-06-25
topic: "AI Research"
author: "Yogendra Manawat"
---

# Adaptive Proxy Evaluation for AI Research Agents

An AI agent trying to improve a machine-learning solution needs to test its ideas. That can get expensive quickly. If every candidate requires a full training run, the agent may spend most of its budget evaluating just a few ideas. If it uses cheap tests, those tests may rank the candidates differently from a full run.

This is the problem we studied in [Adaptive Proxy Evaluation for Autonomously Improving ML Agents](https://openreview.net/forum?id=OEmE8fF4OF), a paper I co-authored. The goal was to make evaluation cheap enough for exploration and reliable enough to guide the final choice.

The paper's abstract and author list are also available on the [ICML paper page](https://icml.cc/virtual/2026/74826).

## The proxy problem

A *proxy evaluation* is a quicker version of the real test. An agent might train for fewer steps, use less data, or otherwise reduce the work needed to get a score. That score helps it decide what to try next.

The danger is that a promising proxy score may disappear when the candidate is trained and evaluated properly. A small run can favor ideas that start quickly but do not hold up with more training. Candidates can also define their own evaluation, which makes reported metrics hard to compare. Running the full evaluation for every idea would solve some of these problems, but at too much cost for a broad search.

Our approach puts an adaptive evaluation layer around the search. Candidates are compared under the same conditions at a given stage. Early in the search, the evaluation is cheaper so the agent can explore more ideas. As the search narrows, the proxy becomes more faithful to the full evaluation. The agent gets fast feedback when it needs breadth and stronger evidence when it needs to choose.

![Figure 2 from the paper: the system combines multi-checkpoint proxy evaluation, an adaptive proxy schedule, a Bradley–Terry comparison arena, an Overseer agent, coding agents, and a composite score.](https://www.yogendramanawat.com/articles/research/adaptive-proxy-method.webp)

*Figure 2 from the paper. The method overview shows how the Overseer changes evaluation fidelity while the shared harness and cross-family comparisons guide the search.*

| Search stage | Evaluation | Purpose |
| --- | --- | --- |
| Early exploration | Cheaper, lower-fidelity proxy; same setup for each candidate | Compare many ideas quickly |
| Later refinement | More faithful, higher-fidelity evaluation; same setup for each candidate | Recheck the strongest ideas with more reliable feedback |

## What the result tells us

The paper applies this approach to MLEvolve, a Monte Carlo Graph Search system that explores candidate machine-learning solutions. On MLE-bench's Ventilator Pressure Prediction task, a reported run reached a **mean absolute error of 0.1354 within 12 hours**, which the paper reports as a state-of-the-art result.

![Figure 3 from the paper: global best proxy run score rises across 401 journal steps, with phase transitions at steps 176, 251, and 326.](https://www.yogendramanawat.com/articles/research/adaptive-proxy-phase-transitions.webp)

*Figure 3 from the paper. The best proxy score improves through the run, with the Overseer moving to a new phase after progress plateaus.*

The number is useful because it ties the evaluation design to an actual search budget and task. The broader point is about what the agent learns from its experiments. A search algorithm can generate many candidates, but it can only make good choices if its feedback is comparable and increasingly trustworthy.

The paper also compares two ways to rank candidates against full-budget performance. For pairs from different solution families, the Bradley–Terry ranking agreed with full-budget performance on 88.2% of pairs, compared with 84.4% for the proxy score. Within a family, the proxy score was slightly higher: 87.7% versus 86.9%. The result suggests that the cross-family comparison helps most when raw proxy scores are harder to compare.

![Figure 7 from the paper: pairwise agreement with full-training performance at 150 epochs, comparing proxy run scores with Bradley–Terry rankings within and across solution families.](https://www.yogendramanawat.com/articles/research/adaptive-proxy-ranking-agreement.webp)

*Figure 7 from the paper. Bradley–Terry comparisons improve agreement across families; within-family agreement is slightly higher for the proxy score.*

## Why I care about evaluation

When building an agent that improves its own work, it is tempting to focus on how it generates ideas. The evaluation loop deserves just as much attention. It decides which ideas survive, which paths the agent explores, and when it should stop searching.

Adaptive proxy evaluation is one way to manage that loop. Its benefit depends on how well the proxy preserves the ranking that matters for the real task. The Ventilator result shows what the approach achieved in one reported setting; testing it across more tasks and budgets is how we learn where it holds up.

## Reference

S. Verboomen, A. Ivanova, Y. Manawat, et al. (2026). [Adaptive Proxy Evaluation for Autonomously Improving ML Agents](https://openreview.net/forum?id=OEmE8fF4OF). [ICML page](https://icml.cc/virtual/2026/74826).