An AI research agent can run experiment after experiment and still miss the most useful direction. It may keep tuning a model when the problem calls for different features, a better validation split, or a closer look at the data. More execution alone does not fix a weak research plan.
That is the idea behind Socrates, a project I co-authored. We paired a tool-using Scientist with an advisor that has a strict job: ask questions about the Scientist's plan. The advisor cannot run code, provide an answer, or tell the Scientist what to do.
A question before the next experiment
The Scientist proposes an experiment. Socrates looks at the plan and asks questions that make the Scientist examine its assumptions. The Scientist can revise the plan and discuss it with Socrates before the experiment runs. Socrates must approve the plan before the next experiment can begin.
Example review exchange from the Socrates project repository. The advisor questions the Scientist's direction; the Scientist changes its approach and improves the task score.
The restriction on the advisor matters. If it supplied solutions, we could not tell whether the Scientist had reasoned more carefully or simply copied another agent's idea. A question instead pushes the Scientist to use knowledge it may already have, but has not brought into the current plan.
One example from our experiments is an iceberg classification task using radar images. The Scientist was spending effort on incremental tuning. Socrates questioned whether that was closing the gap. The Scientist then explored features informed by the domain rather than continuing to adjust the same basic approach. That kind of change in direction is the behavior we wanted to encourage.
What we measured
We compared three setups on five MLE-bench Kaggle tasks: a Scientist working alone, a Scientist with a second agent giving general encouragement, and a Scientist reviewed by Socrates. The tasks covered radar imagery, RNA degradation, ventilator pressure, player contact detection, and GPS positioning.
In the released results, the Socrates setup beat the Scientist working alone on all five tasks. It beat the general-encouragement advisor on four of the five. The reported average improvement over the Scientist alone was 55.9% across those task-specific test metrics. That average needs context: the tasks use different metrics, and one large gain has a strong effect on the mean.
| Task | Metric direction | Scientist alone | General encouragement | Socrates |
|---|---|---|---|---|
| Statoil iceberg classification | Log loss ↓ | 0.255 | 0.251 | 0.229 |
| COVID vaccine prediction | MCRMSE ↓ | 0.389 | 0.308 | 0.294 |
| Ventilator pressure prediction | MAE ↓ | 1.534 | 0.815 | 0.853 |
| NFL contact detection | MCC ↑ | 0.198 | 0.537 | 0.584 |
| Smartphone positioning | Haversine distance ↓ | 6.285 | 5.993 | 5.984 |
The metrics have different scales; the table reports each task's test score, not a pooled score. Lower is better for log loss, MCRMSE, MAE, and Haversine distance. Higher is better for MCC.
The comparison with general encouragement is especially useful. A second agent by itself was not the whole idea. The review protocol gave it a narrow role and forced the Scientist to revisit the reasoning behind a proposed experiment.
Where I think this is useful
Research work often fails at the choice of the next experiment. You can have a strong model and working tools, but still spend your budget on small variations of an unpromising approach. Socrates adds a pause at that decision point. It asks the Scientist to justify the direction before spending more time and compute.
These results come from five machine-learning competitions, so they do not establish how the protocol will behave across every research field. Agent runs can also vary from seed to seed. Still, the experiment points to a practical design choice: better questions at the right moment can change what an agent tries next.
Reference
D. Vrabac, P. Hebbar, Y. Manawat, et al. (2026). Socrates: Structured Questioning Unlocks Latent Knowledge in AI Research Agents. Code and results.