An AI agent is more than a model. There is also the code around it: the instructions it receives, the tools it can use, how it handles errors, and how it turns its work into a final answer. We often call that surrounding code the harness. If an agent does poorly on a task, the problem could be in either place. It might need a better workflow, or the model itself might need to learn something about the task.
That was the starting point for SIA, a project I co-authored. We wanted to see what happens when one feedback loop can improve both parts.
Figure from the SIA paper, comparing the baseline, harness-only updates (SIA-H), and combined harness and weight updates (SIA-W+H) across all three tasks.
Two ways an agent can improve
Imagine an agent writing a GPU kernel. It might fail because its code does not compile, because it does not use the error message well, or because the kernel is correct but slow. A harness update can help with the first two: catch the compiler error, pass it back clearly, retry, and measure runtime consistently. A weight update can help the model produce better kernel designs in the first place.
SIA has a task-specific agent that does the work and a Feedback-Agent that studies what happened. After a run, the Feedback-Agent reads the agent's code, its execution trace, and its score. It can then revise the harness or trigger training that updates the model's weights. The next run uses the improved version, and the loop continues. A verifier supplies the score that tells the system whether a change helped.
The two updates act on different parts of the system. Harness changes improve the workflow around the model: prompts, tools, parsing, retries, and search. Weight changes adapt the model to patterns in the task. In our experiments, the loop began by improving the harness and moved to weight updates when those gains stalled.
What we tested
We tested SIA on three quite different tasks:
- Law: choose the right criminal charge from 191 categories for Chinese legal cases.
- Systems: write a fast CUDA kernel for an operation used in protein structure prediction.
- Biology: improve the reconstruction of missing values in single-cell RNA data.
The results separate the contribution of harness updates from the combined system:
| Task | Initial score | Harness updates | Harness and weight updates | Previous best |
|---|---|---|---|---|
| LawBench charge classification (accuracy) | 13.5% | 50.0% | 70.1% | 45.0% |
| TriMul CUDA kernel (runtime) | 14,254 μs | 12,483 μs | 1,017 μs | 1,161 μs |
| Single-cell RNA denoising (mse_norm) | 0.048 | 0.241 | 0.289 | 0.240 |
On the legal task, harness updates raised accuracy to 50.0%; weight updates brought it to 70.1%. For the GPU task, lower runtime is better. For denoising, a higher mse_norm score is better. Those are three different measures, so they should not be treated as one general score for SIA. The table shows that the combined approach improved each reported result beyond the harness-only version.
Those are three different measures, so they should not be treated as one general score for SIA. What they show in these experiments is that changing the harness helped, and adapting the model brought further gains on each task.
What I find interesting about it
The biology result is a nice example of the distinction. Harness updates explored parameter settings and eventually levelled off. After a weight update, the agent produced a small post-processing step that clipped and rounded the reconstructed values to non-negative integers. It was a change in the solution the model generated, not another change to the surrounding workflow.
There is a practical reason to care about this. Useful AI in law, science, and engineering has to work with the details of each domain. A better prompt or a more reliable retry loop can go a long way, but sometimes the model also needs to adapt. SIA gives the system a way to work on both.
The results are still limited to the three benchmarks in the paper. Both update paths use feedback from a verifier, so a system could become good at that score without improving as much on new cases or a changed task. Testing how well the improvements transfer is an important next step. For now, SIA is an experiment in making improvement itself part of the agent's workflow.
Reference
P. Hebbar, Y. Manawat, et al. (2026). SIA: Self Improving AI with Harness & Weight Updates. arXiv:2605.27276. Code.