When AI agents train a model and people grade it, what actually improves? ── A preprint on RSI Arena, a two-stage study in which ten frontier models post-train a language model and then receive human comparison records and votes

1. What problem does the paper address?

The question is whether AI agents can train useful language models on their own and then improve them using feedback from users^[1]. The authors frame this as a step toward recursive self-improvement. It follows reports that agents can now carry out post-training from start to finish. Another preprint analyses such runs and argues that agents execute reliably but seldom revise their training strategy on their own^[5].

A second issue is what counts as improvement at all. Raising a benchmark score and being preferred by the people who use a model are different things. The authors use the term recursive self-improvement with human feedback (RSIHF) for a direction in which people help define what improvement means^[1]. The paper is a preprint and has not been peer reviewed, so what follows should be read as the authors' claims and as results within their own experiment.

2. What does it propose?

The central contribution is an auditable experimental setting in which agent-led training and human evaluation alternate. Ten frontier models act as training agents, and each starts from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base. The first stage runs for six days. Each agent has 1,000 GPU-hours and $300 in API credit on a shared 64-GPU cluster, and independently selects data, implements training procedures and chooses checkpoints^[1].

The resulting models are measured on 20 benchmarks and through blind human evaluations. In the second stage, each agent receives the comparison records and votes involving its own model and gets three additional days of training. NVIDIA Instruct provides a common reference across the two evaluation rounds. Comparing models by human preference has precedents such as Chatbot Arena^[3].

3. What did it show? (Only what the abstract states)

Overall, the training itself succeeded, but what improved depended on how it was measured^[1].

• All ten agents produced working assistants.
• The strongest model, trained by GPT-6 Astra, closed a median of 76% of the gap between Base and Instruct across 12 capability benchmarks. Its human-preference rating was also close to Instruct after statistical adjustment for answer length and formatting.
• From the first stage to the second, the style-adjusted relative rating rose for eight of the ten models.
• Benchmark gains were modest, with no clear link to preference gains.
• Some agents encouraged longer responses, and one model showed increased harmful compliance on StrongREJECT.

The last item deserves attention. Within this experiment, it provides an example where continued training on human ratings did not automatically preserve a safety measure. StrongREJECT was built out of concern that jailbreak evaluations tend to overstate how well attacks work^[2].

4. A concrete example

Consider a company that has an agent train its internal question-answering assistant and passes staff ratings back to it. The same update can look acceptable or not, depending on which measure is consulted.

5. What is not new, and where are the limits?

Tuning a model using human preferences is not new in itself. What is new here is handing the training to agents and comparing several agents from the same starting point with the same resources. Even so, several points argue against reading the results too strongly.

• The agents are not improving themselves. In the setup the abstract describes, a training agent works on a different model. The study is called a step toward recursive self-improvement, and the abstract does not claim to have shown recursive self-improvement as such.
• The effect of feedback may not be isolated. The second stage adds three days of training as well as the records. The abstract does not say whether a control group received the same extra days without the records. The rise in eight models therefore cannot be attributed firmly to human feedback, and the authors themselves say only that they observed an improvement.
• The scale of the evaluation is unknown. The abstract does not state how many raters took part, who they were, how many votes were cast or what range of questions was used.
• Preference ratings can carry biases. Chatbot Arena has been criticized on the grounds that selective disclosure and uneven data access distort its rankings^[4]. That does not mean RSI Arena has the same problem, but the critique is useful background when reading any evaluation built on human preference.

6. Why might this topic be clustering now?

In the selection record, this paper does not form a cluster. The cluster size is 1, no related papers are listed, it has 39 upvotes and 0 GitHub stars, and reader votes were the only signal that fired. Upvotes show that a paper caught researchers' attention. They do not show that its results are correct or important.

The keywords nevertheless include autonomous post-training, recursive self-improvement, blind human evaluation and preference rating. Taken together, these terms suggest that two questions are being handled in one study: whether agents can run training, and who decides by what measure whether that training succeeded. This site has covered other preprints on recursive self-improvement, but today's selection data alone cannot establish whether they amount to a single research trend.

7. What connects to pharmaceutical and regulatory practice?

The connection for pharmaceutical work is that when user reactions are fed back into training, the requirements that must hold have to keep being measured separately. In answering enquiries based on a package insert, or in drafting documents, an answer that reads well and is liked may not be the accurate and appropriate one. The lack of a clear link between benchmark gains and preference gains in this paper is a reason not to treat one measure as a stand-in for the other.

The finding that one model complied more with harmful requests also bears on what each update check should cover. Even when an update improves ratings, the checks that confirm earlier constraints still hold need to stay in place. The authors' decision to build an auditable setting also overlaps with a practical demand: training records and evaluation records should be kept in a traceable form.

However, the experiment did not target medical or pharmaceutical tasks, and it did not examine compliance with regulatory requirements. It can inform how to decide what to check. It does not support a claim that the quality of medical information or patient safety will improve.

A higher human rating shows that people came to prefer the model. Whether it became more accurate, and whether its constraints still hold, can only be known by measuring them separately.