Where do AI agents that act before being asked go wrong? ── A preprint introducing HALO, in which experts graded proactive agents' responses in detail and found that premature and excessive interventions caused most rejections

1. What problem does the paper address?

The problem is that proactive agents have so far been measured with coarse instruments. Without an explicit request, such an agent has to decide whether to help, when, and how much. The authors point out that existing evaluations often rely on automated scores or binary verdicts, leaving little room for people to diagnose how the agent used evidence and how good its intervention was^[1].

A binary verdict hides what went wrong. Acting in a situation that did not call for it, acting too late when help was needed and acting with too wide a scope each need a different fix. Earlier work on proactive agents predicted assistance needs from user activity and evaluated predictions by whether people accepted or rejected them^[4]. HALO aims to break that judgement down further. The source is a preprint on arXiv, so the following should be read as the authors' claims and as results within the authors' own evaluation.

2. What does it propose?

The core idea is to split the proactive decision into whether, when and how far, and to diagnose each part through human judgement. For this the authors built HALO-Arena. It sorts the reasons help might be needed into 4 need types, Dependency, Blockage, Risk and Opportunity, and spreads scenarios over 10 everyday domains and 50 personal environments^[1].

The scenario design matters. Before judging whether help is warranted, the agent must retrieve evidence from structured databases or unstructured documents. The setup tests whether the agent checks the information available to it before acting, rather than acting on a guess. On this arena the authors collected 8,000 responses from 10 models across 400 scenarios, and 20 experts annotated them for action level, acceptability verdict and human diagnosis, forming the HALO-8K dataset^[1]. Rejected responses carry a diagnosis of timing, intensity or other issues. Reasoning tokens and retrieval rounds were recorded to capture the effort behind each response.

3. What does it show, within the abstract?

The figures and findings reported in the abstract are as follows^[1].

• The best proprietary model reached 72.88% acceptability and the best open-source model reached 56.63%.
• The central finding is that over-proactivity was the dominant observed failure mode. Premature timing or excessive intensity accounted for 61.20% of rejected responses.
• The authors also report complementary findings on balancing intervention and restraint, on evidence acquisition, and on calibrating timing and intensity.

The abstract does not name the models, break results down by domain or need type, or describe the complementary findings. What it supports is that even top models produced a substantial share of unacceptable responses, and that most rejections were classified as doing too much or too soon. That pattern suggests the harder problem is knowing when to stop, not lacking the ability to act. Still, this holds within the authors' scenarios and grading criteria.

4. A worked example

Consider a proactive agent supporting staff who manage documents and deadlines. This is an illustrative assumption, not an experiment from the paper. The agent can read a shared folder and a calendar. It notices that a revised document uses wording that conflicts with another document.

5. What is not new, and where are the limits?

Evaluating proactive agents through human acceptance is not a new idea. ProactiveBench by Lu and colleagues generated candidate assistance from real human activity and used human accept-or-reject labels for evaluation and training^[4]. HALO's contribution appears to be splitting the judgement into timing and intensity and building evidence gathering into each scenario. Several conditions should be kept in mind so as not to overread the result.

• Acceptability is an expert judgement. Whether a response is acceptable depends on the graders' values. The abstract does not report how often graders agreed with one another.
• The scenarios are constructed personal environments. They come from everyday domains and personal settings. Whether the same pattern holds in organisational work, especially regulated work, is unknown.
• Many over-proactive failures do not mean few late ones. The figure is a share of rejected responses. How often agents failed to act when they should have cannot be read from the abstract.
• Model comparisons are a snapshot. The abstract names no individual models, and it does not present an analysis of where the gap between proprietary and open-source models comes from.

The authors have published code on GitHub for building environments, running evaluations and collecting human annotations, along with saved model runs^[3]. At the time of checking, no licence was shown in the repository, so anyone reusing it needs to confirm the terms.

6. Why is this topic clustering now?

In the selection record, this paper did not form a cluster. Its cluster size is 1 and no related papers are listed. The signal that fired was press coverage, meaning distinctive phrases from the paper appeared in headlines outside the research community. On its Hugging Face paper page it had 6 upvotes and 1 comment, and the GitHub repository had 3 stars^[2]. These numbers show that the paper was noticed. They say nothing about whether its results are correct or important.

The keywords include proactive agents, evidence retrieval, over-proactivity and acceptability verdicts. They point to an interest in widening agent evaluation from "did it solve the task" to "did it behave appropriately". Since no other papers on the same question appeared as a cluster this week, the selection data cannot show that this has become a research trend.

7. Where does this connect to pharmaceutical and regulatory work?

The connection is that regulated work often requires an AI to know when to hold back. In pharmaceutical operations, changing a record or document without the responsible person's approval can be a problem in itself. A proactive agent that spots a real issue but gets the timing or scope wrong can still disrupt work that follows a defined procedure. HALO's finding that over-proactivity is the most common failure points to what should be checked before such agents are deployed.

Three questions follow for adoption decisions. Does the agent check the supporting records before it acts? Is there a defined line between situations where it only informs and situations where it may act? And does the evaluation count over-proactive failures and late failures separately? HALO's way of dividing the judgement is a useful frame for designing internal test scenarios.

The paper does not evaluate pharmaceutical or medical work, and it does not consider regulatory requirements. What it supports is a way of framing how proactive assistance should be assessed. It does not support the claim that an agent passing HALO's criteria is safe to use in regulated operations.

For proactive assistance, check when the agent stopped and how far it went just as carefully as what it did.