A new benchmark called DiscoBench put AI search agents to the test on ambiguous queries. The result is surprising: the problem isn't the ability to find information, but the inability to ask the user for clarification when a question is poorly defined. As a result, models that repeat searches instead of asking follow-up questions reach just 51.9% accuracy, which is worse than simply guessing.
On top of that, the best model tested tops out at 43% overall accuracy. Yet when ambiguity is removed from the queries, accuracy jumps by up to 40 percentage points. This changes how we should evaluate AI agents and integrate them into business workflows. In particular, if you're adopting AI-based solutions for research, content or customer intelligence, you need to rethink how you design queries and workflows.
At SHM Studio, we closely follow how AI agents are evolving in digital marketing. That's why we believe these findings have real implications for Italian marketing managers who are evaluating or already using AI tools in their processes. In this analysis, we dig into the anatomy of the problem, practical use cases and actionable guidance.
The benchmark that changes the narrative on AI agents
For months, the debate on AI search agents has focused on the quality of the sources they retrieve. But recent research shifts the focus elsewhere. The DiscoBench benchmark, analyzed by The Decoder, shows that the real bottleneck isn't information retrieval. It's how the agent handles ambiguity in the initial query.
In practice, when a user makes a vague or polysemous request, the AI agent should ask for clarification. Instead, most models keep iterating on searches, hoping to converge on an answer. This approach backfires: models that repeat searches reach 51.9% accuracy. Models that simply guess do better.
So the problem isn't computational. It's communicative. And that distinction has direct consequences for anyone integrating AI agents into marketing and business intelligence processes.
The anatomy of the problem: why the agent doesn't ask
To understand the phenomenon, it helps to look at how a typical AI search agent works. The model receives a query, builds a search plan, calls search engines or databases, and synthesizes the results. This cycle is optimized for retrieval completeness, not for the semantic precision of the request.
So when the query is ambiguous, for example
Related articles
Discover more articles exploring similar topics, selected to offer you a more complete and stimulating perspective. Each piece of content is carefully chosen to enrich your experience.
