A new benchmark called DiscoBench examined AI search agents’ performance on ambiguous queries. The result is surprising: the problem isn’t the ability to search for information, but the inability to ask the user for clarification when the question is vague. Therefore, models that repeat searches instead of following up achieve an accuracy of just 51.9%—worse than those that simply guess.
Furthermore, the best model tested achieves an overall accuracy of only 43%. However, when ambiguity is removed from the queries, accuracy jumps by as much as 40 percentage points. This finding radically changes the perspective on how to evaluate and integrate AI agents into business workflows. In particular, organizations adopting AI-based solutions for search, content, or customer intelligence need to rethink the design of their queries and workflows.
We of SHM Studio We closely follow the evolution of AI agents applied to digital marketing. Consequently, we believe these data have concrete implications for Italian marketing managers who are evaluating or already using AI tools in their processes. In this analysis, we delve into the problem architecture, practical use cases, and operational guidelines.
The benchmark that changes the narrative on AI agents
For months, the debate around AI search agents has focused on the quality of retrieved sources. However, recent research shifts the focus elsewhere. The benchmark DiscoBench, analyzed by The Decoder, demonstrates that the real bottleneck is not information retrieval. It is managing ambiguity in the initial query.
In practice, when a user makes a vague or ambiguous request, the AI agent should ask for clarification. Instead, most models tend to repeat searches, hoping to converge on an answer. This approach proves counterproductive: models that repeat searches achieve an accuracy of 51.9%. In contrast, those that simply guess perform better.
So, the problem isn't computational. It's communicative. And this distinction has direct consequences for those integrating AI agents into marketing and business intelligence processes.
Problem Architecture: Why the Agent Doesn't Ask
To understand the phenomenon, it's useful to examine how a typical AI search agent works. The model receives a query, generates a search plan, makes calls to engines or databases, and synthesizes the results. This cycle is optimized for completeness of recovery, not for the semantic precision of the request.
Therefore, when the query is ambiguous, for example
Related articles
Discover other articles that explore similar topics in depth, selected to give you a more complete and stimulating view. Each piece of content is carefully chosen to enrich your experience.
