AI ROI scorecard: measuring the real value of agents

The problem the scorecard aims to solve

For years, the ROI of artificial intelligence has been measured with rough approximations. Companies counted hours saved, estimated costs avoided and multiplied by the average hourly rate. The result was a reassuring number, often far from operational reality.

But with the rise of autonomous AI agents, this approach becomes risky. An agent working independently on critical processes, from order management to customer care, can't be judged only on theoretical time saved. So you need a metric that measures the value actually produced.

Sarah Friar, CFO of OpenAI, tackled this problem head-on. In a document published on OpenAI's official website, she introduced a practical scorecard for the AI era. The framework is built around four dimensions. Each one answers a specific question about the value artificial intelligence generates in real business settings.

The four dimensions: how the framework is built

First, it helps to understand the overall logic. The four metrics aren't independent. Together they form a coherent system that measures AI along the lines of value produced, cost incurred, reliability and computational efficiency.

1. Useful Work: the work that really matters

The first dimension is useful work. It's not about how many requests the AI has processed, but how many it has completed with a useful outcome for the business. In fact, a chatbot that answers a thousand questions but solves only two hundred problems has a useful work rate of 20%.

This metric forces companies to define up front what a “useful completion” means. For example, in lead generation, a useful task might be correctly qualifying a contact. In technical support, it might be resolving a ticket autonomously, with no human escalation.

As a consequence, useful work requires marketing and operations teams to clarify their goals before implementing any AI solution. It's an exercise in strategic clarity, not just technical rigor. That's why we at SHM Studio see it as the starting point of any digital marketing project with an AI component.

2. Cost per Successful Task: the real cost of autonomy

The second dimension is cost per successful task. This metric calculates how much each successfully completed task costs the company. Not the total cost of the AI system, but the unit cost of a useful result.

However, the calculation is less intuitive than it seems. It includes the cost of APIs or the model, the cost of failure (tasks left unfinished that need human intervention), the cost of supervision and the cost of fixing errors. So a system that looks cheap can turn out to be expensive if its failure rate is high.

According to recent research by McKinsey on the economic potential of generative AI, companies that measure the cost per useful output, rather than the total cost of the system, get a 40-60% more accurate view of actual ROI. In other words, the metric isn't just theory: it directly affects budget decisions.

3. Dependability: reliability as a strategic asset

The third dimension is dependability. It measures how predictably and reliably AI behaves over time. It's not enough for the system to perform well on average. It has to perform well in edge cases, during load peaks and in situations it never saw in training.

Even so, dependability is often the most overlooked dimension in early evaluations. Demos always work. Production systems, on the other hand, run into real-world variability. For example, an AI agent that manages Google Ads campaigns autonomously has to be reliable during peak season too, not just in ordinary periods.

For Italian companies in highly seasonal sectors, such as retail, tourism and food, this dimension is critical. At SHM Studio, we assess it systematically in projects that bring AI into Google Ads campaigns and LinkedIn campaigns.

4. Return on Compute: infrastructure efficiency

The fourth dimension is return on compute. It measures the value produced for each unit of computing power used. It's a more technical metric, but it matters for SMEs too if they use pay-as-you-go cloud services.

What's more, with AI model costs constantly changing, return on compute becomes an indicator of economic sustainability over the medium term. A system that delivers adequate value today could become inefficient if computing costs rise or more efficient models become available.

This metric also pushes companies to choose the right model for the right task. The most powerful model isn't always the most efficient. For example, for repetitive, well-defined tasks, lighter models can offer a significantly higher return on compute.

How it applies in practice to Italian SMEs

Friar's framework was born in an enterprise context. Still, its logic applies to Italian SMEs and the mid-market too, with a few adaptations.

In particular, SMEs rarely have granular monitoring systems. So implementing the scorecard requires an upfront investment in data infrastructure. Without structured logs of AI tasks, outcome tracking and alerting systems, the four metrics stay out of reach.

That doesn't mean the scorecard is useless for SMEs. On the contrary, it can guide the choice of AI tools from the very start. For example, a company evaluating a chatbot for customer support should ask the vendor: How do I measure useful work? How do I calculate cost per successful task? These are the right selection criteria, not the number of available integrations or the look of the interface.

According to the Gartner AI Trends Report, by 2027 more than 60% of mid-market companies will adopt structured frameworks for measuring AI ROI. As a result, those who start building this measurement capability now gain a real competitive edge.

For marketing managers, the framework translates into immediate, practical questions. How many of the AI automations running today produce measurable useful work? What is the cost per qualified lead generated with AI support compared with the manual process? Does the AI-supported SEO strategy produce higher-performing content? Questions like these lead to sounder budget decisions.

Trade-offs to consider before adopting the framework

OpenAI's scorecard is a powerful tool. But it comes with a few trade-offs that decision makers should weigh before adopting it.

The first trade-off is implementation complexity. Measuring the four dimensions takes technical infrastructure, logging processes and analytical skills. For less structured organizations, the cost of setting up the measurement system can outweigh, in the short term, the insights it delivers.

The second trade-off is how you define "success". Useful work and cost per successful task depend on a clear definition of what counts as a positive outcome. In complex contexts, such as SEO content production or multichannel campaign management, that definition is often ambiguous and disputed internally.

The third trade-off is the risk of local optimization. Optimizing the four metrics one by one can lead to suboptimal behavior at the system level. For example, increasing dependability by reducing the edge cases handled by AI can lower overall useful work. So the metrics need to be read together, not in isolation.

Finally, the framework measures the value of AI quantitatively. Yet some benefits of AI, such as the perceived quality of interactions or brand consistency in communication, are hard to quantify with these metrics. That's why the scorecard should be complemented by qualitative assessments.

What the metrics don't tell you

There is one dimension Friar's framework doesn't explicitly capture: the cost of inertia. Companies that don't adopt AI, or adopt it without measuring it, don't have a cost of zero. They have a growing opportunity cost.

According to Harvard Business Review, organizations that build structured AI measurement capabilities see returns 2.5 times higher than those that adopt AI without an evaluation framework. So the scorecard isn't just a control tool. It's an enabler of organizational learning.

The framework also pushes companies to build a culture of measurement around AI. That culture is transferable: teams that learn to measure the ROI of one AI agent apply the same logic to later projects, shortening learning cycles and speeding up time-to-value.

For marketing and digital managers at Italian companies, this means that investing in a measurement framework today, even while AI is still experimental, builds an organizational skill that will matter more and more in the years ahead. The web development and AI integration services we deliver at SHM Studio always start from this premise: measure first, scale later.

Our recommendation: where to start

For Italian companies that want to adopt OpenAI's scorecard, we suggest a three-phase path.

• Phase 1 — Mapping: identify the AI processes already in place and define, for each one, what counts as a "successfully completed task." This exercise needs input from both the technical team and the business team.
• Phase 2 — Instrumentation: set up structured logging of AI outcomes. Without data, metrics stay theoretical. Even simple solutions, like shared spreadsheets or Looker Studio dashboards, are a valid starting point.
• Phase 3 — Periodic review: set a regular cadence for reviewing the four metrics. Monthly for high-volume systems, quarterly for more stable ones. The scorecard is only useful if it's read and acts as an input to decisions.

So adopting the framework doesn't necessarily require a big upfront investment. It requires methodological discipline and clarity about your goals. These are prerequisites any organization can develop, whatever its size.

To explore how to apply these principles to your company's digital marketing and artificial intelligence projects, the SHM Studio team is available for an initial consultation. You can reach us through the contact page or explore more insights on the SHM Studio blog.