AI ROI Scorecard: Measuring the Real Value of Agents

The problem the scorecard aims to solve

For years, the ROI of artificial intelligence has been measured with approximations. Companies counted saved hours, estimated avoided costs, and multiplied by the average hourly rate. The result was a reassuring number, often far from operational reality.

However, with the spread of Autonomous AI agents, this approach becomes dangerous. An agent working autonomously on critical processes—from order management to customer care—cannot be evaluated solely on theoretical time saved. Therefore, a metric is needed that measures the actual value produced.

Sarah Friar, CFO of OpenAI, addressed this issue directly. In a paper published on OpenAI's official website, she introduced a Practical scorecard for the AI era. The framework is structured into four dimensions. Each answers a specific question about the value generated by artificial intelligence in real business contexts.

The Four Dimensions: Framework Architecture

First of all, it's useful to understand the overall logic. The four metrics are not independent. They form a coherent system that measures AI along the axes of value produced, cost incurred, reliability, and computational efficiency.

1. Useful Work: the work that really matters

The first dimension is the useful work. It’s not about measuring how many requests the AI has processed, but how many it has completed with a result that’s useful for the business. In fact, a chatbot that answers 1,000 questions but resolves only 200 problems has a useful work rate of 20%.

This metric requires companies to define in advance what «useful completion» means. For example, in a lead generation context, a useful task could be correct contact qualification. In a technical support context, it could be the autonomous resolution of a ticket without human escalation.

Consequently, useful work compels marketing and operations teams to clarify objectives before implementing any AI solution. It's an exercise in strategic clarity, not just technical. For this reason, we at SHM Studio we consider it the starting point of any project digital marketing with AI component.

2. Cost per Successful Task: The Real Cost of Autonomy

The second dimension is the cost per successful task. This metric calculates how much each successfully completed task costs the company. Not the total cost of the AI system, but the unit cost of the useful result.

However, the calculation is less intuitive than it seems. It includes the cost of APIs or the model, the cost of failure (uncompleted tasks requiring human intervention), the cost of supervision, and the cost of error correction. Therefore, a seemingly cheap system can turn out to be expensive if the failure rate is high.

According to recent research from McKinsey on the economic potential of generative AI, companies that measure the cost per useful output—rather than the total cost of the system—gain a more accurate view of the actual ROI. Therefore, this metric is not merely theoretical: it has a direct impact on budgeting decisions.

3. Dependability: Reliability as a Strategic Asset

The third dimension is the dependability. Measures the AI's ability to behave predictably and reliably over time. It is not enough for the system to perform well on average. It must also perform well in edge cases, during peak loads, and in contexts not foreseen during training.

Despite this, dependability is often the most overlooked dimension in preliminary evaluations. Demos always work. Production systems, however, encounter real variability. In fact, an AI agent autonomously managing Google Ads campaigns must be dependable even during peak seasons, not just during ordinary periods.

For Italian companies operating in sectors with strong seasonality—retail, tourism, food—this dimension is critical. We at SHM Studio we systematically evaluate it in projects that integrate AI into Google Ads campaigns and in the LinkedIn campaign.

4. Return on Compute: Infrastructure Efficiency

The fourth dimension is the ROCI. Measure the value produced per unit of computational power employed. It's a more technical metric, but also relevant for SMEs using pay-as-you-go cloud services.

Furthermore, with the constantly evolving costs of AI models, the return on compute becomes an indicator of economic sustainability in the medium term. A system that provides adequate value today could become inefficient if computational costs increase or if more efficient models become available.

Furthermore, this metric incentivizes companies to choose the right model for the right task. The most powerful model is not always the most efficient. For example, for repetitive and well-defined tasks, lighter models can offer a significantly higher return on compute.

Practical applicability for Italian SMEs

The Friar framework was born in an enterprise context. However, its logic is also applicable to Italian SMEs and the mid-market, with some adaptations.

In particular, SMEs rarely have granular monitoring systems. Therefore, implementing the scorecard requires an upfront investment in data infrastructure. Without structured AI task logs, outcome tracking, and alerting systems, the four metrics remain inaccessible.

However, this does not mean the scorecard is useless for SMEs. On the contrary, it can guide the choice of AI tools from the outset. For example, a company evaluating a chatbot for customer support should ask the vendor: how do I measure useful work? How do I calculate the cost per successful task? These are the correct selection criteria, not the number of available integrations or the graphical interface.

According to Gartner AI Trends Report, by 2027, more than 60% of mid-market companies will adopt structured frameworks for measuring AI ROI. As a result, those who begin building this measurement capability now will gain a tangible competitive advantage.

For marketing managers, the framework translates into immediate operational questions. How many of the AI automations active today produce measurable useful work? What is the cost per qualified lead generated with AI support compared to the manual process? SEO strategy Does AI-powered content produce superior performance? These questions drive more robust budget decisions.

Trade-offs to consider before adopting the framework

OpenAI's scorecard is a powerful tool. However, it presents some trade-offs that decision-makers must consider before adoption.

The first trade-off concerns the implementation complexity. Measuring the four dimensions requires technical infrastructure, logging processes, and analytical skills. For less structured organizations, the cost of implementing the measurement system may, in the short term, outweigh the informational benefits obtained.

The second trade-off concerns the definition of «success». Useful work and cost per successful task depend on a clear definition of what constitutes a successful outcome. In complex contexts—such as the SEO content production or multichannel campaign management — this definition is often ambiguous and internally contested.

The third trade-off concerns the local optimization risk. Optimizing the four metrics individually can lead to suboptimal system-level behavior. For example, increasing dependability by reducing edge cases handled by AI can lower the overall useful work. Therefore, the metrics should be interpreted in an integrated, not isolated, manner.

Finally, the framework quantifies the value of AI. However, some AI benefits—such as the perceived quality of interactions or brand consistency in communication—are difficult to quantify with these metrics. Therefore, the scorecard should be supplemented with qualitative assessments.

What the metrics don't tell you

There is a dimension that the Friar framework does not explicitly capture: the cost of inertia. Companies that don't adopt AI—or adopt it without measuring it—do not have zero cost. They have an increasing opportunity cost.

According to Harvard Business Review, organizations that develop structured AI measurement capabilities achieve 2.5 times higher returns than those that adopt AI without an evaluation framework. Therefore, the scorecard is not just a control tool. It is an enabler of organizational learning.

Furthermore, the framework pushes companies to build a measurement culture around AI. This culture is transferable: teams that learn to measure the ROI of an AI agent apply the same logic to subsequent projects, reducing learning cycles and accelerating time-to-value.

For marketing and digital managers in Italian companies, this means that investing in the measurement framework today—even if AI is still in its experimental phase—builds an organizational competence that will become increasingly relevant in the coming years. The services of web development and of AI integration that we follow at SHM Studio always start from this premise: measure first, then scale.

The recommended decision: where to start

For Italian companies looking to adopt OpenAI's scorecard, we suggest a three-phase approach.

• Phase 1 — Mapping: Identify the AI processes that are already active and define for each what constitutes a «successfully completed task.» This exercise requires the involvement of both the technical and business teams.
• Phase 2 — Instrumentation: Implement structured logging of AI outcomes. Without data, metrics remain theoretical. Even simple solutions – shared spreadsheets, Looker Studio dashboards – are a valid starting point.
• Phase 3 - Periodic Review: Establish a review cadence for the four metrics. Monthly for high-volume systems, quarterly for more stable systems. The scorecard is only useful if it's read and acts as a decision-making input.

Therefore, adopting the framework does not necessarily require a large initial investment. It requires methodological discipline and clarity of objectives. These are prerequisites that any organization can develop, regardless of size.

To further explore how to apply these principles to projects digital marketing e artificial intelligence from your company, the SHM Studio team is available for an initial consultation. You can contact us through the Contact Us to explore other insights in SHM Studio Blog.