AI Won't Measure the Role You Couldn't Measure Before

This post is also available in German: Was du vorher nicht messen konntest, misst auch AI nicht

About fifteen years ago I worked on enterprise search projects, and the business cases back then looked a lot like the AI business cases I see today. People spend so many minutes a day looking for documents, you multiply that by headcount and an hourly rate, and you end up with savings in the hundreds of thousands. The projects went ahead on that basis. A few months after one of them went live, a C-level executive at the customer asked me, dripping with sarcasm, where exactly those hundreds of thousands were now, since they didn’t show up anywhere in the company’s numbers. People probably did find their documents faster, but nobody had decided what should happen with the minutes they saved, so the minutes went back into the day and never reached a line anybody could read.

That question is back, this time about AI, and for some of us the day it gets asked has already come. I think it’s a completely legitimate thing to ask. Somebody signs off a budget, licences, a few months of people’s time, and wants to know what came back. Nobody should be annoyed by that, least of all the people who sell or build the thing.

What I’ve noticed is that the question lands very differently depending on the role it’s asked about, and that’s what this post is about. Some roles in a company have always been measured in numbers. For others, nobody ever managed to say what the impact is, and the honest reason is that it can’t be said in numbers. My claim is simple, and I’ll say it bluntly: if a role can’t be measured today, AI won’t make it measurable. It doesn’t work that way, and pretending otherwise sets up a project to fail on the day the number is due.

Some roles have a number, and some roles never had one

Think of a support team. Tickets come in, tickets get closed, there’s a time to resolution and a backlog, and everybody agrees these are the right things to count. Or a team that processes invoices: documents per hour, error rate, cost per document. If you give those people an AI assistant, you can measure before and after with the same ruler you used all along. The ROI conversation is uncomfortable at times, but it’s a normal one.

Then there’s the other kind of role, and every company has plenty of them. The person who holds a department together. The enterprise architect whose best work is the project that never went wrong. The internal communications lead, the community manager, the colleague everyone calls when something is unclear. Ask a manager what the impact of such a role is and you’ll get sentences like “things run smoothly” or “we’d notice if it wasn’t there”. I’m not saying these roles are worth less. Quite often it’s the opposite. But their impact has never been broken down into numbers, because nobody found numbers that capture it without distorting it.

My football brain has a comparison for this. Look at who wins the Ballon d’Or. The last defender to get it was Fabio Cannavaro in 2006, and most years since then it has gone to whoever scored the goals. Everybody sees the goals, and people who watch closely also see what a good centre-back does. (Sorry for dragging football into this, but as a trained “Vorstopper”, the man-marking centre-back German football had before the position quietly died out, I felt exactly like that for years, admittedly never at a level where anyone was watching.) But every team has a number 6 or a number 8 whose work you only really notice the week they’re injured or suspended, when suddenly nothing connects any more. Rodri is the exception that makes the point rather than breaking it. He won the Ballon d’Or in 2024 as a defensive midfielder, a few weeks after tearing his cruciate ligament, and Manchester City went through one of the worst stretches of the Guardiola years without him. Most of the roles I mean are that number 6, and nobody hands them an award first.

AI is not a measuring instrument

Here’s the assumption I keep running into, often unspoken: once AI is part of the picture, the fuzzy role will somehow become countable. Copilot leaves traces, after all. There are prompts, the Copilot usage report in the admin center, minutes saved according to a survey. So surely the impact will finally show up in a dashboard.

It won’t, and the reason is almost boring. The usage report tells you that something was used, not what it was worth. “Saved twenty minutes per week” is a statement about time, and it only turns into money if somebody can say what those twenty minutes would otherwise have produced. For a ticket queue that’s answerable. For the architect it isn’t, and it wasn’t the day before Copilot arrived either. AI changes how work gets done. It doesn’t change what the organisation knows about the value of that work. If the yardstick didn’t exist before, the project has to bring its own, and the project is a bad place to invent one, because by then everybody has a stake in the answer.

There’s a second effect that makes it worse. Once a role has an AI project attached, the numbers that do exist get all the attention. The measurable part of the job, which was maybe a third of it, is suddenly the whole story, because it’s the only part that fits on a slide. Once that happens, people have every reason to work toward what’s counted.

On paper, the bar is surprisingly low

At Solvion we once sat down with our tax advisor and did the simple math. Take the full cost of an employee in Austria, everything the employer pays and not just the gross salary, put it next to the price of a Microsoft 365 Copilot licence, which was €28.10 per user and month at list price when we did the calculation in 2025, and ask how many minutes a week the licence has to save for the two to cancel out.

For someone earning €2,000 gross a month it’s about 19 minutes a week. At €4,000 it’s under ten, and at €7,000 it’s five and a half. Put against a 40-hour week, the licence has to give back somewhere between 0.2 and 0.8 percent of a person’s working time, and I don’t know many tools that would struggle with that if people actually use them.

To make that less abstract, I looked at my own calendar. Viva Insights counts 111 hours of meeting time for me in the four weeks up to the start of November.

Some of that is travel and a few private blocks, but even what’s left is enough that a Copilot meeting recap wouldn’t have to be particularly good to save me nineteen minutes a week, the worst case on the chart. So meeting summaries might genuinely help me, as it turns out. Whether I’d put the time they save into anything other than a cappuccino is a different question, and it’s the one the chart can’t answer.

That’s what the simple math says, and it isn’t quite that simple, because the chart assumes that every saved minute is a minute of paid work that turns into something else, which is exactly the step the search business cases skipped.

I’m not the only one who has noticed the gap between saved time and anything you can book. The UK Department for Business and Trade gave 1,000 Copilot licences to its staff at the end of 2024 and published its evaluation in 2025. People saved time, mostly on written work, some other tasks took longer than before, and the evaluators found no evidence that the time savings had turned into measurable productivity. In Denmark, Anders Humlum and Emilie Vestergaard linked surveys on chatbot use to the country’s administrative labour market records and found precise null effects on earnings and recorded hours two years after ChatGPT came out. What did change was the structure of work, with new tasks like checking and integrating AI output, so the minutes went into work that nobody had put in a business case, which is the search story all over again, only measured properly this time.

In some of our own customer conversations the pattern has a name by now, the cappuccino use case. It’s the assistant that drafts a meeting summary or polishes an email, and it genuinely saves a few minutes, roughly enough to walk over and get a coffee. I don’t want to talk anyone out of their coffee. But if that’s all a use case does, there’s no impact to report, and adding up a thousand coffee breaks on a slide doesn’t turn them into one.

The sociologist Hartmut Rosa has a good explanation for where those minutes go, and he wrote it long before anyone talked about generative AI. In Social Acceleration he asks why we feel more rushed every year although technology keeps saving us time. His example is email, which is far quicker to write than a letter, except that we now write so many more of them that the time we saved is gone several times over. As long as the amount of work grows faster than the speed at which we do it, every gain in speed is felt as more pressure rather than less, and a cappuccino use case can turn into three more meeting summaries nobody asked for before anyone has finished the coffee.

On the other side, I wouldn’t use that as a reason to give up on the numbers. Seen as a share of what an employee costs, the gain AI has to deliver at today’s licence prices isn’t outrageous. With thousands of seats it’s a different conversation, because nobody can follow what thousands of people do with their minutes, and the cappuccinos add up quietly. Where a company can actually accompany people while they change how they work, it looks very different. That accompaniment costs money as well and belongs in the calculation, but it’s also the part that decides whether the minutes go into a coffee or into the work.

The question tells you something, so use it

Seen from the other side, I find the ROI question useful, and not only as a hurdle. A company that gets asked “what does this role contribute, in numbers?” and can’t answer has just learned something about itself, long before any AI is involved. Either the role has an impact that nobody has described properly, or it has less impact than everyone assumed, or the impact is real but sits in places a spreadsheet doesn’t reach. All three are worth knowing, and none of them is the AI’s job to find out.

So the order matters. If the plan is to give a role AI support and later justify it with a number, the numbers have to be agreed on first, with the people who do the work, and ideally while the old way of working is still around to compare against. That’s unglamorous work: sitting down and asking what a good month looks like in this role, what a bad one looks like, and who would notice the difference. Sometimes the answer turns out to be a number. Quite often it turns out to be a judgement, and then the right thing to do is to say so, instead of dressing a judgement up as a metric.

What I’d do instead

For roles that were never measured in numbers, I’d stop pretending the ROI question has a number-shaped answer, and I’d answer it in a different form. Say what the AI is supposed to change in the work, in plain words, before it starts. Pick what you’d expect to see if it works, and be clear about whether that’s something you can count or something a person who knows the role will recognise. Then have that person look again after a few months, with the same attention you’d give a budget review. That’s a weaker answer than a percentage, and it’s an honest one, which counts for more when the next budget round comes around.

For the roles that are measurable, nothing changes, and I’d happily hold those projects to their numbers. That’s where AI should have to prove itself.

What this means for people who decide

If you’re the one who gets asked for the ROI, the uncomfortable part is that you can’t outsource the answer to a tool. The tool can make people faster, and it can leave traces. Whether faster is better for this particular role is a judgement that somebody has to make, and the clearer the role was before, the easier it’ll be. If you’re the one asking, I’d ask the same question about the role you’re not using AI for, and see how well it holds up. I’d bet the answer to that tells you more than any dashboard from the AI project will.

And if you’re the one rolling it out, keep an eye on the people. If Rosa is right, the risk isn’t only that Copilot saves too little time, it’s that whatever it saves gets refilled with more of the same, and the people in those number-6 roles tend to be the first to feel it and the last to say so.