Process automation gets scoped from the wrong document. Someone draws the process, everyone in the room agrees the drawing is correct, and the project is priced on it. That drawing describes how the work is supposed to go. The cost, the risk and most of the real value sit in the cases it leaves out. So the first thing I ask for now is not the process. It is the last twenty cases that went wrong.
Why is the process diagram the wrong document?
A documented process is a consensus artefact. It exists because several people had to agree on a description, and a description several people can agree on is, almost by definition, the version with the mess taken out. The operations manager recognises her part of it. The finance lead recognises his. Nobody volunteers the case that only works because one colleague on the second floor checks it every Thursday afternoon, partly because it does not feel like part of the process, and partly because nobody has ever asked.
I do not think this is dishonesty. It is simply what happens when you ask a company to describe itself. Across several hundred first conversations with Dutch and Flemish business owners, I have never once been handed a process description that included its own exceptions. Not once. The exceptions are not being hidden. They are just not the sort of thing a process document is written for.
That would be a harmless observation if the document were not load-bearing. A quote is a prediction, and in most automation projects this diagram is the only evidence the prediction is built on. So you end up pricing the version of the work that happens when nothing unusual happens, and discovering the rest of it later, at somebody's expense.
What does Dutch construction law already know about this?
There is a trade that settled this argument long before software existed, and it is not ours. In Dutch construction, the bestek describes the work as agreed. Everyone in bouw also knows that the margin, and most of the arguing, lives in the meerwerk: the additional work that turns up once the walls are open and somebody sees what is actually behind them.
Dutch law does not leave that to goodwill. Under article 7:755 of the Burgerlijk Wetboek, a contractor may only claim a price increase for additions or changes the client wants if he warned the client in time that a price increase would be necessary, unless the client should have understood that necessity himself. In practice the provision cannot be contracted away to the client's disadvantage. The Hoge Raad clarified in July 2022 that the warning does not have to state how large the increase will be. Once warned, it is the client's turn to ask.
Read that as a professional standard rather than as a statute and it says something useful about our trade. The burden of naming the foreseeable extra sits with the professional, before the work starts, not with the client afterwards. A supplier who prices a flowchart and then discovers the exceptions in month three has done precisely the thing that article exists to prevent.
I should be honest about how far the parallel travels. Nobody is going to sue an AI supplier under 7:755. It is Dutch contracting law, written for building work, and Belgium runs its own regime for the same problem. It is a standard I hold myself to rather than one I am held to. I have still not found a better description of what a first conversation is for.
Where do automation projects actually break?
Not on the model, in my experience. On the cases the demonstration never saw.
There is a piece of research worth knowing here, mostly because of where the idea came from. In June 2024 a team at Sierra published a benchmark called tau-bench, for agents that hold a conversation with a user and call tools under a written policy. Its lasting contribution was less the benchmark than a metric: pass^k, the probability that an agent succeeds on all k attempts at the same task rather than on one of them. The paper reported that the strong function-calling models of the day completed under 50 percent of tasks, and that consistency was worse still, with pass^8 below 25 percent in the retail domain.
Frontier models score far better than that today, and anyone quoting those 2024 numbers as current is being careless with them. The metric is the part that aged well. A demonstration is pass^1 on a case somebody chose. What a business needs is the same task, eight times, on cases nobody chose. The distance between those two things is the exception list, and it never appears on a slide.
This is also why the demo-to-production gap is not a technical surprise so much as a scoping one. I wrote separately about what changes between a demo and a production system; almost everything on that list is a case the demo was never asked to handle.
The three kinds of exception, and what each one costs
Once you have a list, the useful move is to stop treating it as a list. Exceptions are not one category. The three that matter behave completely differently, and the mistake that costs the most money is handling all three the same way.
Rare and inexpensive
Something happens twice a year and costs an hour of somebody's time when it does. Do not build for it. Do not let it into the scope. Let it fail where a person can see it, and write down that this is the intended behaviour. The work here is naming it, not solving it. A named exception with an agreed owner is finished business. An unnamed one comes back as a bug report in month four, and by then it is an argument about whether it was in scope.
Rare and expensive
Something happens twice a year and, when it does, it goes out of the door to a customer, into a tax return, or onto a patient record. Frequency is the wrong axis here, because the cost is not proportional to how often the case turns up. These have to be designed for while the system is being built: a rule that recognises the case, a route to a person, and somebody whose actual job it is to be on the other end of that route. This is also the bucket clients most want to defer, precisely because it is rare. Deferring it is how a well-built system eventually produces one very expensive afternoon.
Frequent and invisible
And then the one that is usually the real project. Something happens most weeks, and nobody calls it an exception, because someone has always simply dealt with it. The supplier who sends two documents for one delivery. The customer whose reference number never matches. The order that always arrives by telephone. The field that is technically optional and is filled in by hand every time. These cases are invisible exactly because they are frequent enough to have been absorbed into somebody's routine, and they are the ones that decide whether the automation is worth anything at all.
When a list arrives, that third bucket is mostly what I am hunting for. It is the only one where the exception is not an edge case at all. It is the process, wearing a different coat. The invoice work we described in automating invoice processing is the clearest example I know: the business case there is not the model, it is the count of how many invoices do not arrive the way the diagram says they do.
How do you scope an AI project from the exception list?
This is a concrete thing you can do this week, with no supplier in the room. It takes a morning and it is the least expensive part of the whole project.
- Ask for the last twenty cases that went wrong, or that somebody had to touch by hand. Not a list of exception types. The actual cases, with dates, from the people who handled them.
- Work out what share of the last hundred those twenty represent. If nobody can say, that is not a delay in the project. That is the first finding, and it is worth more than the flowchart.
- Sort each case into rare and inexpensive, rare and expensive, or frequent and invisible. Disagreements about which bucket a case belongs in are the most productive half hour you will spend.
- For each bucket, write one sentence saying what the system does: handles it, refuses it, or routes it to a named person. Refusing is a feature, not a gap in the build.
- Price the routing, not only the happy path. The route to a human being is a real piece of the build. It has a screen, a queue, a notification and an expected response time, and none of that is free.
If the list cannot be produced at all, that is an answer as well. The process is not ready to be automated, and neither is the quote. Establishing that costs a morning. Establishing it in month three costs a project, and usually a working relationship. It also tends to redirect the conversation towards which process to automate first, which is a better question than the one most companies walk in with.
What percentage of cases does the system have to handle?
Eighty percent is the figure almost everyone reaches for, and it is comfortable because it commits to nothing. It says precisely nothing about the other twenty.
The better question is what the system does when it does not know. A system that handles seventy percent of cases and reliably declines the rest is worth more than one that handles ninety-two percent and guesses at the remainder. The first one's failures arrive in a queue with a name on it. The second one's arrive in a customer email, weeks later, with nobody able to say how many others went the same way. Declining is recoverable. An unnoticed wrong answer is the expensive kind, and it stays expensive long after the project closed.
Which means the percentage is the wrong number to negotiate. Negotiate the refusal rule instead: what makes the system hand a case back, and to whom. How much error a process can carry at all is a separate question, and I worked through it in how wrong AI automation is allowed to be. The two answers have to agree with each other, and in most proposals neither one is written down.
Where I have got this wrong
I would rather not write this section, which is usually the sign that it belongs in the piece.
We have scoped from the flowchart. More than once. The tell always has the same shape. A system works well in week one, and by week six it is generating a slow trickle of messages that begin with "can you just have a look at this one". That trickle is the exception list arriving late, one case at a time. By then somebody is absorbing the cost of it: either us, at our own expense, which is survivable and is what a fixed price is for, or the client, which is worse, because they now believe the system is unreliable when what was actually unreliable was the scope.
So the exception list moved. It is no longer something we find during the project. It is something we ask for before there is a price at all, and a proposal that cannot say what happens to the odd cases is not a finished proposal, whatever else is in it.
What goes in the proposal now
Three things, in writing, before there is a number:
- The exception classes, named. Not "edge cases will be handled", which means nothing and is unenforceable. The actual cases, sorted into the three buckets.
- What the system does with each one: handles it, refuses it, or routes it.
- Who sits on the other end of the routing, by role, and how quickly they are expected to look at the queue.
This makes proposals longer and first conversations slower. It has certainly cost us work with firms who wanted a number that week. It is a trade I would make again, because the alternative is putting a fixed price on a description of the work that was written to be agreed with rather than to be true. What an AI project costs and how we run an implementation are both downstream of this. The exception list is the thing that makes either of them honest.
So if you are about to buy automation, do not send the supplier your process description. Send them the twenty cases that went wrong, and watch what they do with it. The ones who ask for more of them are the ones worth talking to.
