Stop an AI project when the number you agreed to hit is missed on the date you agreed to check it. Not when the demo looks worse, not when the budget is gone, and not when someone senior finally runs out of patience. The projects I have watched go wrong rarely lacked a warning. They lacked a rule, agreed in advance, for what happens when the warning arrives. Write that rule before the first sprint.
Why do AI projects keep running after they should have stopped?
Because starting an AI project is a decision, and stopping one is an admission. Everyone in the room signed up for something at the start: the sponsor made a case to a board, the team gave up other work, a supplier forecast a result. By week eight each of them has a personal reason to read the data kindly. Nobody is lying. The metric is moving a little, the data problems are “nearly solved”, the next release will “change the picture”. Every sentence is plausible, and the sum of them is a project that has been unhealthy for a month.
I have sat on both sides of that table. As the supplier I know the pull: the work is interesting, the client is friendly, and a stop feels like failing. That is exactly why I no longer trust myself, or anyone else, to make the stop decision in the moment. A decision taken while the team is tired, the sponsor is exposed and an invoice is open is a negotiation, not a judgement.
The fix is dull. Decide the conditions for stopping while everyone is optimistic and nothing has been spent. I call them kill criteria, and in my own proposals they sit on the first page rather than in an appendix. In “How I know a project will fail before it starts” I wrote about the warning signs I hear in a first conversation. Kill criteria are what you do about the ones that only show up later, once real data and real users are involved.
What can the Fyra teach about stopping a project in time?
Anyone who grew up in the Netherlands or Flanders has an example of a project that carried on past the point of sense, and mine is the Fyra. It was meant to be the fast service between Amsterdam and Brussels, run by NS International (a joint venture of NS and KLM) together with the Belgian railway NMBS/SNCB, using V250 trains from AnsaldoBreda: nineteen were ordered and nine delivered. The service to Antwerp and Brussels began on 9 December 2012. It was suspended on 17 January 2013 after technical problems, and on 31 May 2013 the Belgian side cancelled its contract for three trains and withdrew from the service. NS said on 3 June 2013 that the V250 was no longer an option. Those dates come from the Fyra article on Wikipedia, which cites the contemporary press.
What interests me is not the trains but the sequence. As International Railway Journal summarised the final report of the Dutch parliamentary committee, NS knew that the V250 timetable was unrealistic and had concerns about quality, but did not cancel the contract, and it went ahead with little experience of the fleet and no contingency plans. The warnings existed. What that summary does not describe is an agreed point at which the warnings had to turn into a decision.
I should be honest about how far this travels. A train is not a language model; a railway has regulators and a timetable the public can see, and I am not claiming to know what was said in those meeting rooms. This is also a Dutch and Belgian story with two operators and two parliaments, so I would not generalise about either country's habits from it. But the mechanism is one I recognise from far smaller projects: a known concern, a launch date everyone has already announced, and a contract that makes stopping feel more expensive than continuing. That last part is my own pattern, not something the Fyra sources say. In an AI project at the scale of a Dutch SME (mkb) the numbers are smaller and the structure is identical.
What are the kill criteria I agree before an AI project starts?
I use five. None of them is clever. Their value is that they are written down and dated before anyone has a reason to argue about them.
A number on real work, checked on a date
Not “improve customer service”, but “the draft goes out unchanged in at least a set number of cases out of ten, measured on last quarter’s real tickets”. The baseline comes from real work, the checkpoint date goes in the calendar, and the number is one the client would genuinely pay for. If you are unsure how to choose it, the article “What to measure in an AI pilot” goes through the options.
A data gate by week three
Most failures I have seen were decided by whether the real data arrived, not by the model: real documents, real exports, access to the real system. So the first criterion is not about AI at all. By a fixed week, real data of the agreed kind is flowing, or we stop and ask why. A project still working on sample data in week five has already told you something.
An owner who shows up
A named person on the client side who can say “this is how we do it here” and who has thirty minutes a week to spare. If the owner changes twice, or never appears, the project does not have one. I treat that as a stop signal, because a pilot without an owner almost never leaves the pilot. The article “Who owns an AI implementation” covers what the role involves.
A cost-per-task ceiling
Every task the system handles costs something to run, and it is easy to prove a result by letting that cost float. I agree a ceiling per task up front, taken from what the manual task costs the client today. A pilot that only works at a price nobody would pay is then a finding, not a surprise at the end.
A mistake nobody would catch
I list the errors the process cannot tolerate and ask whether we would notice them. If the likeliest serious error is one nobody would find for a month, the project needs a check before it needs another feature. If it cannot get one, it stops. Giving an AI agent more room only after it has earned it is a good rule for a single agent, and the same logic applies to whole projects.
One test I apply to all five: could a stranger score them? If a colleague who has never seen the project could read the page and say, with the numbers in front of them, which criteria were met, the criteria are specific enough. If it takes an explanation, they are still opinions, and opinions are the first thing to bend when a project is going badly.
How do I tell a slow start from a project that is going nowhere?
At each checkpoint I ask two questions: what moved since last time, and do we know why? A slow start has a metric that moves in the right direction and an explanation that stays the same. A doomed project has a flat metric and an explanation that keeps changing. In week four the data was dirty. In week six the users did not understand it. In week eight the model needs to be bigger. Each reason is plausible, and together they say that nobody knows.
There are two other tells. The first is scope growing to rescue the result. When the answer to a flat number is a new feature, a new data source or an extra department, the project is being fed, not fixed. The second is checkpoints being renamed: a “go or no-go” becomes a “review”, then a “workshop”, then something nobody minutes.
Two checkpoints in a row with a flat metric and no stable explanation is where I propose that we stop or narrow, not extend. Narrowing is a legitimate third option. Drop to the one document type, one product line or one customer group where the system does work, and apply the criteria again to that smaller scope. Whether you are running a pilot, a proof of concept or an MVP changes what you can narrow, and the article “Pilot, PoC or MVP” explains the difference.
What does a stop meeting actually look like?
Shorter than people expect. I put the criteria on one slide, the measured numbers next to them, and ask the owner to read out which ones were met. No narrative, no “but if you consider”. If two or more are missed, we choose one of three outcomes: stop, narrow, or continue with a changed hypothesis and a new date. Continuing is allowed, but only with something genuinely different, and only once.
I also ask the people who use the system, not just the sponsor, to tell us in their own words whether they would miss it. A pilot that hits its number while the people doing the work quietly route around it has met a criterion, but it has not met the purpose. Those two readings pull apart more often than the dashboards suggest, and the meeting is where I want the gap to show.
Who gets to call the stop: the client or the consultant?
The client, and I write that down. A consultant paid by the day has a financial reason to continue. A consultant paid a fixed price has a reason to declare victory on a narrower definition than the client had in mind. I am not criticising either model, and I describe the incentives of each in “Fixed price versus hourly for AI projects”. Either way, the right to stop sits with the person who owns the business problem.
What I commit to is saying plainly, when a criterion is missed, that it has been missed, including on the days I would prefer a more generous reading. I also want stopping early to be cheap for the client: sprints already worked are paid for, nothing further is owed, no penalty. A client who fears a penalty will not call a stop, and then the whole mechanism is decoration.
What happens after you stop an AI project?
Stopping is cheaper than it looks, because most of what was built is not the model. The process map, the cleaned data, the test set of real examples and the list of exceptions all survive, and they tend to be the expensive part of the next attempt. I ask for a one-page note: what we tried, what the number was, why we stopped, and what we would reuse.
Sometimes the right outcome is simply to do nothing for now. Often it is a smaller project later, and in my experience the second AI project tends to go well, because the first has already paid for the lessons.
Underneath all of it sits the sunk-cost problem. The money already spent is not a reason to continue, and it is not a reason to stop either. The only question at a checkpoint is whether the next sprint is worth buying at today’s price, given what the data now says. If you are about to start a project and want the kill criteria drafted alongside the scope, tell me the scope and we draft them together. It is the kind of work I do as an AI consultant for the mkb.
A consultant tells you where AI pays off; Crux Digits also builds it. A fixed price per step, one named expert, from Utrecht.
AI consultancy in the Netherlands →
