Where AI has to stop in a process

Where does AI have to stop in a process?

At three points. First, when a case is unclear: details are missing, contradict each other or fit no category. Second, before anything leaves the business, such as a reply to a customer, a quote or a payment. Third, wherever an individual case is decided, for example a yes, a no or a price.

At these points, a person takes over. In between, the AI may prepare: sorting, extracting, summarising, writing drafts. We call this pattern “AI with a checkpoint”. It is not a brake. It is what makes it possible to leave work to an automated process in the first place.

Where does the idea of a checkpoint come from?

From the Toyota Production System. Sakichi Toyoda, the founder of the Toyota Group, built a loom that stopped by itself as soon as a thread broke. That way, it did not go on producing defective cloth. And one person could look after several machines instead of watching over each one.

Toyota calls this principle Jidoka: automation with human judgement built in. In Japanese, Toyota writes the word with a character that contains the sign for a person. The machine detects the fault, stops and calls someone. The person then fixes the cause, not just the symptom.

For AI in the office, this means that a process which stops when something is unclear does not send out wrong answers in bulk. Your team does not have to look at every case. It sees the cases that have stopped and approves what goes out.

Why isn’t it enough for the AI to be right most of the time?

Because language models write fluently and convincingly even when they are wrong. You cannot spot an error by its form. An invented order number looks like a real one, a misread amount like a correct one.

Add to that a well-researched effect: people who work with a reliable automated system for a long time gradually check less carefully. The research literature calls this automation bias. Rare errors in particular are easily overlooked as a result.

That is why we don’t rely on the AI being tuned well enough. We define where the process stops. And we build it so that it does not depend on a model’s form on the day.

How does a process know it has to stop?

Through rules written down in advance, which can be checked. Typical triggers: required details are missing, such as an address or a preferred date. Details contradict each other, such as two different quantities in the same enquiry. An amount is above a set limit. Or the case touches on a sensitive matter such as a complaint, a cancellation or a deadline.

What matters is that the AI is allowed to say “I don’t know”. We give it a category of its own for this, instead of forcing it to file every case somewhere. A case in this category stops.

The other checks run as ordinary program code around the AI. Is the required field filled in, is the amount below the limit, is the sender known? Rules like these behave the same way every time. They can be tested and read. So the AI does not decide for itself whether it gets checked.

What does a checkpoint look like in practice?

An example, simplified and not a client project: a service business receives enquiries by email and through a contact form. Both feed into one inbox. The AI assigns the emails, extracts the details and flags what is missing. Unclear cases go straight to a person.

A draft quote is put together from building blocks with current prices. A person checks it and approves it. Only then does the quote go out; the follow-up reminder is set automatically. So the process has two checkpoints: one for anything unclear and one before sending.

To keep approval from becoming a mere click, we design it deliberately. Next to every extracted value is the passage it comes from. Uncertain values are flagged. Whatever the person changes is recorded, as is who approved it. That way, you can trace later why something went out as it did.

What did building CuraNexus teach us about this?

CuraNexus, our own product for medical practices, processes health data. For every feature, we asked which data leaves the system. So the confirmation sent to patients gives the date and time, but not the reason for the visit that they enter when booking. Only licence and status data come to us.

We ask the same question for every AI automation before we connect a service: which data does this step really need, and where is it processed? Often it needs less than you might first think.

A second lesson: check where it matters, not just beforehand. In the CuraNexus online booking, the form shows free times. When a booking is submitted, the server checks once more whether the slot is still free. The same applies to AI: a good prediction does not replace the check at the checkpoint.

What do we clarify before AI gets involved?

Three questions, in writing and before the first line of code. What may AI take on? Typically sorting incoming requests, extracting details from documents, drafting replies and quotes, and making internal knowledge easy to find.

What stays with people? Checking and approving what AI writes, decisions on individual cases and responsibility for the result. And what do we clarify first? Which data an AI service really needs, where it processes that data and whether it would be simpler without AI.

We mean that last question seriously. Sometimes a better form, a clear template or a different order of steps solves the problem without any AI at all. When that is the case, we say so.

The law recognises the idea too. The EU AI Act requires effective human oversight for high-risk systems (Art. 14). The GDPR generally gives people the right not to be subject to a decision based solely on automated processing that produces legal effects concerning them or similarly significantly affects them (Art. 22). Our checkpoint is not legal advice. It is a working rule that we apply regardless.

How do you measure whether the checkpoint is in the right place?

With a few figures, measured before and after. How many cases stop? How often do people change a draft before approving it? And how long does a case take from arrival to reply?

If almost everything stops, the process takes work off nobody’s hands. If almost nothing stops but drafts are still changed often, the checkpoint comes too late. If drafts are never changed, it is worth checking whether anyone is still reviewing them at all.

We adjust the limits in small steps and measure again. In the Toyota Production System, this is called Kaizen: improvement in small, verifiable steps. What works becomes the new standard.