Putting an LLM in production without giving it autonomy
Where the confidence threshold goes, what human review has to catch, and which decisions a language model should never be allowed to own in production.
The demo works. Someone pastes in a messy document, the model extracts the fields correctly, and the room agrees this should go into production.
The gap between that moment and a system you can run for a year is not model quality. It is the design around the model: what happens when it is wrong, how you find out, and who is accountable for the output.
Start by naming the cost of a wrong answer
Before anything else, write down what a single wrong output costs. Not the average. The worst realistic case.
That number determines the entire architecture. A model summarizing internal notes and a model coding a medical claim are the same technology in completely different systems.
| Cost of a wrong answer | Model role | Review |
|---|---|---|
| Mild annoyance, easily undone | Acts directly | Sampled after the fact |
| Rework by a person, no external impact | Acts, flags low confidence | Review the flagged tail |
| External party sees it, recoverable | Proposes, person confirms | Every output, before it leaves |
| Money moves, or a regulated record is created | Proposes, person edits and signs | Every output, with an audit trail |
| Irreversible, or legally binding | Assists only | The model never produces the artifact |
Notice that none of the rows say "when the model is accurate enough, remove the human". Accuracy moves you down the review column; it does not move you up the autonomy column. A 99% accurate model in an irreversible process still produces one catastrophe per hundred runs.
The threshold is a business decision
Most production LLM systems need a confidence signal so that uncertain cases get routed to a person. Where you set that threshold is not an engineering choice, and treating it as one is how these projects go wrong.
The threshold is a trade between two costs:
- Below the threshold, work goes to a human. That has a known cost per item and a queue.
- Above the threshold, work goes through unchecked. That has an unknown cost per error and a rate.
Setting the threshold means someone with budget authority deciding how many errors per thousand is acceptable, given what each error costs. Engineering's job is to measure both curves and present them. It is not to pick the number.
Confidence is not the model's self-report
Asking a model how confident it is produces a number that reads like a probability and is not one. Models are systematically overconfident, and the number correlates more with how fluent the answer sounded than with whether it was right.
More reliable signals, roughly in order of usefulness:
- Agreement across independent runs. Same input, different sampling or different prompt framing. Disagreement is a strong signal of an uncertain case.
- Verifiability against a source. If the model extracted a value, can you find that value in the input? An extracted field that does not appear in the document is a hallucination regardless of how confident anything claims to be.
- Structural validation. Does the output parse, does the total add up, does the code exist in the code set, is the date in a plausible range. Cheap, deterministic, and catches a surprising share of failures.
- Distance from the training distribution. Inputs unlike anything the system has seen before deserve a human, independent of what the model produced.
The first three cost extra calls or extra code. That cost is the price of knowing when to stop.
What human review has to actually catch
"A person checks it" is not a control unless the person can realistically catch the error. Three design details decide whether review is real:
Show the evidence, not just the answer. A reviewer looking at an extracted value with a link to the source line can verify in seconds. A reviewer looking at a value alone can only guess.
Make the correct action as fast as the lazy one. If approving is one click and correcting is a form, you will get approvals. The edit path has to be at least as fast as the accept path.
Measure reviewer disagreement. If reviewers approve 100% of what they see, review is not happening. That number is your best signal of whether the control is real, and it is worth alerting on.
Logging, because you will be asked
At some point someone will ask why the system produced a specific output on a specific day. If you cannot answer, the system does not belong in a regulated process.
At minimum, per decision: the input, the model and version, the prompt version, the raw output, the confidence signal, the routing decision, and, when a human touched it, who and what they changed. Retention long enough to cover your audit window.
This is also where data handling gets specific: what goes to a model provider, whether it can be trained on, and how long they keep it. Our position on that is written down in our AI policy, and the access controls around it in our security practices.
Decisions a model should not own
Independent of accuracy, some decisions should not sit behind a model, because the failure is not correctable after the fact:
- Anything that terminates a relationship: denial of care, account closure, termination.
- Anything that creates a legal or regulated record without a named signer.
- Anything where the affected person has a right to an explanation you cannot produce.
For these, the model can prepare, summarize, or draft. A person decides and signs. That is not conservatism, it is the only design where accountability has somewhere to land.
If the surrounding process is really screen automation rather than a decision, the tradeoffs are different: see RPA versus API integration. And if the model needs data from a system nobody can change, that is an anticorruption layer problem first.