Consider a chatbot used to triage patient symptom reports by urgency before any human clinician gets involved. For the large majority of routine cases, the system can confidently make a reasonable triage call on its own. For ambiguous or high-stakes cases, the safer move is to escalate to a human nurse rather than let the system decide unsupervised.
The threshold is the actual product decision
Everything about how this system behaves in practice comes down to where the confidence threshold for escalation sits. Set it low, and the system escalates aggressively, which is safer for patients but adds real cost in nurse time and slows down the majority of straightforward cases that genuinely did not need human review. Set it high, and the system automates more cases, which is cheaper and faster, but risks handling some genuinely risky cases without a human ever looking at them.
Confidence is not the same as safety
The subtlest issue here is that a model's stated confidence is not a perfectly reliable proxy for whether a case is actually safe to auto-handle. Models can be overconfident on cases that turn out to be risky, particularly ones that resemble common patterns on the surface but have an unusual underlying cause. A threshold set purely on confidence, with no correction for this gap, systematically under-escalates exactly the cases where it matters most.
Why this is a genuinely hard trade-off
There is no threshold setting that eliminates both failure modes simultaneously. Real deployments in high-stakes domains like healthcare typically err conservative, accepting a higher escalation rate and the associated cost, and validate the threshold against historical outcomes with clinical oversight rather than optimizing purely for automation rate.
Try it yourself
The AI Healthcare Chatbot Lab simulates 500 intake cases with a realistic gap between stated confidence and true safety, letting you adjust the escalation threshold and see exactly how many risky cases would have slipped through auto-handling at each setting.
🧪 Try it yourself: the AI Healthcare Chatbot Lab simulation lets you experiment with everything described above directly in your browser.