Consider a support chatbot handling roughly three thousand tickets a week. For every incoming ticket, the system generates a confidence score reflecting how likely its proposed automated resolution is to actually satisfy the customer. Above a threshold, the system resolves the ticket directly, deflecting it away from human agents entirely.
Deflection is valuable, but not unconditionally
Deflected tickets free agent time for genuinely difficult problems and often resolve simple, repetitive issues faster than waiting in a support queue. But deflection is only a win when the automated resolution actually works. A ticket auto-resolved incorrectly produces an arguably worse customer experience than a slower human resolution would have, because the customer now has to notice the issue was not actually fixed and escalate a case the system already marked closed.
Why confidence is an imperfect gatekeeper
A chatbot's stated confidence correlates with actual resolution quality, but the correlation is not perfect. Certain ticket types can resemble common, easily-resolved patterns on the surface while having an unusual underlying cause the system misses entirely, producing confidently wrong resolutions that a naive threshold alone will not catch.
Setting the threshold in practice
Most support organizations start conservative, deflecting only very high-confidence cases, and gradually lower the threshold as they accumulate evidence, customer satisfaction scores, reopen rates, follow-up contact volume, that the system handles a broader range of ticket types reliably.
Try it yourself
The AI Customer Service Lab simulates 3000 tickets with a realistic gap between stated confidence and true resolution quality, letting you move the auto-resolution threshold and watch deflection rate, poorly-handled cases, and agent hours saved respond together.
🧪 Try it yourself: the AI Customer Service Lab simulation lets you experiment with everything described above directly in your browser.