Every incoming ticket has a hidden type (Billing, Technical, Refund, Complaint, General). The router is a contextual bandit: for state (ticket type) s it keeps a running value estimate Q(s, a) for each action a ∈ {AI, Human}.
ε-greedy policy:
with prob ε → pick a random action (explore)
with prob 1-ε → pick argmax_a Q(s,a) (exploit)
Value update after observing reward r:
Q(s,a) ← Q(s,a) + α · [ r − Q(s,a) ]
Reward model (hidden ground truth the bandit must discover):
r_AI(s) = satisfaction_AI(s) + noise
r_Human(s) = satisfaction_Human(s) − cost_weight + noise
- ε (exploration) — higher values keep sampling both actions even after the policy has a favourite, at the price of more low-reward tickets in the short run.
- α (learning rate) — how fast each new observation overwrites the running estimate; too high and the bars jitter, too low and the policy converges slowly.
- Escalation cost — the router is penalised whenever it hands a ticket to a human agent, modelling staffing cost; push it high enough and the policy will happily accept lower satisfaction to keep tickets automated.
- Reward noise σ — the spread of the Gaussian jitter added to every observed reward; raise it and even a converged policy keeps flashing occasional red (low-reward) outcomes.
- Ticket types such as Complaint and Technical have low true AI satisfaction, so with a moderate cost the bandit should learn to escalate them while auto-resolving General and Billing.
Real-world relevance: this is the same explore/exploit trade-off that sits behind live AI-triage systems — a support platform must keep testing whether the model has gotten good enough to take a category off human agents, without tanking satisfaction while it experiments.