🤖 AI Chatbot Emergency Escalation Accuracy Simulator

Test Case Set Assembled

Hundreds of symptom scenarios enter the benchmark with known outcomes.

• 400+: Test cases loaded (symptom scenarios)
• 18%: True emergencies (baseline prevalence)
• 12: Symptom categories (chest, neuro, GI, more)
• ED charts: Ground truth source (physician-confirmed outcomes)

Why a labeled test set matters

Ground truth lets accuracy be measured, not guessed.

Case diversity across symptom types

Cases span chest pain, stroke signs, abdominal pain, and more.

Prevalence sets the difficulty baseline

Rare emergencies make high sensitivity harder to sustain.

Prevalence slider mimics real-world rare-emergency triage difficulty.

Chatbot Triage Run

The chatbot classifies every case as emergency or non-emergency.

• 400+: Cases classified (per benchmark run)
• Tunable: Decision threshold (conservative to aggressive)
• 2.1 s: Avg response time (per triage call)
• 40+: Signal features used (symptom keywords)

One decision boundary, many cases

A single threshold sorts every case into a call.

Conservative versus aggressive settings

Conservative settings escalate more, catching more true emergencies.

Shifting the threshold trades false negatives for false positives.

Signal overlap creates errors

Mild and severe cases can share similar symptom signals.

True/False Positive Tally

Chatbot decisions are checked against confirmed patient outcomes.

• Live: True positives (correctly escalated)
• Live: False negatives (missed emergencies)
• Live: False positives (unnecessary escalations)
• Live: True negatives (correctly reassured)

Building the confusion matrix

Every case lands in one of four outcome cells.

False negatives are the costly error

A missed emergency can delay life-saving care.

False negatives carry the highest real-world safety cost.

False positives have a cost too

Unnecessary escalations strain emergency departments and erode trust.

Sensitivity & Specificity Calculated

Confusion matrix counts convert into accuracy percentages.

• TP/(TP+FN): Sensitivity formula (emergency catch rate)
• TN/(TN+FP): Specificity formula (non-emergency accuracy)
• ≥95%: Target sensitivity (clinical safety bar)
• ≥80%: Target specificity (usability bar)

Sensitivity measures missed danger

High sensitivity means few real emergencies slip through.

Specificity measures unnecessary alarms

High specificity means fewer needless emergency referrals.

Both metrics move in tension as the threshold shifts.

Gauges visualize the tradeoff live

Twin gauges track sensitivity and specificity as sliders move.

Escalation Accuracy Benchmark

Overall performance is scored against a clinical safety threshold.

• 95% sens.: Safety threshold (minimum acceptable)
• Live: Current sensitivity (this configuration)
• Live: Current specificity (this configuration)
• Live: Pass/fail status (against threshold)

Passing the safety bar

A benchmark pass requires sensitivity above the safety floor.

Specificity still matters for adoption

Too many false alarms make a chatbot unusable.

A safe chatbot must be both sensitive and specific.

Iterating toward a better model

Benchmark failures guide retraining and threshold tuning.