Test Case Set Assembled
Hundreds of symptom scenarios enter the benchmark with known outcomes.
- 400+: Test cases loaded (symptom scenarios)
- 18%: True emergencies (baseline prevalence)
- 12: Symptom categories (chest, neuro, GI, more)
- ED charts: Ground truth source (physician-confirmed outcomes)
Why a labeled test set matters
Ground truth lets accuracy be measured, not guessed.
Case diversity across symptom types
Cases span chest pain, stroke signs, abdominal pain, and more.
Prevalence sets the difficulty baseline
Rare emergencies make high sensitivity harder to sustain.
Prevalence slider mimics real-world rare-emergency triage difficulty.
Chatbot Triage Run
The chatbot classifies every case as emergency or non-emergency.
- 400+: Cases classified (per benchmark run)
- Tunable: Decision threshold (conservative to aggressive)
- 2.1 s: Avg response time (per triage call)
- 40+: Signal features used (symptom keywords)
One decision boundary, many cases
A single threshold sorts every case into a call.
Conservative versus aggressive settings
Conservative settings escalate more, catching more true emergencies.
Shifting the threshold trades false negatives for false positives.
Signal overlap creates errors
Mild and severe cases can share similar symptom signals.
True/False Positive Tally
Chatbot decisions are checked against confirmed patient outcomes.
- Live: True positives (correctly escalated)
- Live: False negatives (missed emergencies)
- Live: False positives (unnecessary escalations)
- Live: True negatives (correctly reassured)
Building the confusion matrix
Every case lands in one of four outcome cells.
False negatives are the costly error
A missed emergency can delay life-saving care.
False negatives carry the highest real-world safety cost.
False positives have a cost too
Unnecessary escalations strain emergency departments and erode trust.
Sensitivity & Specificity Calculated
Confusion matrix counts convert into accuracy percentages.
- TP/(TP+FN): Sensitivity formula (emergency catch rate)
- TN/(TN+FP): Specificity formula (non-emergency accuracy)
- ≥95%: Target sensitivity (clinical safety bar)
- ≥80%: Target specificity (usability bar)
Sensitivity measures missed danger
High sensitivity means few real emergencies slip through.
Specificity measures unnecessary alarms
High specificity means fewer needless emergency referrals.
Both metrics move in tension as the threshold shifts.
Gauges visualize the tradeoff live
Twin gauges track sensitivity and specificity as sliders move.
Escalation Accuracy Benchmark
Overall performance is scored against a clinical safety threshold.
- 95% sens.: Safety threshold (minimum acceptable)
- Live: Current sensitivity (this configuration)
- Live: Current specificity (this configuration)
- Live: Pass/fail status (against threshold)
Passing the safety bar
A benchmark pass requires sensitivity above the safety floor.
Specificity still matters for adoption
Too many false alarms make a chatbot unusable.
A safe chatbot must be both sensitive and specific.
Iterating toward a better model
Benchmark failures guide retraining and threshold tuning.