🧠 AI Conversational Agent Crisis Detection

Linguistic Markers of Suicide Risk in Conversational Text

Decades of clinical-linguistics and NLP research show that suicide risk leaves measurable fingerprints in language, well before a person explicitly states intent. A crisis-aware chat system runs lightweight feature extraction on every single turn — not just turns that trip an obvious keyword — because the earliest markers are stylistic, not lexical.

• ~1,300: Pestian suicide-note corpus (annotated real + elicited notes, Cincinnati Children's)
• ~1.5–3×: Absolutist-word elevation (in suicidal-ideation forums (Al-Mosaiwi & Johnstone, 2018))
• 2011: CLPsych shared task, running since (annual NLP risk-assessment benchmark)
• 2014–2018: Coppersmith et al. cohort studies (Twitter/Reddit self-reported attempt vs. control)

The five marker families clinicians and NLP models both track

Suicide-risk linguistics research converges on a small set of recurring signal families:

• Absolutist language: words like "always," "never," "nothing," "completely" occur substantially more often in depression and suicidal-ideation forums than in anxiety forums or controls — Al-Mosaiwi & Johnstone's 2018 corpus study found absolutist-word usage roughly 1.5–3× baseline, a stronger discriminator than negative-emotion words alone. • Hopelessness markers: phrases mapping to the Beck Hopelessness Scale ("nothing will change," "no way out") — Beck et al. (1985) found hopelessness a better long-term predictor of eventual suicide than depression severity itself. • Temporal narrowing: a collapsing time horizon — present-tense fixation, loss of future-oriented verbs ("will," "someday," "next year"), consistent with the clinical concept of a "foreshortened future." • Means/method mentions: explicit or oblique references to lethal means, always treated as a high-priority signal regardless of surrounding sentiment. • Goodbye / burden language: farewell phrasing and perceived-burdensomeness statements ("everyone would be better off without me"), central to Joiner's interpersonal theory of suicide.

Pestian et al.'s annotated corpus of suicide notes and Coppersmith et al.'s matched-cohort studies of social-media users with documented attempts remain foundational training and validation resources for these marker families.

Passive extraction pipeline — enrichment, not classification

At this stage the system does not decide anything. Each turn is streamed through: tokenization and dependency parsing; lexicon lookups against clinically-derived word lists (LIWC-style categories: absolutist, sadness, first-person singular, death/means); a sliding embedding window that captures short-range context (negation, sarcasm cues, quotation of someone else's words); and a first-person pronoun density counter, since self-focused language (elevated "I"/"me"/"my") is one of the most replicated linguistic correlates of depressive and suicidal states (Stirman & Pennebaker, 2001; Rude et al., 2004).

The output is a feature vector attached to the turn, not a risk decision — that separation matters: it lets the same enrichment layer feed multiple downstream consumers (the classifier in Stage 2, audit tooling in Stage 5) without re-processing raw text.

Why coverage must be total, not keyword-gated

An early design temptation is to only run deeper analysis when a crude keyword list fires ("suicide," "kill myself"). This under-covers the two marker families that matter most for early detection — absolutist language and temporal narrowing — because neither requires an alarming word. A user who never says "suicide" but writes only in absolutes about a future that has already ended is a documented risk pattern.

Running passive extraction on every turn, at low latency (~45 ms typical), keeps the system sensitive to gradual linguistic drift across a multi-turn conversation rather than only reacting to a single alarming sentence.

Real-Time Risk Scoring — Fine-Tuned Classifier, Safety Head, and Calibration

Feature-enriched turns feed an ensemble scoring layer that must run inline, within the conversational turn budget, and output a calibrated probability rather than a raw logit — because everything downstream (thresholds, escalation, audit) depends on that number meaning what it claims to mean.

• ~355M: Fine-tuned classifier size (typical) (RoBERTa-large-class encoder, domain-adapted)
• 4: Risk tiers (low · moderate · high · imminent)
• ~0.62: CLPsych 2019 shared task best F1 (risk-level classification from social text)
• 2–3: Ensemble components (transformer classifier + LLM safety head + lexicon rules)

Ensemble architecture — three independent votes

Production crisis-detection layers rarely rely on a single model. A typical ensemble combines:

1. A fine-tuned transformer classifier (BERT/RoBERTa-class, ~110M–355M parameters) trained on labeled, de-identified crisis-line-style transcripts — analogous in spirit to the severity-triage models Crisis Text Line has described building on top of counselor-labeled conversation data. 2. The base conversational LLM's own built-in safety/moderation head — a general-purpose classifier trained across many harm categories, of which self-harm is one. 3. An interpretable lexicon/rules layer — the Stage 1 marker counts (absolutist density, hopelessness hits, means mentions) — which cannot "hallucinate" a score and provides an auditable trail for why a turn was flagged.

Disagreement between these three signals is itself informative: a lexicon spike with a low transformer score, or vice versa, routes to more conservative (higher-tier) handling rather than being averaged away.

Confidence calibration — making 0.7 mean 70%

Raw neural network outputs are frequently overconfident. Before a score can be compared against a decision threshold, it must be calibrated — typically via temperature scaling or isotonic/Platt regression fit on a held-out labeled set — so that among all turns scored "0.70," roughly 70% are empirically confirmed elevated-risk on review.

Calibration is re-validated whenever the model, prompt template, or user population shifts, since calibration curves are not guaranteed to transfer across domains (e.g., a model calibrated on adult crisis-line transcripts will typically miscalibrate on adolescent phrasing without re-fitting).

Four-tier output and the latency budget

The ensemble outputs one of four tiers — low, moderate, high, imminent — rather than a bare probability, because tiers map directly to different downstream actions (log only / soft check-in / escalate to review / immediate hand-off). The entire scoring pass must fit inside the conversational turn budget: production systems generally target well under 300 ms end-to-end so the safety layer never becomes the perceptible bottleneck in the chat experience, even though a slower, more thorough second-pass review can run asynchronously behind the scenes.

Escalation Triggers — Threshold Design and the Precision/Recall Trade-off

A risk score only matters once it is compared against a decision boundary. Crisis-detection systems generally implement two trigger paths simultaneously: a rolling, multi-turn trend crossing a sustained threshold, and a single-turn "hard trigger" for unambiguous high-severity content such as an explicit means mention — because averaging away one severe sentence into a calmer trend line is itself a known failure mode.

• >90%: Recall target at operating point (prioritized to minimize missed at-risk users)
• ~30–50%: Typical precision at that recall (common range across CLPsych-benchmarked systems)
• <1 s: Escalation latency budget (trigger event to reviewer-queue alert)
• 2: Single-turn hard-trigger categories (means/method mention, explicit goodbye statement)

Trend integration vs. single-turn spikes

Most conversations drift in risk gradually, so the system maintains an exponentially-weighted moving average (EWMA) of the per-turn risk score across the session, smoothing transient noise while still remaining responsive to genuine escalation. But EWMA smoothing has a blind spot: a single catastrophic sentence ("I have the pills right here") can be diluted by several preceding calmer turns and never push the trend line above threshold in time.

For that reason, means/method mentions and explicit goodbye/farewell language are wired as hard triggers that escalate immediately on a single turn, independent of the rolling average — the one place in the pipeline where the system deliberately does not wait for statistical confirmation.

Why the operating point is deliberately over-triggered

Choosing where to set the threshold is a precision/recall trade-off with asymmetric costs, structurally similar to a medical screening test: missing a true at-risk user (false negative) is categorically worse than an unnecessary check-in with a user who was not actually at risk (false positive). Consequently, crisis-detection systems are usually tuned to a high-recall, lower-precision operating point — commonly recall above 90% paired with precision in the 30–50% range on CLPsych-style benchmarks — accepting that a majority of "high" flags will, on human review, turn out not to require crisis-line escalation.

The explicit cost of this choice is alert fatigue for reviewers and a risk of eroding user trust if check-ins feel intrusive or mistimed; the explicit benefit is that the rare true emergency is far less likely to pass through silently. Section 3 of Stage 5 covers how that trade-off is continuously re-measured rather than set once and forgotten.

The design logic mirrors mammography screening: a program tuned for high sensitivity will generate a meaningful false-positive rate by construction. The alternative — tuning for high precision — quietly converts some fraction of real emergencies into false negatives, which in this domain is the outcome every design choice is oriented against.

Crisis Reviewer Escalation and the Warm Hand-off to 988 / Crisis Text Line

No automated system makes the final call alone. Once a conversation crosses the high or imminent threshold, it enters a human reviewer queue staffed by trained crisis personnel — and only after human confirmation does the system perform a "warm hand-off": actively connecting the user to a live resource rather than silently dropping a hotline number into the chat.

• Jul 2022: 988 Suicide & Crisis Lifeline launched (3-digit number replacing 1-800-273-8255)
• ~4.8 M: 988 contacts, first full year (calls, texts, chats — FY2023 SAMHSA report)
• >6 M: Crisis Text Line volume (conversations since 2013 launch (text HOME to 741741))
• <60 sec: Reviewer response SLA for high-tier flags (time to human eyes-on-transcript)

The reviewer queue — what a human confirms before acting

A flagged transcript arrives in the review queue with the full conversation context, the ensemble's tier and confidence, and the specific markers that triggered escalation (so the reviewer is not starting from a bare number). The reviewer's job is to confirm or downgrade the tier, decide whether the situation calls for a supportive check-in message, an offer of crisis resources, or immediate hand-off, and — only in genuine imminent-danger cases with policy authorization — initiate a welfare-check pathway.

Keeping a trained human as the decision-maker for any consequential action is the core safeguard against both automation bias (over-trusting the model) and the liability and dignity concerns of an AI system unilaterally contacting emergency services on a user's behalf.

What "warm hand-off" means in practice

A warm hand-off is the difference between the chatbot printing a hotline number and the chatbot actively bridging the user to a resource without leaving them alone in the interim: acknowledging what the user shared, explaining what is about to happen and why, offering (not forcing) a connection to 988 Suicide & Crisis Lifeline (call, text, or chat) or Crisis Text Line, and remaining present in the conversation rather than ending the session the moment a resource is surfaced.

988 and Crisis Text Line are staffed by trained crisis counselors and are the standard real-world endpoints referenced by warm hand-off protocols in the United States; equivalent national services exist elsewhere and production systems localize the hand-off target to the user's jurisdiction.

Warm hand-off protocols exist because the handoff moment is itself a documented risk point — an abrupt or impersonal redirect can increase disengagement at exactly the wrong time. The reviewer's presence and the conversational continuity of the hand-off are treated as part of the intervention, not administrative overhead around it.

False-Positive/False-Negative Auditing and Continuous Model Validation

A crisis-detection system is never "done." Safety review boards continuously audit both what the model flagged and — critically — a sample of what it did not flag, because false negatives are invisible by default. Precision and recall are recomputed against real outcomes wherever available, and the threshold and model are recalibrated on a defined cadence.

• 100%: High-tier flags clinician-reviewed (every escalated transcript gets human audit)
• random %: Unflagged transcripts sampled for FN audit (catches misses invisible to flag-only review)
• quarterly: Recalibration cadence (or immediately after red-team findings)
• 1,000s: Adversarial red-team probes per cycle (paraphrase, obfuscation, code-switching attacks)

Auditing both directions of error

Every escalated transcript is reviewed by a clinician or trained safety analyst, who labels it as a true or false positive relative to what actually happened. That alone only measures precision. Measuring recall requires the harder discipline of periodically sampling transcripts that were never flagged at all and checking them for missed risk signals — otherwise false negatives simply never enter the audit trail, and a classifier can silently degrade at recall while precision metrics look stable.

Outcome data, where legally and ethically obtainable (e.g., confirmed hand-off follow-through, aggregated de-identified outcome reporting), is fed back to re-anchor precision/recall estimates to reality rather than to proxy labels alone.

Red-teaming and drift monitoring

Between scheduled audits, dedicated red-team exercises probe the classifier with adversarial phrasing: paraphrases and obfuscations of known risk language, code-switching and dialectal variation, sarcasm and quotation (someone describing a friend's risk, not their own), and slow multi-turn escalation designed to stay under the EWMA trend threshold. Findings from these exercises feed directly into both the training set for the next fine-tuning pass and, when severe, an immediate threshold or lexicon patch rather than waiting for the quarterly cycle.

Model and population drift are also monitored passively: shifts in the distribution of scores, marker frequencies, or reviewer override rates over time can indicate the model is no longer well-matched to current conversational patterns even before an audit formally catches it.

Governance — audit logging and accountability

Every scoring decision, threshold crossing, reviewer action, and hand-off is written to an immutable audit log: what the model saw, what it scored, which tier it assigned, who reviewed it, and what action followed. This log is the substrate for both the precision/recall recomputation above and any external accountability review.

A standing safety review board — spanning clinical, engineering, and policy stakeholders — owns the threshold, reviews aggregate audit findings, and has authority to tighten or loosen the operating point. The system explicitly treats crisis detection as decision support for human reviewers and real crisis services, never as an autonomous clinical diagnosis.

The single most important structural property of a responsible crisis-detection pipeline is that its recall/precision numbers are re-measured against real outcomes on a fixed cadence, not asserted once at launch and left static — because both user language and adversarial evasion attempts evolve continuously, and a classifier frozen in time degrades quietly rather than loudly.