Guardrail Cascade: Latency vs Accuracy

Traffic & Policy

Traffic rate 2.5 req/s

Repeat-prompt rate 35%

Early-exit margin 0.08

Harmful traffic share 25%

Live Readouts

Avg latency

— ms

Cache hit rate

— %

Decision accuracy vs ground truth

— %

Pipeline Key

In transit / undecided

Allow

Transform (redact)

Block

Elevated path = served from cache

How it works

Each request carries a hidden ground-truth harm score s ∈ [0,1] (harmful traffic drawn from a distribution centred near 0.75, safe traffic near 0.22). Three enforcement zones split that score:

s < 0.50        → Allow
0.50 ≤ s < 0.75  → Transform (mask / redact)
s ≥ 0.75         → Block

A cheap keyword filter, a mid-cost classifier and an expensive final classifier each re-estimate the score with shrinking noise (σ ≈ 0.22 → 0.10 → 0.03) and cost more compute time (≈1 ms → 8 ms → 45 ms). A stage commits to a decision only when its noisy estimate sits at least the early-exit margin away from every zone boundary — otherwise the request escalates to the next, more accurate but slower stage. The final stage always decides.

A small cache keyed by prompt identity remembers the last decision for TTL = 8s; a repeated prompt within that window skips the whole cascade (elevated fast lane).

confidence(stage) = min over boundaries b∈{0, .5, .75, 1} of |estimate − b|
commit if confidence ≥ margin, else escalate
latency = cache-lookup + Σ(stage costs actually traversed)

Lower margin → stages commit sooner → lower average latency, more mistakes near the boundaries.
Higher margin → more requests reach the accurate but slow final stage → higher latency, higher accuracy.
Repeat-prompt rate raises the cache-hit rate, which lowers latency without touching accuracy — the same trick real moderation stacks (and CDNs, DLP scanners) use to keep the expensive model off the hot path.

This mirrors how production LLM guardrails are actually built: a lightweight regex/keyword pass and result cache absorb most traffic, and only ambiguous, unseen prompts pay for the expensive classifier.

Cache Cheap filter (~1ms) Mid classifier (~8ms) Final classifier (~45ms)

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.

Guardrail Cascade: Latency, Caching & Early Exit