A golden set of N test cases is split into blocking tests (must finish before a release is allowed) and shadow tests (run in the background, never gate the decision). Each test is either a cache hit — an item near-identical to one already scored, served instantly — or a real model call, grouped into batches of size B:
blocking = N·(1 − shadow) shadow = N·shadow
computeCalls = blocking·(1 − cache)
batches = ceil(computeCalls / B)
latency(B) = L0 + k·B (fixed overhead + per-item cost inside a batch)
gate time = batches · latency(B) ← the number that blocks a release
cost = batches·costFixed + computeCalls·costPerItem + cacheHits·costCache
Instead of watching a single settings point drift through a 3D pipeline, this view computes the whole (batch size × cache-hit rate) surface at once as a 2D heat field — every pixel is an independently evaluated gate-time (or cost) formula for that combination, with your current sliders marked as a crosshair. A discrete batch-fill gauge to the right replays the actual accumulate-then-fire batching mechanism in real time.
- Batch size ↑ — fewer, larger model calls: latency per batch grows roughly linearly but the fixed per-call overhead is amortized over more items, so cost per item falls while gate time can still rise if batches take too long to fill.
- Cache-hit rate ↑ — near-duplicate prompts are served from a cache instead of re-scored, cutting both compute calls and cost with no gate-time penalty.
- Shadow fraction ↑ — more of the suite runs after the decision instead of before it, shrinking gate time at the cost of total spend (the shadow tests still have to run, they just don't block anyone).
This is the real trade-off behind a production LLM evaluation pipeline: a pre-release gate that is fast enough to not slow down shipping, cheap enough to run on every change, and still catches regressions via a golden-set pass/fail threshold and canary rollout.