Eval Pipeline Cost Optimizer: Batching, Caching & Shadow Tests

Evaluation Suite

Golden-set size 500

Batch size 8

Cache-hit rate 30%

Shadow-traffic fraction 20%

Pipeline Result

Gate time (blocking)

0.0 s

Total pipeline cost

$0.00

Blocking model calls

Items animated

Cache hit — skips the model Blocking test — gates the release Shadow test — non-blocking

How it works

A golden set of N test cases is split into blocking tests (must finish before a release is allowed) and shadow tests (run in the background, never gate the decision). Each test is either a cache hit — an item near-identical to one already scored, served instantly — or a real model call, grouped into batches of size B:

blocking = N·(1 − shadow)      shadow = N·shadow
computeCalls = blocking·(1 − cache)
batches = ceil(computeCalls / B)
latency(B) = L0 + k·B            (fixed overhead + per-item cost inside a batch)
gate time = batches · latency(B)  ← the number that blocks a release
cost = batches·costFixed + computeCalls·costPerItem + cacheHits·costCache

Batch size ↑ — fewer, larger model calls: latency per batch grows roughly linearly but the fixed per-call overhead is amortized over more items, so cost per item falls while gate time can still rise if batches take too long to fill.
Cache-hit rate ↑ — near-duplicate prompts are served from a cache instead of re-scored, cutting both compute calls and cost with no gate-time penalty.
Shadow fraction ↑ — more of the suite runs after the decision instead of before it, shrinking gate time at the cost of total spend (the shadow tests still have to run, they just don't block anyone).

This is the real trade-off behind a production LLM evaluation pipeline: a pre-release gate that is fast enough to not slow down shipping, cheap enough to run on every change, and still catches regressions via a golden-set pass/fail threshold and canary rollout.

Cache Blocking Shadow

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.