Eval Pipeline Cost Optimizer: Batching, Caching & Shadow Tests

Continuous evaluation is what keeps an LLM release pipeline honest, but every test case has to be paid for in both time and money. This simulator models a golden-set evaluation run as a queue of test cases flowing through a cache check, a batching collector and a model-call stage, with a fraction of the suite diverted onto a non-blocking shadow path. Four controls — golden-set size, batch size, cache-hit rate and shadow-traffic fraction — drive a real queueing-and-cost model, with live readouts for release-gate time, total pipeline spend and blocking model-call count, while a 3D particle flow shows cached, blocking and shadow test cases taking their different routes through the pipeline in real time.