Every test case in the golden set gets a quality score from the model, then the score is compared against a pass threshold — exactly like an automated evaluation gate in a CI pipeline. Cases that pass are routed to a small canary slice before full production traffic; cases that fail are logged as regressions and blocked from shipping.
score ~ N(μ, σ), μ = μ0 − 0.35·r, σ = σ0 + 0.10·r
μ0 = 0.72, σ0 = 0.12, r = regression severity (0–1)
pass(case) = score(case) ≥ τ τ = pass threshold
pass rate = passed / evaluated
routing (passed only):
canary if random() < c c = canary traffic fraction
prod otherwise
- Golden-set size — how many cases feed each evaluation wave; larger sets spawn test cases faster through the pipeline.
- Pass threshold — raises or lowers the gate plane; a stricter gate rejects more borderline cases.
- Regression severity — simulates a worse model build: it shifts the score distribution down and widens it, so more cases fall under the gate.
- Canary traffic — the fraction of passing cases diverted to the small canary ring instead of full production, for staged rollout.
This mirrors a real continuous-evaluation setup: golden-set regression tests plus a canary release gate catch quality drops before they reach every user.