Flaky Test Detector: Sequential Probability Ratio Test

Ground Truth (hidden from the classifier)

True fail probability p 10%

SPRT Risk Budget

α — false-positive risk 5%

β — false-negative risk 5%

Rerun speed 4/s

Live Classifier State

Log-likelihood ratio

0.00

Reruns so far

Decision

Collecting data…

Trials completed

Avg reruns to verdict

—

How it works

A test that fails once isn't necessarily broken, and one that passes ten times in a row isn't necessarily reliable — the honest question is a hypothesis test, not a fixed rerun count. Wald's Sequential Probability Ratio Test (SPRT) treats each rerun as a Bernoulli trial and accumulates a log-likelihood ratio (LLR):

H0 "reliable": true fail rate = p0 = 2%
H1 "flaky":    true fail rate = p1 = 25%

per rerun:  LLR += ln(p1/p0)          if the run FAILED
            LLR += ln((1-p1)/(1-p0))  if the run PASSED

decide FLAKY     when LLR ≥ A = ln((1-β)/α)
decide RELIABLE  when LLR ≤ B = ln(β/(1-α))
else: rerun again

The two risk sliders set how far apart the boundaries A and B sit — a smaller α or β widens the gap the walk must cross, trading more reruns for fewer wrong verdicts. The "true fail probability" slider is the hidden ground truth used only to generate pass/fail outcomes; the classifier never sees it directly, only the LLR it derives from outcomes, exactly like a real CI system that doesn't know whether a test is flaky or just found a real bug.

This is the same statistic used by CI flaky-test quarantines (Google's, Meta's, and open-source rerun-analyzers) to decide, with a bounded error rate, whether to auto-quarantine a test instead of rerunning it a fixed number of times forever.

Pass rerun Fail rerun Current LLR walker

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.