Test Suite Flakiness, Retries and Defect Discovery: How Many Nightly Runs Find the Bugs?
A grid of 200 code units, a suite of automated tests, hidden bugs and flaky tests that fail at random. Set how many tests run, how much code each touches, how often a test catches a defect, how many tests are flaky and how many retries are allowed, then watch nightly runs light up coverage, catch defects (with the expected growth curve) and raise false alarms. Retries cut false alarms but also hide intermittent real failures. Chips, bugs and server rack modelled in Blender, with Monte Carlo and ODE checks built in.
This simulation demonstrates the principles of automation testing in a CI/CD pipeline: a fixed-size test suite is distributed across a configurable number of parallel runner lanes, each pulling the next queued test the moment it becomes free. A per-test flakiness rate drives a real retry-on-failure policy — a test that fails once is retried automatically, and only a test that fails on both attempts is recorded as a final failure — the same mechanism used by real CI systems to keep transient flakes from blocking a build. Live readouts track completion progress, pass rate, retries consumed, measured throughput, and an estimated time remaining derived from the pipeline's own observed speed, showing directly how parallelism and reliability trade off in an automated test infrastructure.
A grid of 200 code units, a suite of automated tests, hidden bugs and flaky tests that fail at random. Set how many tests run, how much code each touches, how often a test catches a defect, how many tests are flaky and how many retries are allowed, then watch nightly runs light up coverage, catch defects (with the expected growth curve) and raise false alarms. Retries cut false alarms but also hide intermittent real failures. Chips, bugs and server rack modelled in Blender, with Monte Carlo and ODE checks built in.
3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install