Home▸AI & Machine Learning▸Automation Testing Simulation

Test Suite Flakiness, Retries and Defect Discovery: How Many Nightly Runs Find the Bugs?

A grid of 200 code units, a suite of automated tests, hidden bugs and flaky tests that fail at random. Set how many tests run, how much code each touches, how often a test catches a defect, how many tests are flaky and how many retries are allowed, then watch nightly runs light up coverage, catch defects (with the expected growth curve) and raise false alarms. Retries cut false alarms but also hide intermittent real failures. Chips, bugs and server rack modelled in Blender, with Monte Carlo and ODE checks built in.

AI & Machine Learning3DModerate60 FPS📱 Mobile-adapted🟠 Blender⇄ 2D version
Automation Testing Simulation ↗ Open standalone

This simulation demonstrates the principles of automation testing in a CI/CD pipeline: a fixed-size test suite is distributed across a configurable number of parallel runner lanes, each pulling the next queued test the moment it becomes free. A per-test flakiness rate drives a real retry-on-failure policy — a test that fails once is retried automatically, and only a test that fails on both attempts is recorded as a final failure — the same mechanism used by real CI systems to keep transient flakes from blocking a build. Live readouts track completion progress, pass rate, retries consumed, measured throughput, and an estimated time remaining derived from the pipeline's own observed speed, showing directly how parallelism and reliability trade off in an automated test infrastructure.

⚙ Under the hood

A grid of 200 code units, a suite of automated tests, hidden bugs and flaky tests that fail at random. Set how many tests run, how much code each touches, how often a test catches a defect, how many tests are flaky and how many retries are allowed, then watch nightly runs light up coverage, catch defects (with the expected growth curve) and raise false alarms. Retries cut false alarms but also hide intermittent real failures. Chips, bugs and server rack modelled in Blender, with Monte Carlo and ODE checks built in.

Automation TestingWorkflowParallelSoftware TestingFlaky TestsTest CoverageDefect DiscoveryMonte Carlo

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install

What did you find?

Add reproduction steps (optional)