Managed cloud ML platforms (SageMaker, Vertex AI, Azure ML) let a training job run on a fleet of instances of two kinds: on-demand (reserved, never taken away, full price) and spot / preemptible (spare capacity, ~70% cheaper, but the provider can reclaim it at any moment). Data-parallel training synchronizes every worker each step, so when even one spot node is reclaimed, the whole job stalls until it is replaced and catches back up. A checkpoint saved periodically limits how much progress a reclaim can erase.
stalled = any(worker.state != running)
progress += stalled ? 0 : rate·workers·dt
on interrupt: progress → last checkpoint
cost += Σ rate(type) · dt (bills even while stalled)
- Spot share — more spot instances lower the hourly bill but raise the odds any given tick includes a reclaim.
- Interruption rate — how often, on average, a spot worker gets reclaimed; shorter cloud-capacity windows mean more stalls.
- Checkpoint interval — shorter intervals lose less progress per interruption but add their own overhead in a real cluster.
- Restart time — how long the platform takes to provision a replacement instance and resume from checkpoint.
This is the core cost/reliability trade-off every team makes when choosing instance types for a SageMaker, Vertex AI or Azure ML training job: cheaper spot capacity versus predictable on-demand wall-clock time.