Containerising Machine Learning: Why GPU Scheduling and Model Artifacts Change the Rules

A stateless web service versus a model server

A typical web service container is nearly stateless: the image contains the application code and a handful of dependencies, it starts in a couple of seconds, and any instance is functionally interchangeable with any other, which is exactly what makes Kubernetes' horizontal scaling and rolling deployments so effective — kill one pod, start another, nothing is lost because nothing meaningful lived inside that pod. A machine learning model server breaks this assumption in a specific, consequential way: the container isn't just code, it's code plus a large, versioned binary artifact (the trained weights, sometimes tens of gigabytes for a large model) that has to be loaded into memory or GPU memory before the container can serve a single request, and that loading step can take anywhere from seconds to several minutes depending on model size and where the weights are stored.

This changes operational assumptions that Kubernetes' defaults were built around. A readiness probe that just checks "is the HTTP server listening" will report the pod as ready while the model is still loading, causing the load balancer to route real traffic to a pod that will time out or crash; ML deployments need a readiness probe that specifically checks whether the model has finished loading into memory. Startup time also directly affects how fast you can scale up under load or roll out a new model version — a two-minute model load time means autoscaling reacts on a two-minute delay, not the near-instant scaling a stateless web app enjoys, which has to be factored into capacity planning rather than assumed away.

GPUs are not a resource you can slice like CPU or memory

Kubernetes schedules CPU and memory with fine-grained, divisible units — you can request 250 millicores of CPU and 512MB of memory, and the scheduler will happily pack many small pods onto one node, sharing the underlying resource. GPUs, in the standard Kubernetes device plugin model, are scheduled as whole, indivisible units: a pod either gets an entire GPU or it gets none, because until relatively recently there was no standard, safe way to let two unrelated processes share GPU memory and compute without one interfering with or crashing the other. This means a lightweight model that only needs 2GB of a 40GB GPU's memory still monopolises the whole device under naive scheduling, which is a genuinely wasteful default that has driven a whole ecosystem of workarounds — NVIDIA's Multi-Instance GPU (MIG) hardware partitioning on newer data-centre cards, and software time-slicing schedulers that let multiple pods share a GPU's compute cycles in rotation, at the cost of some interference between workloads.

GPU nodes are also expensive enough, and scarce enough in cloud capacity terms, that treating them like ordinary compute nodes in an autoscaling group is a mistake teams often make once and then fix. A dedicated node pool with GPU-specific taints and tolerations (so only pods that explicitly request a GPU get scheduled there, keeping ordinary web traffic off expensive hardware), combined with aggressive scale-to-zero when idle, is standard practice specifically because GPU instances can cost five to twenty times as much per hour as equivalent CPU instances, and paying for an idle A100 overnight because a batch inference job's node pool never scaled down is a common, expensive early mistake.

Versioning the artifact, not just the code

A conventional Docker image tag (myapp:v1.4.2) captures the application code, but for an ML serving container the model weights are a second, independently-versioned artifact that changes on a completely different cadence than the code — a data science team might retrain and ship a new model weekly while the serving code changes only once a quarter. Baking the weights directly into the image (COPY model.bin into the Dockerfile) means every model update triggers a full image rebuild and a multi-gigabyte image push and pull, which is slow and couples two things that should be independently deployable and independently rollback-able. The more common production pattern instead treats the model artifact as external state, pulled at container startup (or mounted via an init container) from a model registry or object store, with the artifact's version pinned by a hash or tag referenced in the deployment configuration rather than the Dockerfile — this way rolling back a bad model doesn't require rebuilding and redeploying the whole image, just repointing the deployment at the previous artifact version, which is a much faster and safer rollback path.

Batch inference versus real-time serving as different container patterns

Real-time inference (a user-facing API that must respond in under, say, 200 milliseconds) and batch inference (scoring ten million records overnight for a daily report) are architecturally different enough that they're usually deployed as different kinds of Kubernetes workload entirely, not just different configurations of the same one. Real-time serving is typically a Deployment with a horizontal pod autoscaler reacting to request queue depth or latency, kept warm continuously because cold-start latency is unacceptable to an end user waiting on a response. Batch inference is much more naturally a Kubernetes Job or, for genuinely large workloads, orchestrated by a workflow engine like Kubeflow Pipelines or Argo Workflows, which can spin up a pool of GPU pods, distribute the ten million records across them, and tear the whole pool down to zero cost the moment the job finishes — a pattern that would be wasteful for real-time serving (constant cold starts) but is exactly right for a bursty, schedule-driven workload where you're optimising for total cost and throughput rather than per-request latency.