Containerization for Machine Learning Workloads

Fundamentals

Core Concepts

• Image layers and caching for faster builds and pulls.
• Multi-arch images and GPU-enabled bases (CUDA/cuDNN).
• Reproducibility via pinned versions and lockfiles.
• Security: least privilege, distroless, SBOM, signature verification.

Performance

• Reduce image size with multi-stage and selective package installs.
• Use runtime-specific optimizations (ONNX Runtime/TensorRT/OneDNN).
• Warm caches for model weights; mount read-only volumes when possible.

How the Algorithm Works

Build Graph

• Separate dependency install from application code for cache reuse.
• Pin CUDA/driver/toolkit versions to prevent cache invalidation.

Layer Ordering

• Place least-changing layers first (OS, runtimes), most volatile last (code).
• Keep model weights as distinct layers or mounted artifacts.

Distribution

• Use geographically replicated registries with immutable tags.
• Enable cross-repo layer mounts and registry-side caching.

Real-World Applications

Finance

• Signed images, strict SBOM checks, and immutable tags for audits.
• Multi-region active-active with deterministic builds.

Healthcare

• PHI handling with encryption and access controls; reproducible pipelines.
• Human review and detailed provenance records.

Retail and Ads

• Frequent rollouts; experiment-driven canaries; strict SLOs.
• Cost control with reserved capacity and serverless hybrids.

Edge

• Offline-tolerant, small images, and robust OTA update strategies.
• Hardware heterogeneity; per-site configuration and telemetry.

Best Practices

Security

• Run as non-root; drop capabilities; read-only rootfs.
• Scan images and enforce policies; sign and verify in admissions.

Performance

• Multi-stage builds; cache-friendly ordering; slim frameworks.
• Model artifacts separated; lazy load where possible.

Governance

• SBOMs and provenance; immutable tags; retention policies.
• Standard templates for base images and CI checks.

Evaluation

Security

• Vulnerability scans (OS and language deps) with enforced gates.
• Signature verification, SBOM validation, and base image policy.

Performance

• Image size budgets, pull time SLOs, and cold-start latency.
• GPU runtime validation and tensor throughput for inference images.

Reliability

• Health/readiness probes and crash recovery behavior.
• Compatibility across orchestrators and node images.

Worked Examples

GPU Inference Image

• Base: NVIDIA CUDA runtime + TensorRT; app: Triton/ONNX Runtime.
• Weights mounted at runtime; pinned memory buffers; health probes.

CPU Tabular Inference

• Base: distroless + ONNX Runtime; small image; fast cold starts.
• Security: non-root, RO rootfs, minimal capabilities.

CI/CD Pipeline

• BuildKit with remote cache; SBOM generation; vulnerability scan; sign.
• Promotion via immutable tags and policy gates.

Implementation

Step-by-step

1. Choose minimal base (distroless/ubi) or NVIDIA CUDA base for GPU.
2. Use multi-stage builds; compile in builder stage, copy runtime artifacts only.
3. Pin versions; lock dependency files; verify checksums.
4. Run as non-root; drop capabilities; readonly rootfs; seccomp/apparmor.
5. Mount secrets at build/run; never bake into images.
6. Add health checks; expose readiness and liveness endpoints.
7. Emit SBOM; sign images; enforce policy in admission controller.
8. Automate vulnerability scans in CI; fail on critical CVEs.

The Math Behind It

Throughput Model

• Expected pull time ≈ total bytes / network throughput + decompression cost.
• Parallel layer pulls reduce wall time up to CPU/network limits.

Layer Economics

• Fewer, larger layers vs many small layers: trade cache reuse and overhead.
• Compression ratios vary by content; binaries vs tensors.

Key Parameters

Core Parameters

• Image size and layer count
• Startup probes and timeouts
• User UID/GID and filesystem permissions
• Linux capabilities, seccomp/apparmor profiles
• Scan policy thresholds (block on critical CVEs)

Recommended Defaults

• Run as non-root; drop all capabilities unless explicitly needed.
• Keep images under practical size budgets; enforce size checks in CI.
• Enable health probes with conservative thresholds.

Training Strategy

Patterns

• Base training image with CUDA/cuDNN, framework, and drivers.
• Parameterize datasets and experiment configs via mounts and env vars.
• Track lineage: link images, code SHAs, data snapshots, and metrics.

Scheduling

• GPU node pools, resource quotas, and preemption for fair sharing.
• Sharded training and NCCL config for multi-GPU/multi-node jobs.