Each edge device runs a compressed copy of the model locally. When its confidence for a given input falls below the offload threshold, it sends the input to the cloud for a full-accuracy pass instead — trading network latency for a better answer.
local_accuracy = baseAcc * (1 - 0.6 * compression)
local_latency = baseLatency * (1 - 0.7 * compression) (smaller model, faster)
cloud_latency = networkRTT / bandwidth + cloudCompute
offload if local_confidence < threshold
- Devices — how many edge nodes are running inference in parallel.
- Model compression — quantizing/pruning the on-device model: faster and lighter, but less accurate.
- Network bandwidth — how fast a device can reach the cloud when it decides to offload.
- Offload threshold — how confident the local model must be before trusting its own answer instead of asking the cloud.
This latency/accuracy/bandwidth triangle is exactly what shapes real edge-AI deployments — smart cameras, wearables, industrial sensors — where every offloaded request costs both time and battery.