Each glowing sphere is one token of your prompt/response falling through the model's transformer layers (the rings) stacked inside the device box. In on-device mode every token stays inside the box — the whole loop is local. In cloud mode tokens burst out to the remote server, turn amber the moment they leave, and fly back — visualizing the network round trip a cloud assistant needs but an on-device one avoids.
tok/s = K / (params_B × bytes_per_param) — bytes/param: FP16=2, INT8=1, INT4=0.5
latency = (tokens / tok/s) × 1000ms + (cloud ? RTT : 0)
memory ≈ params_B × bytes_per_param (GB)
- Model — Phi-3 Mini (3.8B), Gemma 2B and Llama 3 8B: real small/medium LLMs that fit on a phone or laptop; more parameters means richer answers but fewer tokens/sec on the same hardware.
- Quantization — packing each weight into fewer bits (FP16 → INT8 → INT4) roughly halves memory and speeds up inference each step, at a small quality cost.
- On-device vs Cloud — on-device inference (Apple Intelligence, Gemini Nano, local llama.cpp) never sends your prompt anywhere; a cloud assistant (ChatGPT, most voice assistants) sends raw text to a server, adding network latency and a real privacy trade-off.
- Privacy score — a simplified indicator of how much of the conversation ever leaves the device boundary.