Streaming Analytics: Windows, Watermarks & HyperLogLog

Stream Controls

Event rate 15 /s

Tumbling window width 3.0 s

Watermark grace period 2.0 s

Late-event probability 15%

Open Window

Events (this window)

Closes in

0.0s

Exact distinct viewers

HyperLogLog estimate

Estimate error

0.0%

Stream Totals

Windows closed

Late events dropped

on-time event late / dropped closed window

How it works

Real streaming-analytics engines (Flink, Beam, Kafka Streams) group an unbounded event stream into finite tumbling windows of width W, keyed by each event's own event time — not the time it happens to arrive. Because network delay and retries mean events can arrive out of order, the engine tracks a watermark: a moving cutoff W(t) = now − D, where D is a grace period. A window is only finalized once the watermark passes its end boundary; anything that shows up later belongs to an already-closed window and is dropped as late data.

window(e)   = floor(eventTime(e) / W)
watermark   = simTime − D
window k closes when watermark ≥ (k+1)·W
late event: window(e) < currently-open window index

To answer "how many distinct viewers watched?" without storing every id, the panel runs a live HyperLogLog sketch: hash each viewer id to a uniform bit string, split it into an m-bucket index plus a tail, and keep the longest run of leading zeros seen per bucket (a long run is exponentially rare, so it signals many distinct items). The cardinality estimate is:

E = α_m · m² / Σ 2^(−M[j])     (j = 0..m−1)
α_64 ≈ 0.709,  standard error ≈ 1.04 / √m

Event rate — how many viewer events arrive per second.
Window width — the tumbling window size in event-time seconds.
Watermark grace period — how long the engine waits after a window's end before closing it, to absorb reordering.
Late-event probability — chance an event's true timestamp lags behind its arrival, sometimes enough to miss the watermark entirely.

The bar row is the last 10 tumbling windows (oldest at left, currently accumulating one on the right, glowing cyan). The small 8×8 grid is the live HyperLogLog register bank for the open window — brighter cells hold longer zero-runs.

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.

Streaming Analytics: Tumbling Windows, Watermarks & HyperLogLog