SIEM Ingestion Queue Console: Backlog, Drops & Tier Aging

Ingestion & Parsing

Ingestion rate 300 EPS

Parser throughput 350 EPS

Ingest buffer size 400 events

Hot-tier retention 7 days

Simulation speed 1.0×

Live Readouts

Ingest EPS

Parsed EPS

Ingest buffer backlog

Dropped events (buffer overflow)

Storage Tiers

Hot0

Warm0

Cold0

Drag inside the pipeline view to pan it horizontally. The two strip charts below the pipeline track backlog% and tier populations over the last ~24s.

How it works

A SIEM ingestion pipeline is a queueing system. Log events arrive from hosts at rate λ (Ingestion rate) and a normalizer/parser stage drains the queue at rate μ (Parser throughput):

dQ/dt ≈ λ − μ   (Q = queue backlog)
stable  ⇔  μ ≥ λ  →  Q stays near 0
overload ⇔  μ < λ  →  Q grows until the buffer is full,
             then arriving events are dropped (log loss)

Events that clear the parser are written to a hot storage tier (fast, expensive, immediately searchable). After the configured hot-tier retention window they age into warm storage, then into cold/archival storage, and finally expire — the same lifecycle real SIEM platforms (Splunk, Elastic, Sentinel) use to trade search speed against storage cost:

t_hot  = retention_days · k
t_warm = 3 · t_hot
t_cold = 8 · t_hot     (k is a fixed simulation time-compression constant)

Ingestion rate / Parser throughput — the arrival rate λ and service rate μ of the queueing model above. Push λ past μ to watch the backlog and drop counter climb.
Ingest buffer size — the queue's finite capacity; once it is full, new arrivals are dropped rather than queued, exactly like a real log forwarder under backpressure.
Hot-tier retention — how long processed events stay in the fast/expensive hot tier before aging into warm and then cold storage. The pipeline view is capped at a fixed particle count, so at very high combined rate × retention the oldest cold-tier events are evicted early — a stand-in for finite storage capacity.
Simulation speed — a pure time-compression multiplier applied equally to arrivals, parsing and tier aging; it fast-forwards the wall clock without changing the λ/μ balance.

Note: the hot/warm/cold multipliers (×3, ×8) above were checked against the original 3D build's own event-aging code before porting — they matched, so no correction was needed here.

Queued (unparsed) Hot storage Warm storage Cold storage

drag to pan ↔