SIEM Log Ingestion Pipeline: Queueing & Storage-Tier Aging

Ingestion & Parsing

Ingestion rate 300 EPS

Parser throughput 350 EPS

Ingest buffer size 400 events

Hot-tier retention 7 days

Live Readouts

Ingest EPS

Parsed EPS

Ingest buffer backlog

Dropped events (buffer overflow)

Storage Tiers

Hot0

Warm0

Cold0

How it works

A SIEM ingestion pipeline is a queueing system. Log events arrive from hosts at rate λ (Ingestion rate) and a normalizer/parser stage drains the queue at rate μ (Parser throughput):

dQ/dt ≈ λ − μ   (Q = queue backlog)
stable  ⇔  μ ≥ λ  →  Q stays near 0
overload ⇔  μ < λ  →  Q grows until the buffer is full,
             then arriving events are dropped (log loss)

Events that clear the parser are written to a hot storage tier (fast, expensive, immediately searchable). After the configured hot-tier retention window they age into warm storage, then into cold/archival storage, and finally expire — the same lifecycle real SIEM platforms (Splunk, Elastic, Sentinel) use to trade search speed against storage cost:

t_hot  = retention_days · k
t_warm = 3 · t_hot
t_cold = 8 · t_hot     (k is a fixed simulation time-compression constant)

Ingestion rate / Parser throughput — the arrival rate λ and service rate μ of the queueing model above. Push λ past μ to watch the backlog and drop counter climb.
Ingest buffer size — the queue's finite capacity; once it is full, new arrivals are dropped rather than queued, exactly like a real log forwarder under backpressure.
Hot-tier retention — how long processed events stay in the fast/expensive hot tier before aging into warm and then cold storage. The 3D view is capped at a fixed instance count, so at very high combined rate × retention the oldest cold-tier events are evicted early — a stand-in for finite storage capacity.

Queued (unparsed) Hot storage Warm storage Cold storage

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.