Columnar Compression: Dictionary & Run-Length Encoding

Column Data

Row count 100,000

Distinct values (cardinality) 6

Sort order (clustering) 70%

Encoding

Live Readout

Raw size

—

Encoded size

—

Compression ratio

—

Avg run length

—

How it works

Column stores (Parquet, ORC, and the engines behind Snowflake / Redshift / BigQuery) keep every row's value for one column contiguously, which makes two classic compression tricks cheap:

Dictionary encoding
  build a table of the k distinct values,
  replace each row with a ⌈log2 k⌉-bit index
  size ≈ n·⌈log2 k / 8⌉ + k·avgBytes

Run-length encoding (RLE)
  replace each maximal run of r equal
  values with one (value, r) pair
  expected runs R ≈ 1 + (n-1)(1-p)
  size ≈ R·(avgBytes + runLenBytes)

Dictionary + RLE
  RLE runs of dictionary indices instead
  of raw values — compounds both savings
  size ≈ R·(codeBytes + runLenBytes) + k·avgBytes

Row count — how many values sit in the column (log scale, 1K–10M).
Cardinality k — how many distinct values exist; a low-cardinality column (status flags, country codes) compresses hardest.
Sort order p — the probability a value repeats its predecessor. Sorting or clustering a column before writing it is exactly what makes RLE effective: p → 1 means long runs, p → 0 means the value changes almost every row.
The 3D strip is a live sample of the column: each cube is one value, colour = which of the k distinct values it holds. RLE modes visually fuse a run into one wide block tagged ×N; dictionary modes shrink block height to show fewer bits per row and add a dictionary rack on the right.

Real-world relevance: this is why analytical warehouses ask you to sort or cluster tables by low-cardinality columns before loading — the same bytes on disk compress 10–100× better, which is most of why columnar formats outperform row stores for analytics.

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.