Bronze → Silver: filter/dedupe (η≈0.7)
Silver → Gold: aggregate (η≈0.4)
Freshness = time since last orchestration run
Modern lakehouse architectures organize data into progressively refined "medallion" layers: raw Bronze data lands as-is, Silver applies cleaning/deduplication, and Gold holds business-ready aggregates. Streaming mode processes continuously with low latency; Batch mode runs on a fixed orchestration schedule, trading freshness for efficiency.
- Batch / Streaming toggle — switches the pipeline between scheduled batch runs and continuous micro-batch streaming.
- Data volume — scales the raw data arriving into the Bronze layer.
- Orchestration interval — in batch mode, how often a scheduler (like Airflow) triggers a full pipeline run.
- Medallion layer highlight — toggles colored pulse highlighting on the Bronze/Silver/Gold platforms as data moves through them.
Real-world application: this Bronze/Silver/Gold medallion pattern (popularized by Databricks) underlies most modern lakehouse platforms used for analytics and ML feature pipelines.