Data Engineering: Building Modern Data Infrastructure

Data Pipeline Architecture

ETL (Extract, Transform, Load): traditional approach — transform data before loading into warehouse. ELT (Extract, Load, Transform): load raw data first, transform in-warehouse using SQL — enabled by cheap columnar storage. Batch processing: scheduled jobs (hourly, daily) for large datasets. Streaming: real-time processing for low-latency requirements. Lambda architecture: batch layer + speed layer + serving layer. Kappa architecture: streaming-only, simplified. Data mesh: domain-oriented decentralized data ownership. Data contracts: schema agreements between producers and consumers. Idempotency: pipelines produce same result when re-run — critical for reliability.

Storage and Warehousing

Data warehouse: structured, schema-on-write, optimized for analytics. Snowflake: cloud-native, separate compute/storage, auto-scaling. BigQuery (Google): serverless, columnar, ML integration. Redshift (AWS): columnar, MPP architecture. Data lake: raw data in object storage (S3, GCS, ADLS), schema-on-read. Lakehouse: combines warehouse structure with lake flexibility. Delta Lake (Databricks): ACID transactions on data lakes, time travel, schema enforcement. Apache Iceberg: open table format, hidden partitioning, snapshot isolation. Apache Hudi: incremental processing, record-level updates. Parquet: columnar file format, efficient compression and encoding. Storage optimization: partitioning, bucketing, Z-ordering, compaction.

Processing Frameworks

Apache Spark: distributed processing engine, in-memory computation, 10-100× faster than MapReduce. Spark SQL, DataFrames, Datasets: structured API for batch/streaming. PySpark: Python API for Spark — dominant in data engineering. Apache Flink: true streaming engine with exactly-once semantics, event time processing. Apache Kafka: distributed event streaming platform, pub/sub, exactly-once delivery. Kafka Connect: pre-built connectors for databases, APIs, file systems. Apache Beam: unified batch/streaming API, runs on Spark, Flink, Dataflow. dbt (data build tool): SQL-based transformation, version control, testing, documentation. Polars: Rust-based DataFrame library, 10-100× faster than Pandas for large datasets.

Orchestration and Monitoring

Apache Airflow: DAG-based workflow orchestration, Python-defined pipelines. Airflow concepts: DAGs, operators, sensors, hooks, connections, XComs. Dagster: asset-oriented orchestration, type-checked, testable pipelines. Prefect: modern orchestration with built-in retry, caching, notifications. Temporal: workflow engine for long-running, fault-tolerant processes. Data quality: Great Expectations (automated data validation), dbt tests. Data lineage: tracking data from source to consumption (OpenLineage, Apache Atlas). Monitoring: pipeline SLAs, data freshness, volume checks, anomaly detection. Data catalog: searchable metadata (DataHub, Amundsen, Unity Catalog).

Modern Data Stack

Modern data stack: cloud-native, modular, best-of-breed tools. Ingestion: Fivetran, Airbyte (open-source) — automated EL from 300+ sources. Transformation: dbt — SQL-based, version-controlled, tested. Warehousing: Snowflake, BigQuery, Databricks. BI/Analytics: Looker, Metabase, Superset. Reverse ETL: Census, Hightouch — sync warehouse data to SaaS tools. Data governance: access control (RBAC), PII detection, masking, audit logs. Cost management: query optimization, auto-suspend warehouses, materialized views. Skills needed: SQL (essential), Python, Spark, cloud platforms (AWS/GCP/Azure), infrastructure as code (Terraform), containerization (Docker/K8s).

Try it live

Everything above runs in your browser — open Data Engineering: Building Modern Data Infrastructure and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.