From Data Lake to Lakehouse: Why Raw Object Storage Needed a Table Layer
How unmanaged data lakes degraded into unreliable 'data swamps,' and how table formats like Delta Lake and Apache Iceberg bring ACID transactions and schema enforcement back to cheap object storage.
The promise and the trap of the data lake
The data lake pitch was simple and genuinely compelling: store everything — structured tables, semi-structured JSON, raw log files, images — in cheap, durable, effectively infinite object storage like S3, without forcing anyone to define a schema before writing. This 'schema-on-read' philosophy meant ingestion pipelines never had to block on a data modeller deciding column types in advance; you could dump raw data first and figure out how to interpret it later, which was a real improvement over rigid warehouse loading pipelines that rejected any record not matching a predefined structure.
The trap was that a folder full of files is not a database, and most of the guarantees people take for granted from a database simply don't exist on raw object storage. Two jobs writing to the same location concurrently can interleave their outputs into a corrupted mess, because object stores generally don't provide atomic multi-file commits. A job that fails halfway through writing leaves a directory in a half-written state with no way to tell which files belong to the failed attempt. Nothing enforces that a 'users' folder's files actually share a consistent schema, so six months in, half the files have an extra column and downstream jobs start crashing on schema mismatches they can't predict. Data engineers started calling the predictable result a 'data swamp': technically all your data, practically unusable without archaeology.
What a table format actually adds
Table formats — Apache Iceberg, Delta Lake, and Apache Hudi are the three major ones — solve this by adding a metadata layer on top of the same plain files sitting in object storage, without requiring a separate database engine to own the storage. Instead of a table being 'whatever files happen to be in this folder right now,' a table becomes a sequence of immutable snapshots, each one a manifest listing the exact set of data files that make up the table as of that snapshot. A write operation doesn't touch existing files at all; it writes new data files, then atomically swaps in a new manifest that points to the old files plus the new ones (for an append) or a revised set (for an update or delete), and only after that atomic pointer swap succeeds does the new snapshot become visible to readers.
This single mechanism — atomic manifest swaps — is what delivers ACID transactions on top of storage that was never designed to provide them. Two writers racing to commit at the same time will have their commits serialised through optimistic concurrency control: whichever commits its new manifest first wins, and the second writer detects the conflict and either retries against the new base state or fails cleanly, rather than the two writes silently interleaving into corrupted files. Readers, meanwhile, always see a single consistent snapshot for the whole duration of their query, because they're reading a fixed manifest, never a folder whose contents might change mid-scan.
Schema evolution and time travel as side effects
Because the manifest, not the physical file layout, is what defines a table's schema and structure at any given snapshot, table formats can support schema evolution — adding a column, renaming one, changing a type — without rewriting every existing data file. Old data files simply get read with the old schema for the columns they contain, and new columns show up as null for rows written before the column existed; the metadata layer reconciles the difference at query time rather than requiring an expensive backfill across petabytes of historical files.
The same snapshot mechanism also gives you time travel for free: because old manifests aren't deleted immediately (only garbage-collected after some retention window), you can query the table 'as of' an earlier snapshot or timestamp, which is invaluable for reproducing a report that looked different last week, or for recovering from a bad write by simply rolling the table back to the manifest that preceded it. This is conceptually the same trick as Snowflake's time travel over immutable micro-partitions — different implementation, same underlying idea: never truly delete the old state until you're sure you no longer need it.
The lakehouse: one copy of data, many engines
The term 'lakehouse' describes the resulting architecture: data lands and lives in cheap object storage in an open table format, but multiple compute engines — Spark, Trino, Snowflake, Flink — can all read and write the same physical tables through the same open metadata layer, rather than each engine locking data into its own proprietary storage format. This directly attacks the classic warehouse problem of maintaining separate, drifting copies of the same data for different tools: instead of an ETL job copying data from the lake into a warehouse-specific format for BI tools to query, the BI tool's query engine reads the lakehouse table directly.
The trade-off is that a lakehouse table format is a specification, not a managed service, so someone still has to run compaction (merging many small files produced by streaming writes into fewer, larger ones for scan efficiency) and garbage collection (actually deleting old snapshot files once they age out of the retention window) as ongoing maintenance jobs. Skip that maintenance and a lakehouse table degrades in exactly the way a database without vacuuming does: query performance drops as the number of small files balloons, even though the ACID guarantees themselves remain intact throughout.
Frequently Asked Questions
What's the actual difference between a data lake and a lakehouse?
A data lake is just files in object storage with no transactional guarantees; a lakehouse adds a table format metadata layer (like Delta Lake or Iceberg) on top of that same storage, providing ACID transactions, schema enforcement, and time travel while keeping the underlying files cheap and open.
How do table formats achieve atomic writes on object storage that wasn't built for it?
They never modify existing files in place; a write creates new data files and then atomically swaps in a new metadata manifest pointing to the correct set of files, so readers either see the complete old snapshot or the complete new one, never a partial state.
Why did data lakes get nicknamed 'data swamps'?
Without schema enforcement or transactional guarantees, lakes accumulated inconsistent file formats, half-written failed jobs, and undocumented structure over time, making the data technically present but practically unreliable to query.
Do I still need to maintain a lakehouse table, or is it fully automatic?
You still need periodic compaction to merge small files into larger ones and garbage collection to remove expired old snapshots; the table format guarantees correctness but doesn't automatically optimise physical file layout for you.