Snowflake and the Separation of Storage from Compute
How Snowflake's architecture splits cloud storage, virtual warehouses and metadata services into independently scalable layers, and what that means for cost and concurrency.
Untangling storage from compute
Traditional data warehouses — think an on-premises Teradata or Netezza appliance — bundled storage and compute onto the same physical hardware, because that was the only architecture available at the time they were designed. The consequence was that scaling for more storage meant buying more compute whether you needed it or not, and vice versa, and every query, whether run by one analyst or a hundred, competed for the same fixed pool of CPU. Snowflake's central architectural decision was to break that bundle apart into three distinct layers that scale, and bill, independently: a storage layer built on cloud object storage (S3, Azure Blob, or GCS underneath), a compute layer made of independently sized 'virtual warehouses,' and a cloud services layer that handles metadata, query parsing, optimisation and access control.
This separation sounds like a small engineering choice but it changes the economics of running a warehouse fundamentally. A company can store ten years of clickstream history cheaply in the storage layer, essentially at object-storage prices, while only paying for compute during the minutes each day that someone actually queries it — and different teams can spin up their own independently sized compute clusters against the exact same underlying data without ever contending for CPU with each other.
Micro-partitions: the unit of storage
When data is loaded into Snowflake, it is automatically carved into micro-partitions — contiguous, immutable chunks typically holding between 50 and 500 MB of uncompressed data, stored compressed in a columnar format. For each micro-partition, Snowflake's metadata layer records the minimum and maximum value of every column, without you ever defining an index. This is a form of automatic, coarse-grained clustering: when a query filters on a column — `WHERE order_date > '2026-01-01'` — the query planner can skip entire micro-partitions whose min/max range doesn't overlap the filter, a technique called pruning, without ever touching the actual data in those partitions.
Because micro-partitions are immutable, an update or delete doesn't rewrite rows in place; it writes new micro-partitions with the changed rows and marks the old ones as no longer visible to new queries, which is also the mechanism behind Snowflake's time travel feature — you can query a table as it existed at a point in the recent past because the old micro-partitions haven't actually been deleted yet, just superseded. This immutable, append-oriented design is what makes Snowflake's storage layer safe to share across many concurrently running compute clusters: nobody is ever mutating a block another warehouse might be reading mid-query.
Virtual warehouses and independent scaling
A virtual warehouse is a cluster of compute nodes, sized in units called T-shirt sizes (X-Small through 6X-Large, each doubling the compute of the size below), that reads from the shared storage layer but has no persistent state of its own beyond a local SSD cache of recently accessed micro-partitions. Because a warehouse holds no unique data, spinning one up or down is nearly instantaneous, and — this is the important part — you can run any number of warehouses simultaneously against the same tables without them stepping on each other, because they aren't sharing CPU or memory, only reading the same immutable storage layer underneath.
This is why Snowflake became popular for solving the classic 'noisy neighbour' problem in shared warehouses: the finance team's month-end reporting queries and the data science team's ad hoc exploratory queries can run on entirely separate virtual warehouses, sized appropriately for each workload, billed separately, and neither can slow the other down by hogging CPU. Billing follows the same logic: Snowflake charges per-second for the time a virtual warehouse is actually running (with an auto-suspend feature that shuts idle warehouses down after a configurable period, typically minutes), so a warehouse that runs a five-minute batch job once a day costs a tiny fraction of one running continuously — a sharp contrast to the fixed-capacity appliance model where you paid for peak capacity around the clock.
The cloud services layer and where the intelligence lives
Sitting above both storage and compute is the cloud services layer, a set of shared, always-on services that handle query parsing and optimisation, authentication and access control, transaction management, and the metadata catalogue that tracks which micro-partitions belong to which table version. This layer is what makes zero-copy cloning possible: cloning a multi-terabyte table takes seconds because Snowflake is only copying metadata pointers to the same underlying micro-partitions, not physically duplicating any data — the clone only starts consuming new storage once one side or the other writes changes that diverge from the original.
The trade-off inherent in this architecture is that you give up the fine-grained physical control a traditional DBA might have over index structures, partitioning schemes, and storage layout — Snowflake makes almost all of those decisions automatically. For most analytical workloads this is a reasonable trade: automatic clustering and pruning perform well without tuning, and the operational simplicity of never managing physical storage more than pays for the lost control. It becomes a real limitation only for workloads that need extremely specific physical data layout for latency reasons, which is one reason Snowflake is aimed squarely at analytical (OLAP) rather than high-frequency transactional (OLTP) use cases.
Frequently Asked Questions
What actually is a Snowflake virtual warehouse?
It's an independently sized cluster of compute nodes that reads from Snowflake's shared storage layer but holds no unique persistent data itself, which is why any number of warehouses can run concurrently against the same tables without contending for resources.
How does Snowflake avoid needing manually defined indexes?
Data is automatically split into micro-partitions, and Snowflake records the min/max value of each column per partition; queries with filtering conditions can then skip entire partitions that can't match, a technique called pruning, without any index the user has to create or maintain.
Why is Snowflake billing described as per-second and elastic?
Because compute (virtual warehouses) is billed separately from storage and only while a warehouse is actively running, with auto-suspend shutting down idle warehouses, so short or bursty workloads cost a fraction of what a fixed-capacity system would charge for the same work.
How does zero-copy cloning work?
Because micro-partitions are immutable, cloning a table only needs to copy metadata pointers to the existing partitions rather than duplicating the underlying data, so a clone of a huge table appears almost instantly and only consumes new storage once the clone or original starts to diverge.