Tiered storage policies balance access latency against petabyte-scale storage cost
Genomic data lakes exist because sequencing output has outpaced Moore's-law-style storage cost declines: a single high-coverage whole-genome run produces 30–150 GB of raw and aligned data, and a mid-sized sequencing core can generate hundreds of terabytes per month. Every file starts in the hot tier, because pipelines need it immediately after the run completes.
A genomic data lake is an object-storage-backed repository — almost always built on Amazon S3, Azure Blob, or Google Cloud Storage — that holds heterogeneous sequencing artifacts (FASTQ, BAM, CRAM, VCF, gVCF, and increasingly long-read formats like BAM/PacBio HiFi and POD5) alongside structured metadata catalogs. Unlike a data warehouse, a data lake stores files in their native format and defers schema decisions to read time, which suits genomics because different pipelines (variant calling, structural variant detection, RNA-seq quantification) each need different representations of the same underlying sample.
The scale problem is structural: a 30× whole-genome sequencing run produces roughly 90 GB of raw FASTQ plus aligned BAM, and a population-scale cohort study (10,000–500,000 genomes) can accumulate 1–50 petabytes. The UK Biobank whole-genome release alone exceeds 15 PB. At S3 Standard list pricing (~$0.023/GB-month), storing 1 PB indefinitely in the hot tier costs roughly $23,000 per month before any compute, egress, or redundancy overhead — which is precisely the pressure that makes tiering economically mandatory rather than optional.
A single population biobank of 500,000 genomes at 30x coverage, stored uniformly in S3 Standard, would cost on the order of $1M/month in storage alone — tiered storage policy is not an optimization, it is a prerequisite for the economics to work at all.
Four file types dominate genomic ingestion, each with distinct size and access characteristics:
• FASTQ: raw base-called reads with per-base quality scores, uncompressed or gzip-compressed. Largest and least reusable — once aligned, most pipelines never touch the FASTQ again, making it an ideal early archive candidate.
• BAM: Binary Alignment Map, reads aligned to a reference genome with CIGAR strings, MAPQ, and tags. The workhorse intermediate format read by variant callers, QC tools, and visualization software (IGV). Roughly 2–4× larger than the equivalent CRAM.
• CRAM: reference-based compressed alignment format, typically 30–60% smaller than BAM because it stores only the delta against a reference sequence rather than the full base string. Increasingly the default long-term storage format recommended by GA4GH.
• VCF/gVCF: Variant Call Format records the called variants (SNPs, indels) rather than raw reads — three to four orders of magnitude smaller than the source BAM, and the format most frequently re-queried by downstream analysis, joint genotyping, and clinical interpretation pipelines.
Ingestion pipelines typically land all four in the hot tier simultaneously, since the alignment and variant-calling steps that produce BAM/CRAM and VCF from FASTQ happen within hours of sequencing completion.
S3 Standard (and its equivalents) is optimized for millisecond first-byte latency and unlimited request rate — exactly what a running Nextflow or WDL pipeline needs when it is actively reading and writing intermediate files. But S3 Standard is also the most expensive storage class per gigabyte, roughly 5–20× the cost of Glacier Deep Archive depending on region.
The ingestion stage is deliberately cost-blind: files are written to the hot tier regardless of their eventual access pattern, because that pattern is not yet known. A file's "temperature" — how frequently and how urgently it will be read — only reveals itself over the following days to weeks, which is exactly the signal the next stage, hot-tier access-frequency analysis, is built to capture.
Before any file can be safely moved to cheaper storage, the system must know how often it is actually read. S3 server access logs, CloudTrail data events, and S3 Storage Lens metrics are mined to build a per-object access-frequency profile, distinguishing files still driving active analysis from files nobody has requested in weeks.
Amazon S3 Intelligent-Tiering is purpose-built for exactly this problem: it monitors access patterns at the object level and automatically moves objects between a Frequent Access tier and an Infrequent Access tier after 30 consecutive days without a GET request, with optional Archive Instant Access, Archive Access, and Deep Archive Access tiers layered on top for objects untouched for 90, 180, and beyond.
The monitoring itself carries a small per-object fee (~$0.0025 per 1,000 objects per month), which matters at genomic-lake scale: a cohort with 100 million small VCF shards can accrue a meaningful monitoring bill if objects are not first consolidated. This is why many genomic pipelines batch small files into larger archives (tar, or columnar formats like Hail's MatrixTable / VDS) before they ever reach the lifecycle engine — fewer, larger objects reduce both monitoring overhead and per-request archive retrieval fees.
Access logs reveal a highly skewed distribution characteristic of genomic workloads:
• Raw FASTQ: read once by the alignment step, then almost never again — a near-perfect candidate for immediate archival once the derived BAM/CRAM exists • BAM/CRAM: read repeatedly during an active project (QC, realignment, structural variant calling) but access frequency drops sharply once a project's primary analysis phase concludes • VCF/gVCF: the longest-lived hot object — re-queried whenever a new phenotype association study, joint-genotyping run, or clinical reanalysis occurs, sometimes years after original sequencing • Reference and index files (FASTA, BED, .fai, .crai): constantly hot, shared across every sample, and rarely candidates for archival regardless of policy
The access-frequency threshold slider in this simulator controls how many monthly GET requests a file must fall below before it is flagged cold — a low threshold is aggressive (more files archived sooner, higher restore risk) while a high threshold is conservative (more files stay hot, higher storage cost).
GA4GH's Data Repository Service (DRS) specification standardizes how a resolver hands back a signed URL for a genomic object regardless of which storage tier it currently sits in — meaning access-frequency-driven tiering can happen transparently to downstream consumers without breaking pipeline compatibility.
The output of this stage is not a single global decision but a per-object "temperature map" that feeds the lifecycle policy engine: each file is tagged with days-since-creation, days-since-last-access, access-count-30d, and an inferred project-active flag drawn from ELN/LIMS metadata where available.
This map is what allows the next stage to make nuanced, rule-based decisions rather than crude age-based ones — a five-year-old VCF still queried monthly by an active GWAS should stay warm, while a five-day-old FASTQ from a completed alignment run should already be flagged for transition.
With a temperature map in hand, a rules engine evaluates every object against configured age and access thresholds and assigns it to hot, warm, cold, or archive tier. This is where the archive-delay slider and access-frequency threshold set in the UI translate directly into S3 Lifecycle Configuration rules applied at bucket or prefix scope.
S3 Lifecycle Configuration expresses tiering as a set of declarative XML/JSON rules scoped by prefix or object tag: "transition objects under s3://lake/fastq/ to Glacier Flexible Retrieval 30 days after last access" or "transition objects tagged project=completed to Deep Archive after 180 days." Nextflow and WDL pipelines can emit these tags automatically at file-creation time, so a workflow that knows a file is a terminal, non-reusable intermediate can pre-declare it for aggressive archival rather than waiting for the access-frequency monitor to catch up.
Each storage class carries a minimum storage duration charge — objects deleted or transitioned out of S3 Standard-IA before 30 days, Glacier Flexible Retrieval before 90 days, or Deep Archive before 180 days still incur the full minimum-duration fee. This is why the archive-delay slider matters operationally, not just as a UI control: setting it too aggressively (a short delay) risks objects churning between tiers and paying multiple minimum-duration penalties, while setting it too conservatively leaves cheap-to-archive data sitting in expensive tiers.
A poorly tuned lifecycle policy that transitions and re-accesses (and thus re-transitions) the same object repeatedly can end up more expensive than never tiering at all, once transition-request fees and minimum-duration penalties are summed — tiering aggressiveness has a real cost floor, not just a benefit ceiling.
Before deciding where a file should live, well-designed lifecycle policy also decides what format it should live in. Converting BAM to CRAM using reference-based compression (via samtools or GATK) typically shrinks alignment files 30–60%, because CRAM stores only the delta against a known reference sequence and applies more aggressive quality-score binning than BAM's general-purpose BGZF compression.
For a 1 PB alignment archive, converting BAM→CRAM before transitioning to a cold tier can reduce the archived footprint to 400–600 TB — meaning the compression step alone can rival or exceed the savings from tier transition itself. Many lifecycle policies therefore chain two actions: first a Lambda-triggered CRAM conversion job once a BAM crosses the access-frequency threshold, then a standard lifecycle transition rule on the resulting (smaller) CRAM object.
The access-frequency threshold and archive-delay slider jointly define a point on a cost-latency frontier:
• Aggressive (low threshold, short delay): files are archived quickly, storage cost falls fast, but researchers hit more surprise multi-hour Glacier restore waits and pay more per-GB retrieval fees when they do need archived data
• Conservative (high threshold, long delay): files stay hot and instantly available for longer, at the cost of paying S3 Standard rates for data that may never be touched again
There is no globally optimal setting — the right point on the frontier depends on the reanalysis rate specific to the cohort. A rare-disease clinical cohort with an active diagnostic odyssey may re-query VCFs monthly for years, favoring conservative thresholds; a completed population-genetics reference cohort with sequencing finished and papers published may tolerate aggressive archival with near-zero operational impact.
Once the policy engine flags a file, the actual migration is handled transparently by the storage platform: S3 Lifecycle transitions rewrite the object's storage class in place, no application code changes, no new object key. Cold files shrink visually on the storage map as they move from the hot ring outward through warm, cold, and archive zones.
When a lifecycle rule fires, S3 performs the storage-class change as a background, asynchronous operation — the object key, ETag, and metadata are preserved, but the underlying storage class attribute changes from STANDARD to STANDARD_IA, GLACIER, or DEEP_ARCHIVE. Applications addressing the object by key see no interruption for read-eligible tiers, but any GET issued against an object sitting in Glacier or Deep Archive fails with an InvalidObjectState error unless a restore request has first been issued and completed.
This is the critical operational distinction the visualization captures: files in the hot and warm rings remain instantly readable, while files that cross into the cold and archive rings become "read-on-request" rather than "always available." Pipeline authors must explicitly issue a RestoreObject call and poll for completion — a design decision that trades storage cost for engineering complexity in the read path.
Both Glacier Flexible Retrieval and Glacier Deep Archive offer multiple retrieval speed tiers, each priced differently:
• Glacier Flexible — Expedited: 1–5 minutes, ~$0.03/GB, reserved for urgent unplanned access • Glacier Flexible — Standard: 3–5 hours, ~$0.01/GB, the default for planned reanalysis • Glacier Flexible — Bulk: 5–12 hours, ~$0.0025/GB, cheapest for large batch restores • Glacier Deep Archive — Standard: within 12 hours, ~$0.02/GB, intended for data essentially never re-read except for compliance or rare reanalysis • Glacier Deep Archive — Bulk: within 48 hours, ~$0.0025/GB, the cheapest retrieval path in the entire tier ladder
Egress bandwidth charges (data leaving AWS to the public internet or to another cloud/institution) apply on top of retrieval fees, and for large genomic transfers can exceed the retrieval fee itself — a 100 TB restore-and-download of a Deep Archive dataset can incur meaningful five-figure egress costs, which is why many consortia now colocate compute with the storage tier rather than pulling data out.
Amazon S3 Glacier Deep Archive prices storage at roughly $0.00099/GB-month — about 4% of S3 Standard's list price — which is why a well-tuned tiering policy can cut a genomic lake's storage bill by 60–90% even though the archived data still physically exists and remains retrievable.
The four tiers modeled in this simulator differ across cost, latency, and durability in ways that map directly onto the concentric zones in the canvas visualization — hot at the center, archive at the outer ring.
| Product | Indication | Trial Design | Key Result |
|---|---|---|---|
| Hot (S3 Standard) | ~$0.023/GB-mo | Millisecond first-byte latency, unlimited request rate | Active pipeline I/O, no restore step |
| Warm (S3 Standard-IA) | ~$0.0125/GB-mo | Millisecond latency, per-GB retrieval fee applies | Infrequently accessed but latency-sensitive data |
| Cold (Glacier Flexible) | ~$0.0036/GB-mo | Minutes-to-hours restore, tiered retrieval speed options | Reanalysis-eligible archives, project cold storage |
| Archive (Glacier Deep Archive) | ~$0.00099/GB-mo | 12–48 hour restore, lowest cost per GB in AWS | Regulatory retention, near-never-read raw data |
The final stage tallies what tiering actually bought: dollars saved per month against the retrieval latency and operational risk accepted in exchange. A complete cost report separates storage savings, retrieval fees actually incurred, and the residual risk of an unbudgeted large-scale restore — the number that determines whether the policy needs retuning next cycle.
A genomic data lake's cost report is assembled from three data sources: AWS Cost Explorer (or equivalent) broken down by storage class and cost-allocation tag, S3 Storage Lens for per-prefix object-count and size trends, and application-level logs of RestoreObject calls with their retrieval-tier selection. Together these answer three questions: how much are we paying for storage at rest, how much are we paying to read data back out of cold tiers, and is the ratio between those two numbers trending toward or away from the policy's intended target.
Mature genomics platforms tag every object at creation time with project ID, sample type, and expected-reanalysis-frequency, which lets Cost Explorer attribute storage spend per research program rather than reporting an undifferentiated bucket-wide total — essential when a shared data lake serves dozens of concurrent studies with very different access patterns.
The dollar savings from tiering are easy to quantify; the latency cost is not, but it is real. A researcher who needs a Deep-Archive-tier BAM for an urgent clinical reanalysis and discovers a 12-hour restore window has incurred a cost — in delayed diagnosis, delayed publication, or an emergency Expedited-tier retrieval fee — that never appears on the monthly AWS invoice.
Well-run programs quantify this by tracking a "restore SLA breach rate": the fraction of restore requests that exceeded the requester's acceptable wait time. A policy tuned too aggressively toward cost savings will show a rising breach rate even as the storage bill falls, which is the operational signal to loosen the access-frequency threshold or shorten the archive delay for the affected file classes.
Consortia running federated genomic data lakes increasingly report cost not as a single $/month figure but as a Pareto frontier of $/month versus 95th-percentile restore latency — making the cost-latency tradeoff an explicit, negotiated parameter of the data management plan rather than an accidental byproduct of default settings.
The cost report feeds back into stage 3's lifecycle rules: sustained low restore rates on a given prefix justify a shorter archive delay and lower access-frequency threshold (push more data colder, faster); a rising restore rate or breach rate justifies the opposite. This closed loop is why production genomic data lakes re-run access-frequency analysis on a recurring schedule — typically monthly — rather than setting a lifecycle policy once and leaving it static for the life of the dataset.
GA4GH's broader interoperability stack (DRS for object resolution, Passport/visa-based access control, htsget for byte-range genomic queries) is what makes this closed loop safe to automate: because consumers resolve data through an abstraction layer rather than a hardcoded S3 URL, the underlying storage class can change — repeatedly, automatically, invisibly — without breaking any downstream pipeline or requiring institutions to renegotiate data-sharing agreements every time a tiering policy is retuned.