☁️ Cross-Cloud Genomic Data Federation Query Engine

Query Submission to the Federation Layer

Cross-cloud genomic federation begins the moment a researcher writes a cohort-level query — not a data-download request. Instead of pulling BAM/CRAM/VCF files to a local analysis environment, the researcher submits a declarative query (e.g. allele frequency, case-control burden, phenotype correlation) to a federation layer that will resolve where the answer lives without ever centralizing the underlying genomes.

• 50k–2M: Typical cohort scale (individuals per federation)
• SQL / GraphQL: Query languages supported (plus GA4GH Beacon v2)
• ~1.2 KB: Median query text size (vs. TB-scale raw data)
• 3–12: Sites federated (typical consortium) (clouds + institutions)

Why the query moves and the data doesn't

Genomic cohorts assembled by biobanks, hospital networks, and consortia are increasingly split across administrative and legal boundaries: some sites run on AWS, others on GCP, others remain strictly on-premises for regulatory reasons (HIPAA in the US, GDPR and national health-data laws in the EU, data localization mandates elsewhere). Centralizing raw sequence and clinical data into one warehouse is often legally prohibited, operationally expensive (a single deep WGS cohort of 10,000 genomes can exceed 300 TB), and ethically fraught — participants frequently consent to analysis at their originating institution, not indefinite copying elsewhere.

Federated query architectures invert the classic ETL model: rather than moving petabytes of raw reads to a central compute cluster, the federation layer moves a few kilobytes of query logic to wherever the data already lives, and only small, governed answers travel back. This is the same principle behind the GA4GH Beacon API and DRS-based data access, generalized to full analytic queries rather than single-variant lookups.

A typical federated allele-frequency query is under 2 KB of SQL or GraphQL text. Moving that query to three sites costs microseconds of network transfer; moving the 300+ TB of raw genomes those sites hold would take days over a dedicated 10 Gbps link.

The researcher-facing query surface

Researchers rarely write raw distributed-systems code. The federation layer exposes a single logical endpoint — commonly a Trino/Presto coordinator, a GraphQL gateway, or a GA4GH Beacon v2-compliant REST API — that presents the federation as if it were one database, even though the underlying tables are scattered across AWS Redshift/Athena, BigQuery, and on-prem PostgreSQL or Hail/VCF stores.

A representative submission looks like:

SELECT ancestry, COUNT(*) AS carriers, COUNT(*)*1.0/SUM(COUNT(*)) OVER () AS freq FROM cohort.variants WHERE variant_id = 'chr7:117559590:ATCT:A' AND consent_category = 'GRU' GROUP BY ancestry;

Note the consent_category predicate — federated genomic queries routinely filter on GA4GH Data Use Ontology (DUO) codes (e.g. "General Research Use", "Health/Medical/Biomedical Research Use Only") directly in the query, so consent enforcement happens per-site, per-row, before any aggregate is even computed.

Authentication and query admission control

Before a query is admitted, the federation layer performs identity federation (commonly OAuth2/OIDC via GA4GH Passport or eduGAIN/InCommon for academic consortia) and checks the requester's visas against each site's access policy. A researcher may be authorized for allele-frequency aggregates across all sites but denied row-level access everywhere — the query planner enforces this at admission time, not as an afterthought.

Admission control also applies rate limiting and cost estimation: because a naive query could implicitly request a full-cohort join across every site, the planner estimates the query's fan-out and rejects or down-samples queries that would force disproportionate compute at a low-resource on-prem site. This budget-aware admission is analogous to query governors in Trino and BigQuery's slot-based cost controls, extended to per-site fairness across a federation of unequal partners.

Query Planning & Predicate Push-down

Once admitted, the federation planner must turn one logical query into N physically executable sub-queries — one per site — each rewritten to that site's schema, dialect, and access constraints. This planning stage decides exactly how much computation happens locally at each site versus centrally at the coordinator, and it is where the "raw data never moves" guarantee is architecturally enforced.

• 80–150 ms: Planning latency (typical) (catalog lookup + rewrite)
• >90%: Push-down coverage (modern engines) (of filter/aggregate ops)
• ~4,000: Schema catalog entries (large federation) (columns across sites)
• 3–8: Sub-queries per logical query (avg) (one per participating site)

Catalog resolution and query rewrite

The planner first resolves which sites actually hold data relevant to the query — not every site necessarily has the requested variant, ancestry group, or phenotype. This resolution step queries a federation-wide schema catalog (conceptually similar to a Hive Metastore or Trino connector catalog) that tracks table shapes, column types, and — critically for genomics — the reference genome build (GRCh37 vs GRCh38) and variant normalization convention used at each site.

Once relevant sites are identified, the logical query is rewritten into each site's native dialect: a BigQuery sub-query for the GCP site, an Athena/Redshift Spectrum sub-query for the AWS site, and a Hail-on-Spark or plain SQL sub-query for the on-prem site. Column names, coordinate systems, and even allele representations (left-aligned vs. non-normalized indels) are harmonized during rewrite so the eventual merge is comparing like with like.

Predicate and aggregate push-down

Push-down is the core optimization that keeps genomic federation tractable: rather than pulling every row to the coordinator and filtering centrally, the planner pushes WHERE clauses, GROUP BY aggregates, and even partial statistical computations (allele counts, Hardy-Weinberg statistics, case/control tabulations) down into each site's own execution engine.

For a query like "allele frequency of variant X stratified by ancestry," push-down means each site computes its own local COUNT() and GROUP BY ancestry entirely inside its VPC — the coordinator only ever receives per-ancestry counts, never the underlying genotype rows. Modern federated SQL engines (Trino, Presto, Dremio) push down the large majority of relational operators automatically; genomic-specific engines add push-down for variant-level operators (region overlap, LD calculation, PLINK-style association tests) that have no direct SQL equivalent.

Effective push-down typically reduces coordinator-side row processing by 3–5 orders of magnitude: a query touching 40 million genotype rows across sites returns fewer than 200 aggregate rows to the coordinator for final merge.

Transport layer — Arrow Flight and DRS-resolved compute

When a sub-query does need to move intermediate columnar results between a site's execution engine and the federation coordinator (for example, a partial hash-aggregation result too large for a simple REST payload), transport increasingly uses Apache Arrow Flight rather than JSON or CSV over REST. Arrow Flight moves columnar batches with zero-copy (de)serialization over gRPC, which matters even for small aggregate payloads because genomic queries commonly return multi-dimensional stratifications (ancestry × sex × age-band × variant) that can still reach tens of thousands of aggregate cells.

For any operation that genuinely requires object-level data access — e.g., fetching a specific consented CRAM slice for secondary confirmation — resolution goes through the GA4GH Data Repository Service (DRS), which returns a signed, time-limited access URI rather than a copy of the file itself, keeping custody of raw objects with the originating site even when access is granted.

Federation protocol comparison

Distributed Execution Across Sites

With sub-queries planned and pushed down, each site — the AWS region, the GCP region, and the on-prem clinical system — executes independently against its own private cohort. This stage is the architectural heart of federation: three physically and administratively separate execution engines run in parallel, each seeing only its own slice of the data, with no shared memory, shared storage, or shared raw-data access between them.

• ~2.6×: Parallel execution speedup (3 sites) (vs. sequential querying)
• 8M–20M: Rows scanned per site (typical) (genotype/phenotype rows)
• Spark/Hail, BigQuery, Athena: Local compute engines observed (per-site heterogeneity is normal)
• 0 bytes: Cross-site raw data transfer (by architectural guarantee)

Site autonomy and heterogeneous execution engines

Each participating site retains full autonomy over how it executes its sub-query. The AWS site might run its slice as an Athena query over Parquet-encoded genotype matrices in S3; the GCP site might run the equivalent as a BigQuery SQL job over a partitioned, clustered table; the on-prem site — often the most resource-constrained — might run a Hail-on-Spark job against a VCF-derived Hail MatrixTable, or a lighter PLINK2 process against a local binary genotype file.

The federation coordinator does not need any of these engines to be identical; it only needs each site's connector to accept a pushed-down sub-query and return results in a common columnar schema. This heterogeneity is a feature, not a limitation — it lets each institution keep its existing infrastructure, compliance posture, and cost model unchanged while still participating in cross-cloud analysis.

The zone boundary — why raw data cannot leak

The privacy guarantee of federated execution is architectural, not merely policy-based. Each site's execution engine runs entirely inside its own network boundary — a VPC with no inbound raw-data path, an on-prem subnet behind an institutional firewall — and the sub-query contract only ever requests aggregate or derived outputs, never row-level SELECT * results. Because the query planner (Stage 2) already rewrote every sub-query to request only counts, sums, and stratified aggregates, there is no code path in the executing engine that would emit an individual genotype row across the zone boundary.

This is reinforced defense-in-depth: egress filtering at the network layer blocks large data-plane transfers by default, output size caps reject any sub-query response above a small aggregate-sized threshold, and audit logging records every query and response size per site so that unusually large egress is immediately visible to institutional data stewards.

In production genomic federations, per-site egress caps are commonly set below 5 MB per query response — several orders of magnitude smaller than even a single-sample VCF, making inadvertent raw-data egress structurally difficult, not just policy-forbidden.

Handling stragglers and site-level failure

Parallel cross-cloud execution introduces classic distributed-systems challenges: network latency between the coordinator and an on-prem site behind a slow institutional link, transient site unavailability during maintenance windows, or one site's query taking substantially longer due to a larger or less-indexed cohort. The coordinator applies straggler mitigation strategies borrowed from distributed query engines — per-site timeouts, partial-result acceptance with explicit confidence annotation ("answer computed from 2 of 3 sites"), and speculative retries for sites that appear to be lagging rather than failed.

Because each site's aggregate contribution is independently valid and combinable, the federation can gracefully degrade: a researcher can choose to accept a partial cross-site answer immediately or wait for full quorum, a flexibility that a centralized data warehouse — where the query either runs against complete data or not at all — does not offer in the same way.

Partial Result Aggregation & Privacy Budget

As each site finishes local execution, it returns not rows but summaries — counts, frequencies, means, or noised statistics — to the federation coordinator. This stage combines those partials into a single cross-site answer, and it is where differential privacy, secure aggregation, and small-cell suppression are applied to ensure the combined result cannot be reverse-engineered into individual-level information.

• 0.1–1.0: Typical differential privacy budget (ε) (per released statistic)
• <10: Small-cell suppression threshold (individuals per cell)
• ~2–40 KB: Aggregate payload size (per site) (vs. GB-scale raw slice)
• 20–60 ms: Merge/combine latency (coordinator-side)

Combining functions for cross-site aggregates

Different statistics require different combining logic once partials arrive from each site. Simple additive statistics — allele counts, case/control tallies — combine by direct summation across sites. Ratio statistics such as allele frequency require sites to return both numerator and denominator separately (carrier count and total genotyped count), because averaging site-level frequencies directly would incorrectly weight small and large sites equally.

More complex statistics — meta-analytic effect sizes for a GWAS-style association test, for instance — use inverse-variance-weighted fixed- or random-effects meta-analysis, the same combining approach used in traditional multi-cohort GWAS meta-analysis tools (METAL, GWAMA), simply automated inside the federation coordinator rather than performed manually by a statistician after the fact.

Differential privacy and small-cell suppression

Returning even exact aggregate counts can leak individual-level information when cells are small — if a stratified query returns "1 carrier" in a rare ancestry×phenotype cell, that single individual is effectively re-identified within a known cohort. Federated genomic engines apply two complementary defenses:

• Small-cell suppression: any aggregate cell computed from fewer than a site-configured threshold (commonly 5–10 individuals) is withheld or merged into a broader category before leaving the site.

• Differential privacy (DP): calibrated noise — typically Laplace or Gaussian mechanism noise — is added to released statistics, with a formal privacy budget ε bounding how much any single query can narrow the space of possible underlying datasets. Each site tracks a cumulative ε spend per requester, so repeated querying cannot slowly reconstruct individual records by averaging out noise across many queries.

These are applied at the site, before the aggregate ever leaves the zone boundary — the coordinator never sees, and cannot request, the unnoised exact value.

Under a per-query ε of 0.5 and a cumulative per-researcher budget of ε=10, a federation can safely answer roughly 20 independent statistical queries about the same cohort before further queries must be throttled or additional consent obtained — a concrete, auditable privacy accounting model rather than an informal promise.

Result caching and staleness trade-offs

Because re-running a full cross-site execution for an identical or near-identical query is costly — both in compute and in privacy budget consumption — the coordinator caches combined results keyed by query signature, subject to a configurable Time-To-Live (TTL). A longer TTL reduces average query latency and avoids unnecessary re-spend of differential-privacy budget for repeated queries, but risks returning results that do not reflect newly added cohort data (many biobanks ingest new participants continuously).

Operationally, federations set TTLs per query class: static reference statistics (e.g. population allele frequencies for common variants) may cache for hours to a day, while queries touching actively recruiting cohorts use short TTLs (minutes) so newly consented participants are reflected promptly. Cache invalidation is typically event-driven — a site emits an invalidation signal to the coordinator when its underlying table changes materially, rather than relying on TTL expiry alone.

Federated Result Return to the Researcher

The final stage delivers the combined, privacy-preserving answer back to the researcher's notebook or dashboard — typically within seconds of submission, despite having orchestrated compute across three independently governed infrastructures. What the researcher receives looks like a single query result; what actually happened was a fully distributed computation that never centralized a single raw genomic record.

• 1.5–4 s: End-to-end query latency (typical) (submission to delivered result)
• 0 bytes: Raw data centralized (across the full pipeline)
• 1 per site + 1 coordinator: Audit log entries generated (per query, full provenance)
• 0.8–3+ TB: Equivalent centralized transfer avoided (per representative query)

Provenance, reproducibility and result annotation

The delivered answer is returned with structured provenance metadata: which sites contributed, the query signature, the differential-privacy parameters applied, whether the result came from cache or fresh execution, and confidence flags if any site failed to respond within its timeout. This provenance record is essential for downstream scientific reproducibility — a published allele-frequency estimate needs to state not just the number, but the federation configuration and privacy parameters under which it was computed, analogous to how a meta-analysis paper must report which cohorts and models contributed.

Many federation platforms render this provenance directly alongside the numeric answer in the researcher's notebook (a JSON or dataframe attribute set), so the answer is self-documenting without requiring the researcher to separately query an audit system.

What the researcher can — and cannot — do with the result

Because only governed aggregates were ever returned, the researcher's downstream analysis is inherently bounded by what aggregation was requested. This is a deliberate constraint: a researcher cannot retroactively "drill down" into a returned aggregate to recover individual records, because those records were never transmitted to begin with — there is nothing to drill into. Any follow-up analysis requiring finer granularity must be submitted as a new, separately governed query, subject to the same push-down, suppression, and privacy-budget controls as the original.

This stands in contrast to legacy data-sharing models where a full extract is downloaded once and then queried arbitrarily offline — an approach that is operationally convenient but makes enforcing evolving consent, withdrawal requests, and privacy budgets effectively impossible after the fact, since copies proliferate beyond the originating institution's control.

Scaling federations — beyond three sites

Production genomic federations regularly extend well past a single AWS + GCP + on-prem triad. Multi-national consortia such as the GA4GH-aligned Beacon Network, the Global Alliance's Genomic Data Infrastructure pilots, and disease-specific federations (e.g. rare-disease networks spanning dozens of hospitals) coordinate tens of sites simultaneously. At that scale, the coordinator itself is often deployed redundantly across regions, sub-query fan-out is batched to avoid overwhelming smaller institutional sites, and hierarchical aggregation (regional coordinators aggregating before a global merge) reduces coordinator-side load.

The architectural principle stays constant regardless of scale: the query travels, the data doesn't, and only small, governed, privacy-bounded answers ever cross an institutional boundary.

Federations that scale past ~10 sites typically introduce a two-tier aggregation hierarchy — regional coordinators merge nearby sites first — cutting coordinator fan-in load by roughly 60–70% compared to flat single-tier aggregation.