🤖 Federated Learning Across Pharma Partners

Partner Silos — Confidential Molecule Data That Never Leaves Home

Every large pharmaceutical company sits on a proprietary chemical library built over decades — millions of tested compounds, their bioactivity against thousands of targets, toxicity readouts, ADMET profiles. This data is among the company's most valuable and closely guarded assets. Historically, that meant it could never be used to train a shared machine learning model. Federated learning was built to change that equation without moving a single molecule off-site.

• 10: MELLODDY consortium partners (pharma companies)
• >21 M: MELLODDY compounds pooled (virtually) (never physically merged)
• 0: Confidential molecules shared (raw structures transmitted)
• €16 M+: IMI project funding (2019 – 2022)

Why proprietary chemical libraries stay locked away

Each major pharma company maintains an internal compound collection of typically 1–5 million proprietary molecules, annotated with results from thousands of high-throughput screening (HTS) campaigns against internal and licensed biological targets. This chemical-and-bioactivity matrix is the direct product of decades of R&D spend — often tens of billions of dollars — and represents durable competitive advantage in drug discovery.

Sharing this data openly would leak information about which chemical series a company is pursuing, which targets it considers promising, and which compounds have failed for toxicity or selectivity reasons — commercially and strategically sensitive information, independent of any patient-privacy concern. Traditional collaboration models therefore stalled: either partners shared nothing and each trained a weaker model on a fraction of the true chemical space, or they attempted risky bilateral data-sharing agreements that rarely scaled past two or three companies.

A single company's internal QSAR (quantitative structure-activity relationship) model, trained only on its own library, systematically underperforms on chemical space it has never explored — exactly the blind spot a competitor's library could fill, if only it could be used without ever being seen.

The MELLODDY project — ten competitors, one shared model

MELLODDY (Machine Learning Ledger Orchestration for Drug Discovery) was an Innovative Medicines Initiative (IMI) project running 2019–2022, bringing together ten pharmaceutical companies — including Novartis, AstraZeneca, Janssen (Johnson & Johnson), Merck KGaA, GSK, Bayer, Boehringer Ingelheim and Amgen — alongside AI partners Owkin, and academic and technical collaborators.

The goal was audacious: train a single multi-task QSAR model on the combined chemical and bioactivity knowledge of all ten companies' internal libraries — together representing more than 21 million compounds and upward of 40 pharmacologically relevant assay panels — without any partner ever seeing another partner's structures or results. This is precisely the scenario federated learning was designed to solve, and MELLODDY became its largest real-world validation in the pharmaceutical industry.

What "never shared" means in practice

In the silo stage, each partner's infrastructure holds three things locally and only locally: (1) chemical structures (SMILES / molecular graphs), (2) assay readouts (IC50, binding affinity, ADMET, toxicity flags), and (3) any derived features computed from them. None of this is copied, uploaded, or exposed via API to the aggregator or to other partners at any stage of the federated learning process.

What does eventually leave the silo — starting in later stages — is only a numeric summary of what the model learned from that data: gradient vectors or weight deltas. These carry far less identifiable information about individual molecules, and can be further protected with encryption and privacy-preserving aggregation before they are sent anywhere at all.

Global Model Initialization & Distribution

Before any local training happens, every partner needs to start from the exact same model. A central aggregator server defines the shared architecture — a multi-task neural network in MELLODDY's case — initializes its weights, and pushes an identical copy out to every participating partner. This round-trip of weights, not data, is the fundamental unit of communication in federated learning.

• 2017: FedAvg algorithm published (McMahan et al., Google)
• 10⁶–10⁸: Typical model parameter count (weights distributed per round)
• 10–500: Federated rounds in a full run (depends on task convergence)
• Owkin: Substra ledger platform (blockchain-secured orchestration)

FedAvg — the algorithm that makes this possible

Federated Averaging (FedAvg), introduced by McMahan et al. in 2017 ("Communication-Efficient Learning of Deep Networks from Decentralized Data"), is the foundational algorithm behind almost all practical federated learning systems, including MELLODDY's. Each communication round proceeds in four steps:

1. The server selects a subset (or all) of the available partners and sends them the current global model weights w_t. 2. Each partner k initializes its local model with w_t and trains it on its own private data for E local epochs, producing updated local weights w_t^k. 3. Each partner sends back only w_t^k (or the delta Δw_t^k = w_t^k − w_t) — never the data used to produce it. 4. The server aggregates: w_{t+1} = Σ_k (n_k/n) · w_t^k, a weighted average by each partner's local dataset size n_k.

This loop repeats for many rounds, with the global model asymptotically approaching the performance of a hypothetical model trained on all partners' data pooled together — despite that pooled dataset never actually existing anywhere.

Why distribution, not just training, needs an architecture

For weight-level averaging to make sense, every partner must train the identical model architecture with identically shaped weight tensors and identically ordered task heads. In MELLODDY, this meant designing a shared multi-task deep neural network in advance — a common molecular featurization (e.g. Morgan/ECFP fingerprints), a shared backbone of fully connected layers, and per-assay output heads — and agreeing on it collectively before any training began, since re-architecting later would break weight compatibility across all ten companies' pipelines.

Blockchain-secured orchestration on Substra

MELLODDY ran its federated rounds on Substra, an open-source federated learning platform built by Owkin, with a blockchain ledger (Substra uses a permissioned Hyperledger Fabric-style ledger) recording every round's metadata — which model version was sent, which partners trained on it, when updates were returned — without ever recording the underlying data itself. This gave the consortium an auditable, tamper-evident trail proving that no partner's raw compound data crossed the network, which was essential for legal sign-off given the antitrust sensitivity of ten competing companies collaborating at all.

Because the ledger only ever transports model weights and cryptographically signed round metadata, MELLODDY's legal teams could certify that the entire multi-year collaboration never once transmitted a proprietary chemical structure between competitors — a claim that would be impossible to make about any conventional data-pooling consortium.

Local Training on Private Data — Learning Without Leaving Home

This is where the actual learning happens — entirely inside each partner's own infrastructure. The freshly received global model is trained for several epochs against that partner's private molecules and assay results, exactly as if it were a normal in-house model. The only thing that will ever leave the building afterward is a small update describing how the weights changed.

• 1–20: Local epochs per round (typical) (before returning updates)
• 0 bytes: Raw data leaving the partner site (strictly on-premise compute)
• on-prem GPU cluster: Compute location (behind partner firewall)
• high: Non-IID data challenge (each partner's chemical space differs)

Standard supervised training, unusual deployment boundary

From a pure machine-learning standpoint, local training in round t looks exactly like conventional supervised learning: forward pass, loss computation against known assay labels, backpropagation, gradient descent steps for E epochs (or a fixed number of mini-batch steps) using the partner's private molecules and their measured bioactivities. What is unusual is the deployment boundary — this training loop runs entirely inside the partner's own data center, orchestrated by client software (e.g. Substra's local node) that receives the model, runs training against locally mounted data, and returns only the resulting weights.

The non-IID problem — every partner's chemistry looks different

A core theoretical challenge in federated learning is that partner datasets are typically non-IID (not independently and identically distributed): one company may be chemically enriched in kinase inhibitors, another in GPCR-targeted fragments, another in large biologics-adjacent molecules. Standard FedAvg convergence guarantees assume roughly similar data distributions across clients; strong non-IID skew can slow convergence or bias the aggregated model toward partners with larger or more influential local updates.

MELLODDY and similar consortia mitigate this with techniques such as weighting contributions by validated local performance rather than raw dataset size alone, applying regularization terms (e.g. FedProx-style proximal terms that penalize local models drifting too far from the global model), and running per-assay evaluation to detect when a task's aggregated performance is being dragged down by distributional mismatch.

The free-rider problem and contribution incentives

Because the global model benefits every participant equally regardless of how much useful signal their local data contributed, federated consortia face a free-rider risk: a partner with a small or low-quality dataset receives the same improved model as a partner that trained on millions of high-quality, diverse compounds. MELLODDY addressed this partly through its consortium governance (membership and cost-sharing agreements) and partly through weighting schemes that reward larger, more informative local updates — an active research area, since verifying the "quality" of an update without seeing the underlying data is itself a nontrivial cryptographic and statistical problem.

Secure Aggregation — Only Encrypted Model Updates Travel

Once local training finishes, each partner has a new set of weights that implicitly encodes what its private molecules taught the model — but sending those weights back in the clear could still leak information. Secure aggregation techniques let the server compute the average update across all partners without ever seeing any individual partner's update in isolation.

• HE, SMPC: Techniques in production use (homomorphic encryption, secure multi-party computation)
• 10–100×: Update compression typical (sparsification / quantization)
• tunable ε: Differential privacy noise budget (often ε < 10 per round)
• 0: Individual updates visible to server (only the aggregate is decrypted)

Why gradients alone are not automatically safe

Even though weight updates are vastly more abstract than raw molecules, research on gradient inversion and membership inference attacks has shown that raw gradients can, in some circumstances, leak information about the underlying training examples — especially with small batch sizes or highly expressive models. In a setting with ten direct commercial competitors, "probably safe" is not an acceptable bar. MELLODDY therefore layered cryptographic protections on top of the basic FedAvg protocol rather than transmitting plain-text weight deltas.

Homomorphic encryption and secure multi-party computation

Two complementary cryptographic tools are used in production federated systems:

• Homomorphic encryption (HE): allows the server to perform arithmetic — specifically, summation — directly on encrypted weight updates without decrypting them. Each partner encrypts its local update before sending it; the server homomorphically sums the encrypted values across all partners; only the final aggregate is ever decrypted, using keys that require multi-party cooperation to unlock.

• Secure multi-party computation (SMPC): partners jointly compute the aggregate function (the weighted average) by splitting their updates into cryptographic secret shares distributed across multiple non-colluding servers. No single server, and no single partner, ever reconstructs another partner's individual update — only the final sum is reconstructable, and only when contributions from enough partners are combined.

Both approaches can be combined with differential privacy — deliberately adding calibrated statistical noise to updates — to provide a formal, quantifiable bound (ε) on how much any single partner's data could have influenced the aggregate, closing the door on even a computationally unbounded adversary inferring individual contributions.

With secure aggregation in place, the central server mathematically cannot isolate any one partner's update — it can only ever compute and decrypt the sum across a minimum quorum of partners, making it cryptographically meaningless to try to reverse-engineer a single competitor's proprietary chemistry from the aggregation step.

FedAvg with secure aggregation, formally

The averaging step itself is unchanged mathematically — w_{t+1} = Σ_k (n_k/n) · w_t^k — but it is now evaluated inside an encrypted or secret-shared computation rather than in plain text on the server. Communication efficiency remains important: weight updates are typically compressed via sparsification (sending only the largest-magnitude coordinates) and quantization (reducing floating-point precision) before encryption, since encrypted payloads and multi-party protocols add computational and bandwidth overhead on top of already large parameter vectors.

Improved Global Model — Collective Chemical Knowledge, Zero Shared Data

After the aggregated update is applied, the global model is redistributed to every partner and the cycle repeats. Round after round, the model implicitly absorbs the combined chemical intuition of all participating companies' proprietary libraries — without any partner ever gaining visibility into another's molecules. The result, reported across MELLODDY's runs, is a multi-task model that generalizes measurably better than any partner could achieve training alone on its own library.

• reported gains: Accuracy gain vs single-partner models (across most partners and assays)
• 10 / 10: Partners retaining full data control (throughout the project)
• 21 M+ compounds: Combined virtual chemical space (40+ pharmacological assays)
• yes: Antitrust-compliant structure (no raw data pooling, ever)

Convergence — more rounds, more partners, better generalization

As federated rounds accumulate, the global model's effective training signal approaches that of a model trained on the union of every partner's chemical space — even though that union is never materialized as an actual dataset anywhere. MELLODDY's own published results (2021–2022) reported that federated multi-task models outperformed each individual partner's locally-trained-only baseline on a majority of the shared assay panels, with the gains most pronounced for smaller partners and for chemically under-represented target classes, exactly where a single company's own library would have been thinnest.

More partners generally raise the ceiling on final accuracy, since they widen the effective chemical space the model implicitly learns from; more aggregation rounds let that signal fully propagate and let local overfitting to any one partner's idiosyncratic chemistry average out.

Regulatory and competitive motivations for the whole exercise

None of this would have been attempted without strong regulatory and competitive incentives on both sides. On the IP-protection side, no participating company could risk exposing its proprietary compound library or screening results to nine direct competitors — the entire architecture exists specifically to make that exposure structurally impossible, not merely contractually forbidden. On the antitrust side, EU and US competition law restricts direct data or strategy sharing between competitors; a federated protocol where only encrypted, aggregated model statistics move between parties — with the process independently audited and blockchain-logged — gave legal teams a defensible position that no competitively sensitive information was exchanged.

On the incentive side, every partner individually benefits: a company with a narrow chemical franchise gains predictive power over chemical space it has never explored, without paying the cost — in money, risk, or IP — of actually acquiring that data.

Open challenges that remain

Federated learning at this scale is not a solved problem. Communication overhead remains substantial when models have hundreds of millions of parameters and dozens of rounds are needed — bandwidth and encryption costs scale with both. Non-IID data across partners can bias the aggregate toward whichever partners' chemistry happens to dominate a given round's weighting. The free-rider problem persists: partners contributing less informative data still receive the fully improved model. And verifying that no partner is subtly attempting to reconstruct others' data from repeated rounds of aggregate statistics remains an active area of cryptographic and differential-privacy research, not a fully closed question — every additional round of aggregate information technically leaks some infinitesimal amount, which is precisely why formal privacy budgets (ε) are tracked and capped over a project's lifetime.

MELLODDY formally concluded in 2022 having demonstrated, at unprecedented industrial scale, that ten direct pharmaceutical competitors could jointly improve a shared predictive model over their combined 21-million-plus compound chemical space — while independently verified audits confirmed zero raw molecular structures ever crossed company boundaries.

Data-collaboration models compared