ChEMBL Data Mining for Drug Discovery ML

The quality of a QSAR or property-prediction model is fundamentally bounded by the quality of the underlying bioactivity data. ChEMBL aggregates data extracted from medicinal chemistry literature, with varying assay confidence levels (0–9), multiple activity types (IC50, Ki, Kd, EC50, AC50), and diverse standard relation operators. Systematic curation before modelling is non-negotiable.

ChEMBL API Data Extraction

import requests
import pandas as pd
import time


CHEMBL_BASE = "https://www.ebi.ac.uk/chembl/api/data"


def fetch_target_bioactivities(
    chembl_target_id: str,
    activity_types: list[str] = ("IC50", "Ki"),
    assay_type: str = "B",          # B=Binding, F=Functional, A=ADME
    confidence_score_min: int = 4,
    standard_units: str = "nM",
    max_rows: int = 10_000,
) -> pd.DataFrame:
    """
    Fetch binding/functional bioactivity data for a specific target from ChEMBL API.

    Args:
        chembl_target_id: ChEMBL target ID, e.g. "CHEMBL279" (EGFR)
        activity_types: Activity measurement types to include
        assay_type: B=binding, F=functional, A=ADMET, T=toxicity, P=physicochemical
        confidence_score_min: Min assay confidence (4=single protein, 7=direct single, 9=curated)
        standard_units: Only include activities in these units
        max_rows: Maximum rows to retrieve

    Returns:
        DataFrame with standardised bioactivity records.
    """
    all_records = []
    offset = 0
    limit = 1000

    while offset < max_rows:
        params = {
            "target_chembl_id": chembl_target_id,
            "assay_type": assay_type,
            "standard_units": standard_units,
            "format": "json",
            "limit": min(limit, max_rows - offset),
            "offset": offset,
        }
        r = requests.get(f"{CHEMBL_BASE}/activity.json", params=params, timeout=30)
        r.raise_for_status()
        data = r.json()
        activities = data.get("activities", [])
        all_records.extend(activities)

        if len(activities) < limit or offset + limit >= max_rows:
            break
        offset += limit
        time.sleep(0.3)  # polite rate-limiting for public API

    df = pd.DataFrame(all_records)
    if df.empty:
        return df

    # Filter by activity type and confidence score
    if "standard_type" in df.columns:
        df = df[df["standard_type"].isin(activity_types)]
    if "assay_confidence_score" in df.columns:
        df = df[pd.to_numeric(df["assay_confidence_score"], errors="coerce") >= confidence_score_min]

    return df.reset_index(drop=True)

IC50 Curation and pChEMBL Computation

import numpy as np
import pandas as pd
from rdkit import Chem
from rdkit.Chem.MolStandardize import rdMolStandardize


def curate_chembl_activities(
    df: pd.DataFrame,
    keep_relations: tuple[str, ...] = ("=",),
    pchembl_min: float = 4.0,
    pchembl_max: float = 11.0,
    remove_duplicates: bool = True,
    conflict_threshold_log_units: float = 1.0,
) -> pd.DataFrame:
    """
    Apply standard curation rules to a raw ChEMBL activity DataFrame.

    Curation steps (Zdrazil et al. 2024 ChEMBL best practices):
    1. Keep only exact measurements (relation = "="); drop ">", "<" censored values.
    2. Compute pActivity = −log10(IC50_nM × 1e-9) = 9 − log10(IC50_nM)
    3. Filter to physically plausible range [pChEMBL 4–11]
    4. Standardise SMILES (strip salts, neutralise, canonical SMILES)
    5. Remove duplicate molecule-target pairs where pActivity spread > conflict_threshold
       by keeping the median and flagging high-conflict compounds

    Args:
        df: Raw ChEMBL activity DataFrame from fetch_target_bioactivities()
        keep_relations: Standard relation operators to retain
        pchembl_min: Lower bound for pActivity filter
        pchembl_max: Upper bound for pActivity filter
        remove_duplicates: Aggregate duplicate mol-target measurements
        conflict_threshold_log_units: Max std dev across replicates before flagging

    Returns:
        Curated activity DataFrame with pchembl_value column.
    """
    df = df.copy()

    # Step 1: Keep exact relations
    if "standard_relation" in df.columns:
        df = df[df["standard_relation"].isin(keep_relations)]

    # Step 2: Numeric value, convert to pChEMBL
    df["standard_value"] = pd.to_numeric(df.get("standard_value", pd.Series(dtype=float)), errors="coerce")
    df = df.dropna(subset=["standard_value"])
    df = df[df["standard_value"] > 0]
    df["pchembl_value"] = 9.0 - np.log10(df["standard_value"])

    # Step 3: Plausibility filter
    df = df[(df["pchembl_value"] >= pchembl_min) & (df["pchembl_value"] <= pchembl_max)]

    # Step 4: SMILES standardisation
    if "canonical_smiles" in df.columns:
        df["smiles_std"] = df["canonical_smiles"].apply(_standardise_smiles)
        df = df.dropna(subset=["smiles_std"])
    else:
        df["smiles_std"] = df.get("smiles", pd.Series(dtype=str))

    # Step 5: Deduplicate by smiles_std + target
    if remove_duplicates and "smiles_std" in df.columns:
        key_cols = ["smiles_std"]
        if "target_chembl_id" in df.columns:
            key_cols.append("target_chembl_id")
        agg = df.groupby(key_cols)["pchembl_value"].agg(
            pchembl_mean="mean", pchembl_std="std", n_measurements="count"
        ).reset_index()
        agg["high_conflict"] = agg["pchembl_std"].fillna(0) > conflict_threshold_log_units
        df = agg.rename(columns={"pchembl_mean": "pchembl_value"})

    return df.reset_index(drop=True)


def _standardise_smiles(smiles: str) -> str | None:
    try:
        mol = Chem.MolFromSmiles(smiles)
        if mol is None:
            return None
        lfc = rdMolStandardize.LargestFragmentChooser()
        mol = lfc.choose(mol)
        uncharger = rdMolStandardize.Uncharger()
        mol = uncharger.uncharge(mol)
        return Chem.MolToSmiles(mol) if mol else None
    except Exception:
        return None

ChEMBL Assay Confidence Levels

Key Takeaways

• Always filter ChEMBL data to confidence score ≥7 for primary QSAR work. Score 4 data can supplement larger datasets but carries multi-protein and cell-line assay noise that degrades model calibration.
• Censored values (IC50 >10 µM or <1 nM) represent 20–40% of ChEMBL assay readouts. Naively dropping them discards useful structural information; consider ordinal regression or survival-analysis-inspired approaches to include censored data.
• Scaffold-based splitting is mandatory to measure real generalisation. Random splitting artificially inflates performance by allowing nearly-identical molecules in both train and test sets. Use Bemis-Murcko scaffold hashing or Butina clustering for proper splits.
• The pChEMBL value (−log₁₀[M]) unifies IC50, Ki, Kd, and EC50 across concentration units, enabling multi-assay training. However, different assay types measure different quantities — always stratify or encode assay type as a feature when mixing.

Try it live

Explore a related model hands-on: open Molecule Screening Engine — Similarity Search Live and change its parameters while it runs. Everything stays in your browser.