Building the Data Foundation for Machine Learning: A Data Quality Score Framework

Why "we don't have data" is almost never true

Roughly 80% of the effort in a typical machine learning project goes into collecting, cleaning, and preparing data; only about 20% goes into building and tuning models. A sophisticated algorithm trained on poor data will consistently lose to a simple algorithm trained on clean, relevant data — which is why the data foundation, not the modelling technique, is usually the deciding factor in whether an ML project succeeds.

Organisations that believe they "don't have data" almost always do — it is simply scattered across a CRM, an accounting system, spreadsheets on individual laptops, web analytics tools, email platforms, server logs, and even paper archives that could be digitised with OCR. The first practical step is a data inventory: a table listing every source, what it contains, where it lives, its approximate volume, a rough quality score, and how accessible it is. Most teams are surprised by how much material this exercise surfaces once departmental silos are mapped in one place.

Five criteria for judging data quality

Data readiness for machine learning can be assessed along five independent dimensions, each of which fails a project in a different way.

Volume — how much data is available relative to what the technique requires. Tabular classification or regression models typically need on the order of 1,000 to 10,000+ rows depending on feature count; anomaly detection needs tens of thousands of normal examples to characterise what "normal" looks like; recommendation systems need tens of thousands of interactions minimum because the user-item matrix is inherently sparse. When volume falls short, transfer learning, data augmentation, synthetic data generation, or simply a less data-hungry model are the standard fallbacks.

Quality — the presence of missing values, duplicate records, inconsistent formatting (the same city spelled four different ways), outliers from data-entry errors, incorrect data types, invalid values, stale records, class imbalance, data leakage, and sampling bias. Each of these problems degrades model performance in a distinct way, and most raw operational data exhibits several of them simultaneously.

Relevance — whether the available fields actually answer the question being asked. A churn model built only on demographic fields (age, location) without behavioural signals (recency, frequency, engagement, support tickets) will typically perform little better than random guessing, because the features that actually predict churn were never collected.

Accessibility — whether the data can be retrieved when needed. Instant API access is the gold standard; scheduled exports are workable; manual extraction that requires filing an IT ticket introduces delays that make automation impractical; data that is technically available but legally or practically unreachable is effectively unusable.

Freshness — how quickly the data goes stale relative to how fast the underlying behaviour changes. A fraud model needs real-time data; a demand forecast can tolerate a day's lag; a credit-scoring model can often work from monthly-refreshed data because financial circumstances change more slowly. A useful diagnostic is a staleness test: train on data that is six months old, evaluate on current data, and if accuracy drops by more than roughly 10%, the underlying signal decays faster than the refresh cycle can track.

Turning quality into a single number: the Data Quality Score

A Data Quality Score (DQS) averages five component scores — completeness, accuracy, consistency, timeliness, and validity, each expressed as a percentage — into a single tracking metric. A worked example: completeness 85% (15% of records have missing values), accuracy 90% (10% contain errors found via manual sampling), consistency 70% (many duplicates and inconsistent formats), timeliness 95% (data refreshes daily), validity 80% (20% fail business-rule checks). Averaged, DQS = (85+90+70+95+80)/5 = 84%.

As a rough interpretation guide: 90–100% is excellent and ready for ML work with minimal further cleaning; 70–89% is acceptable but needs a dedicated cleaning pass; 50–69% is poor and needs substantial remediation before modelling begins; below 50% means the underlying collection processes need fixing first, because no amount of downstream cleaning will compensate for a broken pipeline feeding in bad data continuously.

Seven strategies for collecting data that doesn't yet exist in usable form

Where genuinely new data collection is needed, seven approaches cover most situations. Integrating existing systems via connectors (open-source tools with hundreds of pre-built connectors, or managed ETL services) pulls data automatically from a CRM, analytics platform, or e-commerce system into a central warehouse. Event logging adds structured tracking for user actions, system events, and context at the moment they happen, rather than reconstructing them after the fact. Surveys and feedback (NPS, CSAT, star ratings) generate explicit ground-truth labels that are otherwise hard to infer from behaviour alone. Web scraping of public data supports competitive pricing and market research, subject to terms-of-service and privacy constraints. Data marketplaces let a team buy a ready-made dataset rather than collect one from scratch, useful for filling gaps like historical market pricing. Synthetic data generation — rule-based, statistical, or generative-model-based — helps when real examples are scarce, especially for testing, privacy protection, or balancing a minority class. Crowdsourced labelling turns unlabelled images or text into training data through distributed human annotation, typically with each item reviewed by three to five annotators and reconciled against a small "golden set" of known-correct answers to catch low-quality labellers.

A six-step cleaning pipeline

Once raw data is assembled, a consistent cleaning sequence turns it into something a model can learn from reliably. Start with exploratory data analysis — summary statistics, missing-value counts, duplicate counts, distribution histograms, and a correlation matrix — to understand what is actually in the dataset before changing anything. Handle missing values next, choosing between deletion (appropriate when under roughly 5% of records are affected and missingness looks random), imputation (mean/median for numeric fields, mode for categorical, or a predictive imputer such as k-nearest-neighbours for more structure), explicitly flagging the fact that a value was missing as its own feature (useful when missingness itself is predictive — for instance, customers who never completed their profile disproportionately churn), or creating an explicit "unknown" category for categorical fields. Remove duplicates, including near-duplicates caught by fuzzy string matching for cases like inconsistent company-name formatting. Detect and address outliers using the interquartile-range method or a z-score threshold, then decide case by case whether to delete, cap, transform, or retain each one. Standardise formats — dates into a single parseable format, text lower-cased and trimmed, currency strings converted to numeric types, and known synonyms (different spellings of the same city, for example) mapped to one canonical value. Finally, validate against explicit business rules — plausible age ranges, positive prices, valid email formats — and drop or flag records that fail.

A tiered storage architecture and a compliance checklist

A practical ML data architecture has three layers. Operational databases (transactional systems such as relational or document databases) handle real-time reads and writes but are too slow for large-scale analytics. A data warehouse holds structured, historical, denormalised data optimised for analytical queries and feature engineering. A data lake stores raw, unstructured material — logs, images, JSON — cheaply and flexibly, at the cost of needing further processing before use. The typical flow runs from source systems through an ETL/ELT pipeline into the lake and warehouse, then into a feature store that centralises reusable, versioned features before they reach model training and inference.

Data protection has to be designed in from the start rather than retrofitted. Practical steps include hashing identifiers such as email addresses for irreversible anonymisation, using reversible encryption (with the key stored separately) for pseudonymisation where re-identification is sometimes legitimately needed, generating realistic fake data for non-production/testing environments so real customer records never leave production, enforcing role-based access control with audit logging, and setting a data-retention policy that deletes records automatically once they are no longer needed. Under GDPR-style regimes, this also means being able to explain any automated decision, honouring data-portability requests, and building privacy considerations into the architecture from day one rather than bolting them on after a regulator asks.