Experiment Tracking with MLflow: Taming the Chaos of Model Versions

The model_v2_final_FINAL_v2.pkl problem

Anyone who has worked on a machine learning project past its first few weeks has encountered a folder that looks something like this: model_v1.pkl, model_v2_new.pkl, model_v2_final.pkl, model_v2_final_FINAL.pkl, model_v2_final_FINAL_v2.pkl. Each file represents a real training run, with a real set of hyperparameters and a real evaluation score behind it, but none of that context survives in the file name. Nobody remembers, three weeks later, whether v2_final used a learning rate of 0.01 or 0.001, whether it was trained on the dataset before or after a data cleaning fix, or whether its validation F1-score was actually better than v1's. The information existed at the moment the model was trained. It just wasn't captured anywhere durable.

This is not a discipline problem that better naming conventions fix. It's a structural problem: a machine learning project can easily generate dozens of training runs in a single week, each with its own combination of hyperparameters, feature set, random seed and code version, and manually recording all of that in a spreadsheet doesn't survive contact with a deadline. Experiment tracking tools exist specifically to make this bookkeeping automatic rather than optional, and MLflow is the most widely used open-source option for doing it.

What MLflow actually logs

MLflow organises work into experiments, and each individual training run within an experiment is logged as a run with three categories of information attached to it automatically, as soon as a few lines of logging code are added around the training loop:

• Parameters: the hyperparameters and configuration used for that run — learning rate, number of estimators, regularisation strength, the train/test split ratio, which feature set was used. These are typically logged once per run, before training starts.
• Metrics: the numbers that measure how well the run performed — accuracy, F1-score, RMSE, MAPE, AUC. These can be logged as single final values or as a series over training epochs, which lets MLflow plot a metric's trajectory across an entire training run, not just its endpoint.
• Artifacts: files produced by the run — the trained model file itself, a confusion matrix plot, a feature importance chart, a requirements file capturing exact package versions. Because the model file is logged as an artifact tied to the specific run that produced it, there is never a question of which parameters produced which model; they're stored together.

Every run also automatically records things a human would forget to note down: which git commit was checked out, the exact start and end time, and often the environment's package versions. The result is that a run from three months ago can be reconstructed with confidence — not just "the model scored 0.83 on F1," but the specific parameters, code version and data that produced that number.

Comparing runs instead of guessing from memory

Logging a single run in isolation is useful for record-keeping, but the real value shows up when comparing dozens of runs against each other. MLflow's tracking UI presents every logged run as a row in a table, with parameters and metrics as sortable, filterable columns, so a question like "did increasing max_depth past 6 actually help, or did it just increase training time" can be answered by sorting the run table by that parameter and comparing the resulting metric column, instead of scrolling back through old notebook cells trying to remember what was tried. Runs can also be plotted against each other directly — metric-versus-parameter scatter plots, or parallel coordinate plots that show how several hyperparameters moved together across the best-performing runs, which is often the fastest way to spot that, say, a particular combination of learning rate and batch size consistently outperforms every other combination tried so far.

This turns what used to be an unstructured trial-and-error process — try something, eyeball the result, half-remember whether it was better than last time — into something closer to a searchable experiment log. It also has a quieter benefit: because every run is logged the same way regardless of who ran it, MLflow makes hyperparameter search techniques like grid search, random search and Bayesian optimisation more useful in practice, since the dozens or hundreds of runs those methods generate would otherwise be nearly impossible to review by hand.

The model registry: from experiment to production candidate

Tracking individual runs solves the "which parameters produced this model" problem. A second, related problem is "which model is actually running in production right now, and what's the process for replacing it." MLflow's model registry addresses this by letting a specific run's model artifact be registered under a named, versioned model — for example, registering a run's output as version 4 of a fraud-detection model. Each registered version can carry a stage label such as Staging or Production, and promoting a model from Staging to Production becomes a deliberate, auditable action rather than a file being copied over an existing one on a server somewhere.

This distinction matters more than it might first appear. Without a registry, "the production model" is whatever file currently sits in a particular directory, with no record of what it replaced or when. With a registry, there's a clear, queryable history: version 3 was in production for six weeks, version 4 replaced it after showing a 2-point F1 improvement in Staging, and if version 4 turns out to have a problem after deployment, rolling back to version 3 is a matter of changing a stage label rather than trying to reconstruct an old training run from memory.

Where experiment tracking fits in a wider ML pipeline

MLflow is deliberately narrow in scope: it tracks experiments and manages a model registry, and leaves serving, containerisation and infrastructure orchestration to other tools. In a typical production pipeline it sits alongside, not instead of, a serving layer built with something like FastAPI, a container built with Docker, and a monitoring layer watching for data drift once the model is live. The connection between these pieces is direct: the model artifact that FastAPI loads and serves is the exact artifact MLflow logged for a specific, traceable run, and the Docker image that packages that service can pin the exact package versions MLflow recorded alongside it. None of these tools substitutes for good judgement about which model to ship — but without experiment tracking, the judgement has to be made from memory and scattered file names, and with it, the judgement can be made from an actual comparable record of what was tried and how well each attempt performed.