Why Git Can't Track a Changing Dataset (and What DVC-Style Tools Do Instead)

Git was built to diff text, not to store gigabytes

Git's core mechanism is content-addressed storage of line-based diffs: when you commit a change to a source file, Git stores a compact representation of what changed, and its speed and small repository size depend on files being text and changes being small relative to the whole file. A training dataset breaks both assumptions at once. It's frequently binary (images, audio, serialized tensors, Parquet files) so Git can't compute a meaningful line diff at all and falls back to storing the entire new copy of the file on every commit, and it's frequently enormous — training sets measured in tens or hundreds of gigabytes are common — which makes every clone, every checkout, every branch switch prohibitively slow and the .git directory itself balloon to an unusable size. Git LFS (Large File Storage) patches over some of this by replacing large files with lightweight pointer files in the actual Git history and storing the real content in separate LFS storage, but it's still fundamentally file-level versioning bolted onto a tool designed for text, and it doesn't solve the deeper problem underneath.

The deeper problem: which exact data trained which exact model

The real requirement isn't just "can I store a big file in version control," it's "given a model sitting in production right now, can I reconstruct the exact dataset, the exact preprocessing code, and the exact hyperparameters that produced it, months later, when someone asks why the model behaves a certain way or a regulator asks for an audit trail." This is a much harder problem than file storage because a real ML pipeline's dataset is rarely a single static file — it's usually the output of a chain of transformations (raw data pulled from a warehouse, filtered, joined against other tables, features engineered, a train/validation/test split applied), and each of those steps can independently change over time as upstream sources get corrected, backfilled, or extended. Two models trained a month apart, using code that looks identical in Git, can have been trained on genuinely different data if the underlying data warehouse tables were updated in between and nothing captured that fact.

This matters for reasons well beyond academic tidiness. A model that suddenly performs worse in production needs its training data compared against current production data to diagnose distribution drift, which is impossible if you can't reconstruct exactly what the training data looked like. A regulator investigating a lending or hiring model needs to verify the training data didn't contain a legally protected feature, which requires being able to actually retrieve that historical dataset, not just the code that generated it. And basic scientific reproducibility — another engineer rerunning your pipeline and getting the same result — silently fails if the input data has drifted underneath an unversioned pipeline, even when every line of code is untouched.

How DVC actually decouples data from Git

DVC (Data Version Control) solves this by never asking Git to store the data at all. When you run dvc add on a dataset, DVC computes a content hash of the file (or directory) and moves the actual data into a local cache directory named by that hash, then writes a small, human-readable .dvc metadata file (just the hash, the size, and the path) that gets committed to Git in place of the real data. Git ends up versioning a lightweight pointer, exactly the way Git LFS does, but DVC goes further by wiring this into a pipeline definition system: a dvc.yaml file describes each stage of your pipeline (a preprocessing step, a training step) along with its dependencies (specific data files and code files) and outputs, and DVC computes hashes of all of them to determine whether a stage needs to be rerun — if none of a stage's tracked dependencies have changed since the last run, DVC skips it and reuses the cached output, which turns your whole pipeline into something closer to a build system like Make, but one that's aware of large binary data rather than just source code.

The actual bulk data lives in remote storage — S3, GCS, Azure Blob, or a plain network drive — configured once as a DVC remote, and dvc push/dvc pull move data between that remote and the local cache the same way git push/git pull move commits, meaning a colleague (or a CI pipeline) can check out a specific Git commit, run dvc pull, and get back the exact bytes of the dataset that existed at that commit, even though Git itself never touched those bytes directly.

lakeFS and the git-for-data-lake approach

lakeFS takes a related but architecturally distinct approach aimed specifically at teams already working with large object-storage data lakes (S3-backed tables, often queried by Spark or similar engines) rather than discrete files pulled down to a local machine. Instead of hashing individual files and pointing Git at them, lakeFS sits as a transparent layer in front of your existing object storage and gives the entire bucket Git-like semantics directly: you can branch an entire multi-terabyte data lake in roughly constant time because branching doesn't copy any data, it only creates new metadata pointers to the existing immutable objects (much like how Git branches are just pointers to commits, not copies of a whole repository); you can commit a snapshot of the lake's current state and reference that exact commit hash from your training job's metadata, so "which data trained this model" becomes a single stored string rather than an open investigation; and you can merge or roll back changes to production data with the same conflict-detection semantics Git applies to code, which matters a great deal when multiple pipelines are writing to overlapping parts of the same lake.

The two tools sit at different points in a typical stack rather than directly competing: DVC is commonly used to version the specific, curated dataset snapshots and pipeline stages that feed a given training run, often pulled from a lake that lakeFS or an equivalent system is separately versioning at the infrastructure level, giving reproducibility guarantees at both the fine-grained (this exact training set) and coarse-grained (this exact state of the whole data lake at 09:00 on a given date) levels simultaneously.