NUMA Memory Access & False Sharing

Cache Lines: The Real Unit of Sharing

Programmers think in terms of variables, individual named pieces of data like an integer counter or a struct field, but the memory hierarchy has no concept of a variable at all. Caches move, store, and track data exclusively in fixed-size blocks called cache lines, 64 bytes on essentially all mainstream x86 and ARM processors today, meaning any single byte access pulls in, and any coherence action affects, the entire surrounding 64-byte-aligned block. If a programmer declares two int counters, each only 4 bytes, next to each other in a struct or as adjacent global variables, the compiler and memory allocator will very likely place them within the same 64-byte line, since nothing about ordinary variable declaration guarantees cache-line alignment. This means that from the hardware coherence protocol's point of view, the two logically independent counters are, in fact, one single unit: writing to either one requires the exact same cache-line-wide invalidation and transfer machinery as writing to both simultaneously. The mismatch between the programmer's mental model, independent variables, and the hardware's actual granularity of tracking, whole cache lines, is the root cause of false sharing, and it is entirely invisible by reading the source code, since nothing about the variables' declarations looks wrong; the problem only exists in how they happen to be laid out in memory at a byte-address level.

How MESI Turns Proximity Into a Performance Bug

Recall that cache coherence protocols like MESI ensure that whenever one core writes to a cache line, every other core's cached copy of that same line must be invalidated before the write can proceed, guaranteeing all cores eventually observe a consistent view of memory. This mechanism is completely correct and necessary when cores are genuinely sharing data, but it cannot distinguish between genuine sharing and false sharing, since it only ever sees whole cache lines, never individual variables within them. If core A repeatedly writes variable X and core B repeatedly writes variable Y, and X and Y happen to share a cache line, every single write by either core forces that line to invalidate on the other core, so the line ping-pongs back and forth between the two caches' Modified states continuously, even though A never touches Y and B never touches X, and there is no actual data race or dependency between the two variables at all. Each ping-pong costs a full coherence round trip, invalidation broadcast, acknowledgment, and data transfer, typically tens to well over a hundred cycles depending on the system, compared to a handful of cycles for a write that hits cleanly in a core's own private cache with no contention. Multiply this by millions of writes per second in a hot loop, which is exactly the kind of code most likely to be parallelized for performance in the first place, and false sharing can turn an intended speedup from parallelism into a program that runs slower with multiple threads than with just one, a genuinely counterintuitive and frustrating outcome for anyone encountering it for the first time.

The Fix: Padding and Alignment

The standard remedy for false sharing is straightforward once diagnosed: ensure that variables written by different threads are placed on separate cache lines, so that each core's writes only ever invalidate a line that no other core actually cares about. In practice this means adding explicit padding bytes between fields that different threads will write independently, often using compiler-specific alignment directives to force a structure or field to start on a fresh 64-byte boundary, or using language features like C++17's std::hardware_destructive_interference_size, which exposes the platform's cache line size specifically so code can pad against it portably. This fix does cost real memory, padding a 4-byte counter out to a full 64-byte line wastes 60 bytes per counter, which is why padding is applied selectively, typically only to hot, frequently contended fields like per-thread counters, spinlock state, or ring buffer head and tail pointers, rather than blanket-applied to every structure in a codebase. Some designs go further and deliberately group data that is always accessed together onto the same line, a technique sometimes summarized as structuring data by access pattern rather than by logical grouping, since two fields that are always read and written together by the same thread benefit from sharing a line, fetching both in a single cache miss, while fields touched by different threads should be kept apart. This tension, pack related same-thread data tightly, but separate cross-thread contended data widely, is a recurring theme in performance-oriented systems programming and is essentially the entire lesson that false sharing teaches.

NUMA: When Even the Fix Has a Location

On single-socket, uniform memory systems, all cores reach any address in RAM with roughly equal latency, so once false sharing is eliminated, further tuning mostly concerns cache behavior alone. But multi-socket servers use a Non-Uniform Memory Access, or NUMA, design, where the total RAM is physically divided into banks, each electrically attached to and owned by one specific processor socket, called a NUMA node. A core can access its own node's local memory quickly, but accessing another node's memory requires traversing an inter-socket interconnect, such as Intel's UPI or AMD's Infinity Fabric, adding meaningfully higher latency, sometimes 30 to 60 percent more or worse, and consuming shared interconnect bandwidth that other traffic also needs. This means the cost of any given memory access, and critically the cost of coherence traffic when cores on different sockets contend for the same cache line, depends heavily on where in physical memory that data actually lives relative to which core is accessing it. Operating systems provide NUMA-aware allocation policies, such as first-touch allocation, where a page is physically placed on the node of whichever core first accesses it, and explicit APIs to pin memory or threads to specific nodes, precisely so that performance-sensitive software can arrange for each thread's frequently accessed data to live physically close to that thread's core. False sharing across NUMA nodes is a particularly severe case: not only does every write trigger coherence invalidation, but the invalidation and data transfer itself must cross the slower inter-socket link, compounding two distinct sources of overhead into one considerably worse penalty.

Diagnosing These Problems in Real Software

Because both false sharing and poor NUMA placement produce entirely correct results and leave no obvious trace in source code, they are usually found through profiling rather than code review. Hardware performance counters on modern CPUs can report cache-line contention events and remote-memory-access counts directly, and tools like Intel VTune, Linux perf c2c, which specifically visualizes cache-line contention across cores, and numastat for NUMA locality statistics are built specifically to surface these otherwise invisible costs. A telltale symptom of false sharing is a multithreaded program that scales poorly or even negatively as more threads are added, despite the workload appearing embarrassingly parallel and free of any genuine logical dependency between threads' data. A telltale symptom of NUMA misplacement is inconsistent performance depending on which specific cores a thread happens to be scheduled on, since a thread randomly placed on a socket far from its data will run slower than one placed nearby, even though the code and data are identical. Both problems share a common root lesson that this simulation is built to make visceral: the physical layout and placement of data in memory is not a cosmetic implementation detail, it is a first-class performance variable, sitting on equal footing with algorithmic complexity, that experienced systems programmers actively design around rather than discover by accident.

Frequently asked questions

Try it live

Everything above runs in your browser — open NUMA Memory Access & False Sharing and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.