Learning #24 – Thermodynamics: Entropy, the Second Law, Phase Transitions and Statistical Mechanics

Thermodynamics is unusual among physical theories: it makes no assumptions about the microscopic constitution of matter. The laws are macroscopic statements about measurable quantities — temperature, pressure, volume, heat, work — that hold universally, whether the working substance is an ideal gas, a rubber band, a black hole or a biological cell. This generality is both its power and its mystery.

Statistical mechanics, developed by Boltzmann, Gibbs and Maxwell in the second half of the nineteenth century, fills in the microscopic picture: macroscopic thermodynamic quantities emerge from averaging over the enormous number of microscopic degrees of freedom. The link between the two — most compactly expressed in Boltzmann’s inscription S = k log W — is one of the deepest insights in all of physics.

1. The Laws of Thermodynamics — A Concise Statement

The four laws of thermodynamics are among the most condensed statements of physical principle ever formulated. Each encodes a profound constraint on what processes are possible in nature.

Zeroth Law (thermal equilibrium):
  If A is in equilibrium with C, and B is in equilibrium with C, then A and B are in
  equilibrium with each other. This defines temperature as a property — the quantity that
  is equal between systems in thermal contact.

First Law (energy conservation):
  dU = δQ − δW
  U = internal energy (state function)
  δQ = heat added to system (path-dependent)
  δW = work done by system (path-dependent)
  For a reversible process: dU = T dS − P dV + µ dN + …

Second Law (entropy):
  In any spontaneous process, the total entropy of an isolated system never decreases.
  dS_total ≥ 0, with equality for reversible processes.
  Clausius statement: heat cannot spontaneously flow from cold to hot.
  Kelvin statement: no cycle can convert heat entirely into work with a single reservoir.

Third Law (absolute zero):
  As T → 0, the entropy of a perfect crystal approaches a constant (often taken as zero).
  S(T→0) = 0 for a non-degenerate ground state.
  Consequence: T = 0 K is unattainable in a finite number of steps.

State functions vs path functions:
  State functions: T, P, V, U, H = U+PV, A = U−TS (Helmholtz), G = H−TS (Gibbs), S
  Path functions: Q (heat), W (work) — depend on how a process is carried out.
  Only state function differences are well-defined for a given change of state.

2. Entropy — From Clausius to Boltzmann

Entropy was introduced by Rudolf Clausius in 1865 as a state function defined by the ratio of reversible heat to temperature: dS = δQ_rev / T. Clausius could calculate entropy changes for steam engines without knowing anything about atoms. Twenty years later, Ludwig Boltzmann gave entropy a microscopic meaning that transformed it from an accounting device into a window on the nature of disorder.

Clausius entropy (1865):
  dS = δQ_rev / T
  S is a state function; only ΔS = ∫ δQ_rev/T is measurable.
  For the universe: ΔS_universe ≥ 0 (second law)

Boltzmann entropy (1877):
  S = k_B ln W
  W = number of microstates consistent with the macrostate (multiplicity)
  k_B = 1.380649 × 10^−23 J/K (Boltzmann constant)
  Inscribed on Boltzmann’s tombstone in Vienna.

Gibbs entropy (for a probability distribution over microstates):
  S = −k_B Σ_i p_i ln p_i
  Reduces to Boltzmann’s expression when all W microstates are equally probable.
  Used in statistical mechanics and information theory (Shannon entropy: replace k_B with 1/ln 2).

Entropy and information:
  Shannon entropy H = −Σ_i p_i log&sub2; p_i  (bits)
  For a uniform distribution over W outcomes: H = log&sub2; W
  Physical entropy S = k_B ln 2 · H  (conversion factor between bits and joules/kelvin)
  Erasing one bit of information dissipates at least k_B T ln 2 of energy (Landauer limit).

Entropy and irreversibility (example: free expansion):
  Ideal gas, N molecules, doubles its volume freely:
  W_final / W_initial = 2^N  (each molecule independently has twice the volume available)
  ΔS = k_B ln(2^N) = N k_B ln 2 = nR ln 2
  For 1 mol: ΔS ≈ 5.76 J/K — a macroscopic increase from microscopic combinatorics.

The connection between entropy and probability resolves the apparent paradox of irreversibility. The laws of mechanics are time-reversible: any microstate trajectory can be played backward without violating Newton’s laws. Yet macroscopic processes are irreversible. Boltzmann’s answer: the initial states of macroscopic systems are extraordinarily improbable compared to the final states — not because the reversed trajectory is forbidden, but because it is astronomically unlikely. An ice cube melting in warm water could in principle re-solidify; the probability is just so vanishingly small that it would take many times the age of the universe to observe it.

3. The Carnot Cycle — Maximum Efficiency

Sadi Carnot showed in 1824 that no heat engine operating between two reservoirs at temperatures T_H and T_C can be more efficient than a reversible engine operating between the same reservoirs. This is not a statement about friction or imperfect materials — it is a fundamental limit set by the second law.

Carnot cycle (ideal gas):
  Step 1: Isothermal expansion at T_H, absorbs Q_H
    W_12 = nRT_H ln(V_2/V_1) = Q_H
  Step 2: Adiabatic expansion T_H → T_C
    TV^(γ−1) = const,  W_23 = c_v(T_H − T_C)
  Step 3: Isothermal compression at T_C, rejects Q_C
    W_34 = −nRT_C ln(V_3/V_4) = −Q_C
  Step 4: Adiabatic compression T_C → T_H
    W_41 = c_v(T_C − T_H)

Carnot efficiency:
  η_C = W_net / Q_H = 1 − T_C/T_H   (temperatures in Kelvin)
  This is the maximum efficiency of any heat engine between T_H and T_C.
  All reversible engines between the same reservoirs have the same efficiency.

Why η < 1:
  Some heat Q_C = Q_H(1 − η) must always be rejected to the cold reservoir.
  This is required by the second law: the entropy deposited in the cold reservoir
  must be at least as large as the entropy extracted from the hot reservoir.
  ΔS_univ = −Q_H/T_H + Q_C/T_C ≥ 0  ⇒  Q_C/T_C ≥ Q_H/T_H  ⇒  η ≤ 1 − T_C/T_H

Real power plant efficiencies (typical):
  Coal / gas turbine:   η ~ 40–55%  (T_H ~ 600°C, T_C ~ 30°C ⇒ η_C ~ 66%)
  Combined cycle (CCGT): η ~ 60–63%
  Gasoline engine:      η ~ 25–35%  (much waste heat in exhaust)
  Diesel engine:        η ~ 40–50%
  Solar thermal:        η ~ 20–35%

4. Maxwell–Boltzmann Distribution — The Statistics of Gas Molecules

In a gas at thermal equilibrium, molecules move at a wide range of speeds. James Clerk Maxwell (1860) and Ludwig Boltzmann (1872) derived the probability distribution of molecular speeds from first principles — a landmark result that opened the door to statistical mechanics.

Probability density of speeds v (3D):
  f(v) = 4π (m/2πk_BT)^(3/2) · v² · exp(−mv² / 2k_BT)

Characteristic speeds:
  Most probable speed:  v_p = √(2k_BT/m) = √(2RT/M)
  Mean speed:           <v> = √(8k_BT/πm) = √(8RT/πM)     (  ≈ 1.596 v_p / √(2) )
  RMS speed:            v_rms = √(3k_BT/m) = √(3RT/M)            (used in ½mv² = ½k_BT ×3 )
  Ordering: v_p < <v> < v_rms

At 300 K, air (∼ N&sub2;, M = 0.028 kg/mol):
  v_p ≈ 422 m/s,  <v> ≈ 476 m/s,  v_rms ≈ 517 m/s

Energy distribution (Maxwell–Boltzmann energy):
  g(ε) = 2π (1/πk_BT)^(3/2) · √ε · exp(−ε/k_BT)
  Mean kinetic energy per molecule: <ε> = ¾k_BT

Equipartition theorem:
  Each quadratic degree of freedom contributes ½k_BT of average energy.
  Monatomic ideal gas: 3 translational DOF → U = ¾Nk_BT = ¾nRT
  Diatomic (rigid):    5 DOF (3 trans + 2 rot) → U = 5/2nRT   (c_v = 5/2R)
  Diatomic (vibrating): 7 DOF at high T → U = 7/2nRT  (vibration activates at high T)

The existence of the Maxwell–Boltzmann distribution means that at any given temperature a small fraction of molecules always have enough energy to escape a liquid’s surface (evaporation), overcome an activation barrier (chemical reactions), or penetrate a nuclear barrier (stellar fusion). These high-energy tail events drive much of chemistry and astrophysics, which is why reaction rates are so temperature-sensitive and why nuclear fusion requires temperatures of tens of millions of degrees.

5. Maxwell’s Demon — Information and the Second Law

In 1867 James Clerk Maxwell proposed a thought experiment that seemed to refute the second law. A tiny intelligent being — later called Maxwell’s demon — sits at a trap door between two chambers of gas. By watching individual molecules and opening the door selectively, it could sort fast molecules into one chamber and slow molecules into the other, creating a temperature difference from an initially uniform gas without doing any work. The paradox took nearly a century to resolve.

The original paradox (1867):
  Demon observes each molecule and opens/closes trap-door accordingly.
  Result: fast molecules accumulate on right, slow on left.
  Temperature difference ΔT created → heat engine can extract work.
  No apparent work input → second law violated?

Resolution via information theory (Szilárd 1929, Bennett 1982, Landauer 1961):
  The demon must store information about each molecule’s velocity in its memory.
  To operate cyclically, the demon must eventually erase its memory.
  Landauer’s principle: erasing one bit of information dissipates at least k_B T ln 2:
    ΔQ_erase ≥ k_BT ln 2 per bit
  The entropy cost of erasure exactly compensates the entropy reduction in the gas.

Landauer limit (minimum energy per logic operation at 300 K):
  k_BT ln 2 ≈ 2.87 × 10^−21 J ≈ 0.018 eV ≈ 0.0179 meV
  Modern CPUs dissipate ~1000× more per operation → still far from Landauer limit.
  Ultimate computing limit (physical): ~10^21 operations per joule at 300 K.

Key insight:
  Information has physical reality. Acquiring information about a system does not
  require work; erasing (or overwriting) information does. This is the origin of the
  thermodynamic arrow of time in computation.

Modern realisations:
  Experimental Szilárd engines (2010s): single-electron boxes or colloidal particles
  in feedback traps have extracted near-kT of work per bit of information, confirming
  the Szilárd–Landauer framework experimentally.

6. Phase Transitions — Order, Symmetry and Critical Phenomena

A phase transition is a sharp, macroscopic change in the properties of a material triggered by a smooth change in a control parameter such as temperature or pressure. Ice melts, water boils, a magnet loses its magnetisation at the Curie temperature, a superconductor transitions at T_c. The Ising model — the simplest model of a ferromagnet — captures the essential physics of phase transitions and critical phenomena in a form that is exactly solvable in two dimensions.

Classification of phase transitions (Ehrenfest):
  First-order: discontinuity in the first derivative of G (volume, entropy)
    → latent heat, phase coexistence (e.g. melting, boiling)
  Second-order (continuous): discontinuity in second derivative of G (heat capacity)
    → no latent heat; order parameter grows continuously from zero
    → diverging correlation length and susceptibility at T_c

Ising model (lattice of spins σ_i = ±1):
  Hamiltonian: H = −J Σ_{<ij>} σ_i σ_j − h Σ_i σ_i
  J > 0: ferromagnetic coupling (aligned spins preferred)
  h: external field
  The competition between J (ordering) and k_BT (thermal disorder) drives the transition.

Critical temperature:
  Mean-field: k_B T_c = zJ (z = coordination number, e.g. 4 for 2D square lattice)
  2D exact (Onsager 1944): k_B T_c = 2J / ln(1 + √2) ≈ 2.269 J
  Mean-field overestimates T_c; fluctuations are crucial near T_c.

Order parameter and critical exponents (2D Ising universality class):
  Magnetisation: m ~ (T_c − T)^β      β = 1/8
  Susceptibility: χ ~ |T − T_c|^−γ    γ = 7/4
  Correlation length: ξ ~ |T − T_c|^−ν   ν = 1
  Specific heat: C ~ |T − T_c|^−α    α = 0 (logarithmic divergence)

Universality:
  The critical exponents depend only on dimensionality and symmetry, not on microscopic
  details. The 2D Ising universality class describes many physical systems near their
  critical points: lattice gas (liquid–vapour transition), binary alloys, polymer
  networks, even social dynamics models.

Monte Carlo simulation (Metropolis algorithm):
  1. Pick a random spin σ_i
  2. Compute ΔE = energy change if flipped.
  3. Accept flip: always if ΔE < 0; with probability exp(−ΔE/k_BT) if ΔE > 0.
  4. Repeat. Generates a Boltzmann-distributed ensemble of microstates.

The exact solution of the 2D Ising model by Lars Onsager in 1944 was one of the great mathematical achievements of twentieth-century physics. It confirmed that critical exponents are non-trivial rational numbers (not the mean-field predictions) and established that near a critical point, systems behave in a universal way characterised only by their symmetry and dimensionality. This universality underlies the renormalisation group, which earned Kenneth Wilson the Nobel Prize in Physics in 1982 and which forms the conceptual framework for modern particle physics, condensed matter and statistical field theory.

Thermodynamics Beyond Physics

The conceptual reach of thermodynamics extends far beyond steam engines and gases. The free energy G = H − TS determines the direction of chemical reactions and biological processes: a reaction proceeds spontaneously when ΔG < 0, meaning the enthalpy decrease and/or entropy increase is sufficient to drive the system downhill in free energy. ATP hydrolysis in cells, protein folding, membrane transport and DNA replication are all governed by this single criterion.

Information theory (Shannon 1948) is formally identical to statistical mechanics (Jaynes 1957): maximum entropy inference is the basis of Bayesian probability theory and modern machine learning. Black hole thermodynamics (Bekenstein–Hawking entropy proportional to horizon area) suggests that spacetime itself may be an emergent thermodynamic phenomenon. In every domain, from cold atoms to cosmology, the second law — entropy never decreases — stands as one of the most universal statements in all of science.