Corners, Blobs and Keypoints: How Harris, SIFT and ORB Find Features

What Makes a Point Worth Tracking

Before a computer can stitch photos into a panorama, recognise a landmark from two different angles, or track an object moving through a video, it needs to answer a deceptively hard question: which specific points on an image are worth paying attention to? A useful point, or feature, has to satisfy several properties at once. It must be repeatable — findable again in a second image of the same scene taken from a different angle or under different lighting. It must be distinctive, standing out clearly from its surroundings so it is not confused with dozens of similar-looking points nearby. And ideally it should be invariant to common transformations: scale changes as the camera moves closer or further, rotation as the camera tilts, and moderate lighting changes.

Not every kind of image region qualifies. A flat patch of blue sky provides no distinguishing information — every 5×5 window of sky pixels looks like every other. A straight edge is better, since it tells you which direction structure runs, but it still slides ambiguously along its own length — a point halfway along a fence rail is indistinguishable from a point ten pixels further along the same rail. A corner, where intensity changes sharply in two different directions at once, is the sweet spot: it pins down a location precisely, because shifting a small window in any direction from a true corner changes what's inside that window substantially.

The Harris Corner Detector: Measuring Intensity Change in Every Direction

Published in 1988, the Harris corner detector formalises the intuition above into a concrete calculation. For each pixel, it examines a small local window and asks: if I shift this window slightly in any direction, how much does the content inside it change? In a flat region, shifting the window barely changes anything. Along an edge, shifting perpendicular to the edge changes things a lot, but shifting along the edge changes almost nothing. At a true corner, shifting the window in any direction produces a large change.

Harris captures this mathematically with a small 2×2 matrix built from the image gradients I_x and I_y (computed with a Sobel operator) summed over the local window:

M = [ Σ(Ix²)    Σ(IxIy) ]
    [ Σ(IxIy)   Σ(Iy²)  ]

The eigenvalues of this matrix, λ₁ and λ₂, describe how much intensity varies along the two principal directions at that point. If both eigenvalues are small, the region is flat. If one is large and the other small, it's an edge. If both are large, it's a corner. Rather than computing eigenvalues directly (which is more expensive), Harris uses a cheaper proxy called the corner response:

R = det(M) − k·trace(M)²  =  λ1·λ2 − k·(λ1 + λ2)²

with a sensitivity constant k typically between 0.04 and 0.06. A large positive R signals a corner, a large negative R signals an edge, and R near zero signals a flat region. Sweeping this calculation across the whole image and keeping local maxima above a threshold gives the final set of corner points. Harris corners are fast to compute and rotation-invariant — a corner rotated 30 degrees is still a corner — but they are not scale-invariant: a corner that is sharp when viewed up close may look almost flat when the same object is photographed from far away, and vice versa, which limits Harris to situations where the camera distance stays roughly constant.

SIFT: Solving Scale Invariance With an Image Pyramid

David Lowe's Scale-Invariant Feature Transform, published in 1999, was designed specifically to survive the zoom problem that Harris corners cannot handle, along with rotation and moderate lighting and viewpoint changes. It does this through a four-stage pipeline.

Scale-space extrema detection. Rather than looking for corners at a single resolution, SIFT builds a pyramid of progressively more blurred and downsampled copies of the image, and computes the Difference of Gaussians (DoG) between adjacent blur levels — an efficient approximation of the Laplacian of Gaussian discussed in edge-detection theory. A candidate keypoint is any pixel that is a local extremum (brighter or darker than all its neighbours) not just spatially, but also across neighbouring scales in this 3D stack. Finding an extremum across scale as well as space is what lets SIFT report a natural "size" for each keypoint that adapts automatically to zoom level.

Keypoint localisation. Candidate points are refined with sub-pixel accuracy via quadratic interpolation, and unstable candidates — low-contrast points or points lying along an edge rather than at a true corner-like blob — are discarded, mirroring the eigenvalue reasoning from Harris.

Orientation assignment. A histogram of local gradient directions around each keypoint is computed, and the dominant direction becomes the keypoint's canonical orientation. Every subsequent description of that keypoint is measured relative to this orientation, which is exactly what makes the resulting descriptor invariant to image rotation — rotate the image, and the histogram peak rotates with it, keeping the relative description unchanged.

Descriptor construction. Finally, the 16×16 pixel neighbourhood around the keypoint is divided into a 4×4 grid of sub-blocks, and each sub-block contributes an 8-bin histogram of gradient orientations, producing 4×4×8 = 128 numbers — the SIFT descriptor. This 128-dimensional vector is distinctive enough that two descriptors from the same physical point on an object, photographed under different conditions, end up much closer to each other (by Euclidean distance) than to descriptors from unrelated points, which is precisely the property needed for reliable matching.

ORB: A Fast, Free Alternative for Real-Time Use

SIFT's accuracy comes at a computational cost, and for years its most direct competitor, SURF, was patent-encumbered like SIFT itself (SIFT's patent expired in 2020). ORB — Oriented FAST and Rotated BRIEF, released by OpenCV Labs in 2011 — was built explicitly to be fast, free, and good enough for real-time applications running on ordinary CPUs, including mobile devices.

ORB combines two separate, lightweight algorithms. FAST (Features from Accelerated Segment Test) locates candidate keypoints by examining a ring of sixteen pixels around a candidate centre and checking whether a contiguous arc of them is all consistently brighter or darker than the centre by some threshold — a test that requires only a handful of pixel comparisons and can reject non-corners almost immediately, making it extremely fast. BRIEF (Binary Robust Independent Elementary Features) then describes each keypoint not with a 128-dimensional float vector like SIFT, but with a compact 256-bit binary string, generated by comparing pairs of pixel intensities around the keypoint and recording a 1 or 0 for each comparison.

The binary descriptor is the key to ORB's speed advantage in matching: comparing two SIFT descriptors requires computing a Euclidean distance over 128 floating-point numbers, while comparing two ORB descriptors requires only a Hamming distance — counting differing bits — over 256 bits, an operation modern CPUs execute via a single instruction (XOR plus population count) almost instantly. ORB adds orientation information to FAST and a rotation-aware sampling pattern to BRIEF specifically so the resulting descriptor tolerates image rotation, closing much of the gap with SIFT's invariance properties while running roughly two orders of magnitude faster.

From Keypoints to Matches: Descriptors, RANSAC and Homography

Detecting keypoints in two images is only half the job — the real value comes from matching them, finding pairs of keypoints in image A and image B that correspond to the same physical point. This is done by comparing descriptors: for float-based descriptors like SIFT, a brute-force matcher finds, for each keypoint in image A, its nearest neighbour by Euclidean distance in image B; for binary descriptors like ORB, the same idea applies using Hamming distance.

Raw nearest-neighbour matching produces plenty of incorrect matches, since some points genuinely resemble each other despite being unrelated, or the true match may be missing entirely if it fell outside the second image's field of view. Two techniques clean this up. Lowe's ratio test discards a match unless the best candidate is substantially closer than the second-best candidate — if two candidates are nearly tied, the match is ambiguous and unreliable, so it's thrown out. Then, when the surviving matches are used to compute a geometric relationship between the two images (a homography, describing how a flat plane maps from one camera view to another), the RANSAC algorithm repeatedly samples small random subsets of matches, fits a candidate homography to each subset, and keeps whichever candidate is consistent with the largest number of the remaining matches (its "inliers"), discarding the rest as outliers. This combination — descriptor matching, ratio test, RANSAC — is the backbone of panorama stitching, augmented reality marker tracking, and structure-from-motion 3D reconstruction.