Weak Supervision: Combining Noisy Labeling Functions

Labeling Functions

Number of labeling functions (K) 5

Average LF accuracy 70%

Average LF coverage 55%

Aggregation

Live Readouts

Aggregate accuracy

–

Coverage

–

Best single LF vs aggregate

–

Aggregated label: class A Aggregated label: class B Abstain (no firing LF) Mislabeled vs ground truth (ring)

How it works

Each of the K labeling functions (LF) is a cheap heuristic rule. It doesn't see every point — it fires with probability equal to its coverage c_k, and only when it fires does it cast a vote. When it does vote, that vote matches the true label with probability equal to its accuracy a_k; otherwise it is flipped. This is the standard (coverage, accuracy) model behind weak-supervision systems such as Snorkel.

fires_k ~ Bernoulli(c_k)
vote_k  =  true_label      w.p. a_k
        = -true_label      w.p. 1 - a_k   (given fires_k = 1)

A majority vote just sums the raw votes: label = sign(Σ vote_k). It weights every LF equally, so one noisy rule can outvote several good ones.

The generative label model instead estimates each LF's own accuracy from how often it agrees with the current majority vote (â_k), then re-weights every vote by its log-odds before summing — exactly the idea behind Snorkel's generative model and the classic Dawid–Skene estimator:

â_k = agreement(vote_k, majority) over points where LF k fires
w_k = ln( â_k / (1 − â_k) )
label = sign( Σ_k firing  w_k · vote_k )

A confident, high-accuracy LF gets a large positive weight; a near-coin-flip LF gets a weight near zero and is effectively ignored — which is why the weighted aggregate usually beats both majority vote and every single LF once accuracies start to differ.

Class A Class B Abstain

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.