VLM Object Hallucination Lab — Evidence/Prior Phase Plane (2D)

Scene

Decoding Weights

Visual evidence weight w_v 1.00

Language prior weight w_l 0.60

Occlusion / sensor noise 20%

Metrics

CHAIR_i hallucination

Grounding accuracy

100%

Objects mentioned

0 / 0 present

How it works

Real vision-language models (GPT-4V, LLaVA, Florence-2) caption an image by generating one word at a time, scoring each candidate object token as a mix of two signals:

score(o) = w_v · evidence(o) + w_l · prior(o | mentioned) + ε
P(next = o) = softmax( score(·) / T )

This 2D build plots that exact score function as a phase plane: the x-axis is an object's real visual evidence (how strongly the encoder detects it — near 0 if occluded or absent), the y-axis is the language prior (the strongest co-occurrence score with anything already mentioned, e.g. "fork" strongly implies "knife"). The background field is the live decision score w_v·e + w_l·p for every point in that plane, so as you drag the sliders you can watch the field itself lean toward the evidence axis or the prior axis. Every candidate object is a dot at its own (evidence, prior) coordinate, sized by its actual softmax probability of being the next generated word — a large dot sitting on an absent object is a hallucination about to happen, visible before it's spoken.

The standard metric for this is CHAIR_i (Caption Hallucination Assessment with Image Relevance, instance-level):

CHAIR_i = |{hallucinated objects mentioned}| / |{all objects mentioned}|

w_v / w_l sliders — reweight the decoding score toward real evidence or learned bias; watch the background field tilt toward the x- or y-axis accordingly.
Occlusion slider — randomly suppresses true evidence and injects noise, pulling present-object dots left toward the low-evidence wall.
Scene buttons — reseed which of each vocabulary's objects are actually present; the co-occurrence prior matrix stays fixed per scene, just like a model's training-derived biases don't change per image.

Present object (dot size = P(next)) Absent object — a phantom risk Decoding score field w_v·e + w_l·p