Home▸AI & Machine Learning▸VLM Object Hallucination Lab

VLM Object Hallucination Lab

Watch a vision-language model caption a 3D scene word by word — tune how much it trusts real visual evidence vs. learned language priors, and watch the CHAIR hallucination rate rise as co-occurrence bias overrides what the camera actually sees.

AI & Machine Learning3DAdvanced60 FPS📱 Mobile-adapted⇄ 2D version
VLM Object Hallucination Lab ↗ Open standalone

Vision-language models like GPT-4V, Gemini Vision, LLaVA and Florence-2 caption images by blending two signals at every generated word: real visual evidence from the image encoder, and a language prior learned from co-occurrence statistics in training captions. This simulator renders an actual 3D scene — a kitchen table, an office desk, or a beach — with a real subset of objects placed in it, then generates a caption for that scene word by word using a live softmax over both signals. Push the language-prior weight or the occlusion/noise slider up and watch strongly-associated objects (a knife next to a fork, a mouse next to a laptop) get named even when they were never placed in the scene — a hallucinated mention, tracked live by the same CHAIRi metric researchers use to benchmark real VLMs.

⚙ Under the hood

Watch a vision-language model caption a 3D scene word by word, blending real visual evidence with learned co-occurrence priors — push the language-prior weight or sensor noise up and watch it name objects that were never actually there, tracked live with the CHAIR hallucination metric used to benchmark GPT-4V, LLaVA and Florence-2.

vision-language modelVLMhallucinationCLIPcaptioningCHAIR metric

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install

What did you find?

Add reproduction steps (optional)