Robot Language Disambiguation 2D: Ask vs. Act

Instruction

"Pick up the red block."

Policy Settings

Ask/act threshold (confidence gap) 35%

Descriptors given upfront

Candidate objects on table 4

Session Stats

Confidence gap

—

Clarifications asked

Success rate

—

Attempts

Ready. Press "Execute Command" to run the ask-vs-act policy.

How it works

A Vision-Language-Action (VLA) system doesn't just parse an instruction into an action — it must first ground the referring phrase ("the red block") in the objects it actually perceives, and decide whether it is confident enough to act. This simulator isolates that decision:

score(object) = matches(object, given descriptors) / |given descriptors|
gap = score(top-1) − score(top-2)
if gap ≥ threshold → act on top-1
else                → ask a clarifying question, reveal one more descriptor, recompute

Each candidate object carries three attributes — color, size, side of the table. The instruction reveals only some of them upfront (the "descriptors given" setting). When two or more objects tie on every revealed descriptor, the confidence gap collapses to 0% and a threshold-based policy must choose: act anyway (risking a wrong grasp) or spend a dialogue turn asking the human for one more descriptor, at the cost of latency.

Low threshold — the robot acts even on small confidence gaps: fast, but more wrong grasps when objects share attributes.
High threshold — the robot asks more clarifying questions before acting: safer, but slower and more turns.
Fewer descriptors upfront — more ambiguity to resolve; a scene with duplicate colors under a 1-descriptor instruction is the hardest case.
If every descriptor has been revealed and objects still tie, the robot is forced to guess among the tied candidates — the residual ambiguity a real VLA system's language model cannot resolve from vision alone.

Real systems (RT-2, OpenVLA, PaLM-E-style robot policies) use exactly this kind of confidence-gap gating to decide when to execute versus when to fall back to a disambiguating dialogue turn with the user.

drag to pan · scroll / pinch to zoom