IVG-Bench Tests Vision-Language Models in Closed-Loop Visual Interaction

Summary:Interactive Visual Gym evaluates vision-language models as real-time assistants for physical tasks by jointly simulating executable environments and human users. The resulting IVG-Bench contains 315 scenarios spanning cooking, household manipulation, and mobile interfaces, with explicit treatment of response latency, factual grounding, actionable guidance, execution errors, and out-of-plan user behavior. The best evaluated model, Gemini 4 Argon, completed only 53.4% of tasks within the time limi

---

Closed-loop interaction becomes the unit of evaluation

Most vision-language model benchmarks use offline replay: a fixed input, a predefined answer, and a single inference pass. That setup misses the mechanism that determines whether an assistant is useful in a live physical task. Its answer changes what the user does next; incorrect guidance causes execution errors; and a response that arrives too late may no longer prevent the failure it was meant to avoid.

Interactive Visual Gym evaluates vision-language models at this operational level. It jointly runs an executable task environment and a persona-conditioned user simulator. Model outputs affect the environment state, which in turn changes user behavior and the eventual task trajectory. The framework is instantiated as IVG-Bench, with 315 interaction scenarios spanning three domains:

• Cooking tasks with multi-step procedures, timing constraints, and state tracking;
• Household manipulation involving object location, action ordering, and exceptions;
• Mobile-interface tasks that test cross-app operations, interface-state understanding, and instruction execution.

The user simulator can also inject controlled execution errors, deviations from the plan, and other out-of-plan behaviors. This is more realistic than replaying a perfectly cooperative script. The system must recover when the user does not follow its advice instead of attributing every failure to the model.

Why latency belongs inside the task contract

A visual interaction assistant must observe the current environment, infer user intent, and generate the next actionable instruction. Correctness alone is insufficient in a live task. Even a semantically correct answer loses operational value if generation takes longer than the action window and arrives after the user has already moved past the relevant decision point.

IVG-Bench incorporates response time into task-completion criteria rather than treating it as a secondary latency chart. A useful interaction score should consider whether the assistant grounds its advice in visible state, whether that advice is concrete and executable, whether it arrives before the deadline, whether the user moves the environment toward the correct state, and whether the system can replan after an execution error.

This explains why offline visual-reasoning performance can diverge sharply from closed-loop results. A model may recognize objects and generate plausible instructions well while still lacking continuous state tracking, deadline sensitivity, and exception recovery. In physical tasks, one hallucinated step can make the user execute the wrong action, causing all subsequent reasoning to operate on a corrupted state.

Results: strong visual QA does not imply dependable assistance

The paper evaluates advanced vision-language models and identifies Gemini 4 Argon as the best-performing system, with a time-limited task completion rate of 53.4%. Even by the combined evaluation criteria, current models fail to complete nearly half of the tasks within the allowed time.

Three failure modes are especially important:

• Insufficient factual grounding: descriptions of objects, steps, or current state can disagree with the executable environment;
• Non-actionable guidance: an answer may point in roughly the correct direction but omit the next action, ordering, or required parameters;
• Late interventions: the model completes reasoning only after the user has advanced beyond the point where the advice could help.

A human study provides an external validity check for the protocol. The interaction-quality metric agrees with participant preferences 78.5% of the time. This supports the usefulness of the closed-loop rubric, but it also leaves meaningful disagreement. Scenario design, weighting choices, and participant sampling may still materially affect the score.

Reinforcement learning: the environments can also train models

The team also reports a proof-of-concept GRPO experiment on Qwen3.5-4B in the cooking environment. Closed-loop training improves both task success and interaction quality. Its significance lies in the source of the reward: it can come from environment execution and task trajectories rather than only offline labeled answers.

In GRPO, multiple responses or policy trajectories are sampled for the same prompt, and within-group rewards estimate relative advantages. For interaction tasks, the reward can combine final task state, step quality, error recovery, and latency penalties. Unlike conventional supervised fine-tuning, these signals directly optimize whether an answer produces the correct action.

The proof of concept should not be read as proof of scalable training. Its engineering costs are substantial:

• Every rollout runs both a user simulator and an environment state machine, reducing throughput relative to offline generation;
• Rewards based only on final success are sparse, motivating careful step-level reward design;
• The simulator differs from real users, allowing policies to exploit fixed simulator patterns;
• Latency penalties encourage shorter outputs and may trade away factual reliability or explanation completeness;
• GRPO group sampling and rollout counts increase training compute substantially.

IVG is therefore better understood as research and training infrastructure than as a complete substitute for real user behavior.

Practical guidance for developers

The study offers a more useful evaluation model for teams building real-time visual assistants:

• Define task deadlines and irreversible operations before building scenarios; do not mistake a conventional VLM score for real-time competence.
• Record the full trajectory of environment state, user actions, and assistant outputs to verify whether guidance actually changes behavior.
• Report factual errors, non-actionable advice, and late responses separately rather than hiding them inside one aggregate success rate.
• Add execution errors and out-of-plan behavior to test replanning rather than only cooperative instruction following.
• Treat latency as a training reward or hard constraint, with an explicit engineering trade-off between grounding quality and response speed.

The simulator-to-reality gap remains a deployment concern. Real users may misunderstand tasks, ignore advice, or misinterpret steps; personas only approximate that noise. A model may also learn simulator-specific strategies and improve its benchmark score without helping real users. Closed-loop evaluation should therefore be followed by human testing, with continuous checks on simulator fidelity, demographic coverage, and metric calibration error.

Value and limitations

The main contribution of IVG-Bench is not a new vision-language model. It evaluates visual assistants as closed-loop interactive systems rather than offline question-answering models. The results expose three practical gaps: reliable state understanding, executable instruction generation, and timely response. A 315-scenario benchmark is still too small to cover all real-world tasks, and Gemini 4 Argon's score should not be generalized to every model.

Nevertheless, the 78.5% human agreement and the simultaneous improvement in task success and interaction quality after GRPO indicate practical research value. The next steps are to broaden task distributions, specify reward functions more rigorously, reduce simulation-to-reality bias, and measure real end-to-end benefits within explicit latency budgets.

---

Source: Hugging Face Daily Papers (2026-11-05)

— END —

Tags: #Vision-Language Models Follow #Closed-Loop Evaluation Follow #User Simulation Follow #GRPO Follow #Multimodal Agents Follow