SimAudit Finds 61 Success-Checker Defects in Robot Manipulation Benchmarks
Summary:SimAudit treats the success checker itself as the object of evaluation. An LLM compares each task instruction with its checker implementation, identifies potential semantic disagreements, reconstructs the corresponding scenarios in simulation, and runs the original checker without replacing it. Multi-view renderings are then used to verify whether the observed behavior actually satisfies the instruction. Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 simulated test cases produced the predic
SimAudit shifts the unit of evaluation from the learned policy to the benchmark's success checker. The paper generated 183 simulated test cases across RoboTwin 2.0, RoboCasa365, and RoboDojo. After visual verification and deduplication, it reports 61 distinct defects affecting 49 tasks: 40 phantom successes and 21 silent failures.
Technical Mechanism
For an execution trajectory τ, let the original checker output be c(τ)∈{0,1}, while the task instruction's semantic truth is o(τ)∈{0,1}. The benchmark score is Ŝ_checker = E[c(τ)], whereas the true task success rate is S_true = E[o(τ)]. Their difference can be written as:
Ŝ_checker - S_true = P(c=1,o=0) - P(c=0,o=1)
The first term is a phantom success: the checker accepts an incomplete task. The second is a silent failure: the checker rejects a valid completion. Phantom successes inflate reported performance, while silent failures depress it. The net direction depends on how often each error occurs for a particular task and policy, rather than simply on the aggregate number of defects.
SimAudit implements a four-stage audit loop:
- Semantic-contract extraction: An LLM reads both the natural-language instruction and the checker implementation, extracting the goal, termination conditions, required object states, and permissible alternative solutions. It then searches for possible mismatches between the instruction and the code predicates.
- Disagreement-case reconstruction: It constructs trajectories that may be accepted despite incomplete execution, as well as valid trajectories using different paths, action orders, or states, and recreates the corresponding scenarios in the robot simulator.
- Original-checker execution: SimAudit runs the checker under audit rather than replacing it. This preserves the exact implementation used by the benchmark and makes it possible to attribute defects to existing scoring logic.
- Multi-view verification and deduplication: Renderings from several viewpoints are used to inspect object states, spatial relationships, and task-completion conditions against the instruction. Cases exposing the same underlying code defect are then consolidated. The abstract does not specify whether this verification is performed by humans or a vision model, nor does it report annotation agreement.
The main contribution is therefore methodological: it turns a normally hidden evaluation component into an executable and falsifiable test target.
Engineering Trade-offs and Empirical Results
Under the accounting in the abstract, 61 confirmed defects emerged from 183 generated test cases, corresponding to a candidate-defect retention rate of roughly 33.3%. This measures the yield of the targeted generation-and-audit process; it is not the prevalence of checker failures across all tasks or checkers, because the cases were deliberately generated rather than sampled from the benchmark's natural task distribution.
Of the 61 defects, 40 are phantom successes and 21 are silent failures, or approximately 65.6% and 34.4%, respectively. The error direction also differs by benchmark. RoboCasa365 mostly contains checkers that accept incomplete executions, RoboDojo more often rejects valid alternative solutions, and RoboTwin 2.0 contains substantial numbers of both. A single benchmark-wide success rate can therefore conceal offsetting errors: the same benchmark may overstate some policies while understating others.
The LLM component provides scalable semantic reasoning over instructions and imperfect checker code, but introduces nondeterminism, prompt sensitivity, and possible hallucinated scenarios. Simulator replay and multi-view rendering add compute, storage, and scheduling costs. Images also remain an imperfect oracle for hidden physical state, contact forces, or constraints that exist only inside the simulator. Deterministic geometry predicates and direct simulator-state assertions should therefore remain the primary verification layer, with visual review reserved for ambiguous cases.
Most importantly, the paper demonstrates that checkers contain defects, but the abstract does not show how much policy rankings change after correction. Evaluating that downstream impact requires task-stratified error rates, policy-conditioned net bias, confidence intervals, before-and-after success rates, rank changes, and independent human validation.
Developer Adoption and Deployment Guidance
Benchmark maintainers should express each instruction as a testable semantic contract and add three classes of counterexamples for every checker: near-complete executions that violate a terminal condition, valid solutions using alternative paths or action orders, and trajectories with correct object identity but incorrect pose or spatial relationships. The verification oracle should read simulator geometry, contact information, and object state directly rather than invoking the checker under audit, which would create circular evidence.
For continuous integration, an LLM can generate candidate disagreements, the simulator can construct and execute the trajectories, deterministic physical assertions can issue the final verdict, and human reviewers can handle only low-confidence or visually ambiguous cases. Reports should separately track false positives, false negatives, affected tasks, and checker versions instead of publishing only an aggregate defect count.
Policy developers should inspect the original success-checker code for hard-coded expert trajectories, fixed object locations, or action-order assumptions. Evaluation should report both the raw checker score and an independent semantic-oracle score, then test whether a policy remains successful across multiple valid strategies. This prevents a policy from appearing strong merely because it reproduces one demonstrated path while failing to satisfy the actual task contract.
— END —Source: Hugging Face Daily Papers (2026-11-18)
Tags: #SimAudit #Robotic Manipulation #Simulation Benchmarks #Success Checkers #LLM Auditing
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments