ZICQ
中 Log in / Sign up
Newsroom Research & Papers #ForceHarness #Robot Manipulation #Force Feedback #Frozen Multimodal Models #HiddenPhys

ForceHarness Gives Frozen API Models Force Feedback for Zero-Shot Robot Control

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:ForceHarness is a systems layer that makes force feedback usable by frozen, API-only multimodal models. It closes the fast contact-control loop below the model with self-terminating force primitives, then translates each force trace into structured physical quantities with uncertainty and returns them as a tool result. This design avoids both high API latency and the need for force-specific demonstrations or fine-tuning. On a seven-task MuJoCo benchmark called HiddenPhys, the authors report gai


Core mechanism: separating semantic planning from the fast contact loop

ForceHarness is not a new vision-language model. It is a control-adaptation layer placed between a frozen API-only multimodal model and a robot's actuators. The central problem is that a frontier model can infer an action from images but cannot see contact forces, while physical contact often begins and evolves before a remote API request completes.

The system operates across two time scales:

  • Semantic layer: the model selects a force operation from visual observations and the task instruction.
  • Contact layer: a local force primitive regulates force, detects a contact condition, terminates itself, and returns the contact event and force trace.

The model does not need to consume raw force samples at every control cycle. Instead, it receives a structured tool result after a primitive finishes. This reduces the impact of API latency on contact regulation and avoids collecting large force-aware demonstration datasets or fine-tuning a frozen model specifically for tactile reasoning.

1. Force acquisition and calibration

ForceHarness can obtain hand or end-effector force from a wrist sensor, or infer external force from joint torques alone when no wrist sensor is available. These paths have different cost and error profiles. A wrist sensor measures force more directly but adds hardware, cabling, and calibration overhead. A joint-torque estimator reuses the robot's existing sensors, yet its accuracy depends on the manipulator dynamics, friction, payload, and collision geometry. Changes in payload or model mismatch can therefore become contact-force estimation errors.

The reported real-arm result without a wrist force sensor thus depends on both multimodal reasoning and the quality of force reconstruction. The supplied abstract does not disclose sensor range, filtering, the torque-to-force estimator, or calibration error. The 90% success rate should not be interpreted as a consequence of prompting alone.

2. Self-terminating force primitives

Self-terminating primitives are the core mechanism for handling fast contact dynamics. After the model issues a high-level operation, a local controller regulates force using sensor feedback and ends the operation when a predefined contact condition is met, without waiting for another remote API call.

From a control-engineering perspective, such a primitive can be understood as a loop around an error signal such as e(t) = F_measured(t) - F_target, combined with termination and event-detection logic. However, the abstract does not identify whether the implementation uses PID, impedance control, or another method, and it provides no sampling rate, gains, stopping thresholds, or convergence guarantee. Stability claims therefore cannot be inferred from the summary alone.

The hierarchy provides a clear latency benefit: even when a model round trip takes hundreds of milliseconds, a local high-frequency loop continues to regulate contact. The trade-off is reduced semantic adaptability. A primitive that terminates too early may miss the contact, while one that terminates too late can apply excessive force, continue squeezing an object, or damage the target.

3. Physics translation layer

Raw force streams are high-frequency, continuous, and tied to device-specific units. Passing them directly to an API wastes context and is difficult for a frozen model to interpret. ForceHarness instead converts each force trace into physical quantities with uncertainty and returns them as a tool result.

This translation changes the model task from reading and interpreting a force waveform to reading a physical state and choosing the next operation. It follows the tool-calling pattern common to embodied agents. The reported ablation—smaller models making significantly better decisions when given physical quantities rather than raw force—indicates that structured representation is a central component rather than a cosmetic preprocessing step.

The abstraction also discards some high-frequency tactile detail. Errors in calibration, outlier filtering, or uncertainty estimation can produce incorrect beliefs about contact intensity, sliding, or collision state. The abstract also does not establish that the model was explicitly trained to use the uncertainty field; merely returning uncertainty does not prove that downstream decisions are risk-aware.

Key engineering trade-offs

Control bandwidth versus task adaptability

Moving the fast force loop below the model is the system's most effective design choice. It avoids asking a remote model to perform millisecond-scale servo control, but it also limits the contact phase to what predefined primitives can express. Insertion, precision assembly, and deformable-object manipulation may require continuous visual-force fusion beyond a fixed primitive library and should be evaluated separately.

Sensor accuracy versus deployment cost

A wrist force sensor provides a more direct signal but increases hardware and calibration requirements. Joint torque alone minimizes added hardware while injecting model error, friction, and payload variation into the feedback path. Deployment should measure both absolute estimation error and contact-detection delay, not just final task success.

Physical abstraction versus detail loss

Structured quantities reduce cognitive load and improve zero-shot behavior for smaller models, but high-frequency impacts, microscopic slips, and contact texture may be lost. Tasks requiring material recognition or fine assembly may need a hierarchical interface that combines low-frequency physical state with discrete high-frequency contact events.

Frozen-API convenience versus behavioral drift

No fine-tuning lets ForceHarness connect to models from different providers and reduces training-data requirements. It does not eliminate sensitivity to API version changes, output nondeterminism, network tail latency, or service limits. Production logs should record the model version, prompt, tool arguments, and timestamps for every call; otherwise it is difficult to determine whether a success came from model reasoning or an accidental execution path.

Benchmark results: strong claims, incomplete evidence chain

| Evaluation | Experimental setup | Reported result | |---|---|---| | Real arm without a wrist force sensor | GPT-6 Astra zero-shot manipulation | Success rises from 3.3% to 90.0%, an absolute gain of 86.7 percentage points | | Public ForceBench | Same-condition comparison with controls | Success improves by 17.5 percentage points | | HiddenPhys, a seven-task MuJoCo benchmark | Compared with expert controllers and force-aware policies | Success improves by 34.3 points, exceeds policies trained from force-aware demonstrations, and matches handwritten expert controllers on HiddenPhys | | Model-scale ablation | Physical-quantity tool result versus raw force values | Smaller models make significantly better decisions when given physical quantities |

These results support the effectiveness of the system composition, not a new foundation-model capability in GPT-6 Astra. An image-only baseline cannot observe contact forces, so a large improvement after adding force information is consistent with an information bottleneck in robotic manipulation. The HiddenPhys result, where the system exceeds force-aware policies trained from demonstrations, further suggests that a physical-quantity interface can replace part of task-specific policy learning.

The abstract does not provide trial counts, confidence intervals, random seeds, per-task scores, model sampling settings, API latency, sensor noise, payload configuration, or the success criterion. It is therefore impossible to determine whether the 90% result is highly sensitive to a particular task, arm, or prompt. The isolation of HiddenPhys's hidden physical variables and the extent of benchmark-specific harness tuning also require a full-paper and code audit to rule out hidden overfitting.

Task success alone is not a sufficient deployment metric. Two systems with the same success rate can differ substantially in peak contact force, completion time, false-contact events, and motion smoothness. The summary does not report force overshoot, collision rate, cycle time, API-call count, human-intervention rate, or recovery behavior after failure.

Deployment guidance for developers

1. Establish interpretable baselines

Test image-only control, raw-force input, and ForceHarness physical-quantity input while holding the model version, prompt, objects, and success criterion constant. This separation is necessary to distinguish the value of newly available force information from the value of the harness's control strategy.

2. Treat calibration and drift detection as release blockers

For both wrist sensing and joint-torque inference, log static bias, repeatability, thermal drift, payload sensitivity, and collision-point error. Tool results should include physical units, measurement time, uncertainty, termination reason, and calibration version so the model does not mistake drift for a genuine contact-state change.

3. Define explicit tool and failure semantics

A useful result schema should contain target force, measured force, rate of change, uncertainty, contact status, termination reason, and timestamp. The model must be able to distinguish reaching the target, detecting no contact, exceeding a limit, and sensor failure rather than compressing every unsuccessful state into one Boolean value.

4. Keep safety outside the model

Force limits, joint position and velocity constraints, workspace boundaries, watchdog timers, and a physical emergency stop should run independently of the API model. The model may request an action but must not override the lower-level safety envelope. High-risk deployments should also cross-check contact events with an independent collision or motor-current sensor.

5. Evaluate multiple operational metrics

Alongside task success, track peak force, force RMSE, overshoot rate, false-contact rate, time to first contact, task duration, API tail latency, intervention rate, and cost per successful task. Simulation should vary contact stiffness, friction, payload, sensor noise, actuator delay, and visual occlusion before gradual real-world transfer.

Engineering conclusion

ForceHarness's main contribution is not a stronger robot model, but a clearer division of responsibility among a frozen multimodal model, a local controller, and a physics-aware tool interface: the model performs high-level semantic reasoning, local primitives close the fast contact loop, and the translation layer converts sensor traces into a representation the model can use. The real-arm and HiddenPhys results make this approach worth reproducing. Until variance data, controller details, cost measurements, and public implementations are available, however, the reported 90% should be treated as an author-reported system result rather than a generalized capability ceiling.


Source: Hugging Face Daily Papers (2026-11-16)

— END —

Tags: #ForceHarness #Robot Manipulation #Force Feedback #Frozen Multimodal Models #HiddenPhys

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…