Hugging Face Proposes Reviso Framework: Priority-Guided Active Perception for Video Captioning

Summary:Hugging Face introduces Reviso, a novel video captioning framework that enhances caption quality and context coherence by learning to selectively observe video content and gather relevant visual evidence. Reviso employs a hierarchical Context-Focus-Points (CFP) observation plan, combining active perception and goal-directed tool usage to improve caption generation. Experiments across four benchmarks demonstrate Reviso's superior performance and its potential in video understanding tasks.

---

Core Mechanisms of Reviso

Reviso achieves video captioning through the following steps:

1. Global Preview and CFP Plan Construction: Reviso begins with a low-frame-rate global preview of the video to construct a hierarchical Context-Focus-Points (CFP) observation plan. This plan identifies the video's context, prioritizes key aspects, and defines specific observation targets and corresponding tools.

2. Active Perception and Evidence Collection: Guided by the CFP plan, Reviso selectively revisits temporal segments of the video and combines multiple tools for fine-grained inspection. This process ensures that Reviso collects visual evidence highly relevant to the video content.

3. Evidence Interpretation and Plan Update: Reviso interprets the collected evidence and updates its observation plan as needed. This mechanism allows Reviso to dynamically adjust its perception strategy to adapt to changes in the video content.

4. Global and Local Information Integration: Reviso integrates global and local information to generate the final caption. This process ensures the accuracy and contextual coherence of the caption.

Innovations and Advantages

• Priority-Guided Active Perception: Reviso learns the importance of video content to prioritize key information and avoid overemphasizing minor details.

• Hierarchical Observation Plan: The CFP plan allows Reviso to systematically organize its perception process, ensuring a comprehensive understanding of the video content.

• Tool Combination and Fine-Grained Inspection: Reviso can combine multiple tools for fine-grained inspection, collecting richer visual evidence.

• Verification-Guided Training Pipeline: Reviso employs a verification-guided training pipeline, constructing high-quality training trajectories through factual verification and teacher-guided refinement.

• Reward Mechanism Optimization: Reviso's training process incorporates a key-fact reward, a Plan Alignment Reward (PAR), and an Unproductive Tool-Call Penalty (UTCP) to optimize its observation strategy.

Experimental Results and Performance

Reviso outperforms evaluated open-source baselines across four video captioning benchmarks, with captions that are more accurate and contextually coherent. Additionally, Reviso's captions better support downstream video question answering tasks, demonstrating its potential in multimodal understanding.

Developer Recommendations

• Model Fine-Tuning: Developers can fine-tune Reviso for specific application scenarios to further enhance its performance.

• Tool Extension: Reviso's architecture supports the combination of multiple tools, and developers can extend its tool library to accommodate different task requirements.

• Cross-Modal Applications: Reviso is not limited to video captioning and can be applied to other multimodal tasks such as video question answering and video retrieval.

---

Source: Hugging Face Daily Papers (2026-11-02)

— END —

Tags: #Hugging Face Follow #Video Captioning Follow #Multimodal AI Follow #Active Perception Follow