Hugging Face Releases iPrefBench: Benchmarking Inference and Interaction for Personalized Mobile Agents

Summary:Hugging Face has released iPrefBench, a novel benchmark designed to evaluate the inference and interaction capabilities of personalized mobile agents. iPrefBench simulates real-world user behavior and interaction scenarios to test agents' performance in task completion and user engagement. The benchmark includes 130 underspecified tasks across 26 apps, assessing agents' ability to infer user intent and request necessary information. Experiments reveal that while agents perform well in task execu

---

1. Background and Motivation

Personalized agents represent a key direction in the development of AI assistants, with the core challenge being how to accurately understand user intent and engage in effective interaction. However, existing agents struggle with incomplete or ambiguous user requests, often failing to deliver the expected level of personalization. iPrefBench aims to address this issue by providing a more realistic testing environment that simulates real user behavior and interaction scenarios.

2. Core Mechanisms of iPrefBench

iPrefBench is built on a coherent persona with 3,051 past activity events and includes 130 underspecified user tasks across 26 apps. Each task contains user-dependent requirements that are either inferable from the user's activity history or require explicit user input. iPrefBench evaluates agents on two main aspects:

1. Personalized Task Completion: Whether the agent correctly completes the user's task.
2. User-Agent Interaction: Whether the agent requests user input when necessary.

3. Experimental Results and Findings

Evaluations of seven frontier agents reveal that while agents perform well in task execution (up to 91.6% action completion rate), they lag in personalized task completion (up to 47.7%). The main issue lies in agents' inability to request necessary information, particularly for ask-only requirements. This highlights a significant gap in agents' ability to infer user intent and choose appropriate interaction strategies.

4. Future Directions and Challenges

The release of iPrefBench opens new avenues and challenges for personalized agent research. Agents need to improve in the following areas:

1. Inference Capabilities: Enhance understanding of user behavior history to more accurately infer intent.
2. Interaction Strategies: Optimize the timing and manner of requesting user input to avoid unnecessary queries or missing critical information.
3. Personalization Modeling: Build more refined user models to achieve higher levels of personalization.

5. Developer Recommendations

For developers, iPrefBench offers a systematic evaluation method to identify shortcomings in personalized task performance. It is recommended that developers:

1. Utilize iPrefBench for continuous testing and optimization to improve agents' inference and interaction capabilities.
2. Focus on collecting and analyzing user behavior data to build more accurate user models.
3. Explore new interaction paradigms, such as context-based proactive querying and dynamic strategy adjustments.

---

Source: Hugging Face Daily Papers (2026-10-26)

— END —

Tags: #Hugging Face Follow #Intelligent Agents Follow #Benchmarking Follow #Personalized AI Follow #User Interaction Follow