MatrAIx AI Releases PersonaCUABench: A Benchmark for Evaluating AI Agents Serving Diverse User Personas

Summary:MatrAIx AI has introduced PersonaCUABench, a novel benchmark designed to evaluate computer-use agents (CUAs) in the context of diverse user personas. The benchmark encompasses 74 tasks across 11 domains, simulating 44 distinct personas to assess how CUAs handle user-specific requests, constraints, and preferences. Results show a 36% average drop in task completion rates when CUAs process personalized instructions, highlighting the challenges in understanding and fulfilling user-specific needs. T

---

Background and Motivation

In the field of artificial intelligence, agents are widely used for various tasks, from customer service to process automation. However, current evaluation methods typically rely on fixed task specifications, ignoring the personalized needs, constraints, or approval requirements that users may express. This limitation causes agents to underperform when handling real user requests, failing to meet actual user expectations.

Innovations of PersonaCUABench

MatrAIx AI's PersonaCUABench addresses these issues through the following approaches:

1. Diverse Persona Simulation: The benchmark includes 44 distinct personas across 4 groups, representing a wide range of user needs from novice to expert users.
2. Dynamic Instruction Rewriting: Each persona rewrites task instructions to simulate the varied ways users express themselves.
3. Agent-User Isolation: The agent only accesses dialogue content, without seeing the user profile or original instructions, reflecting real-world scenarios where agents lack prior knowledge.

Experimental Results and Findings

Across 9 model architectures and 3,330 runs, PersonaCUABench revealed the following key issues:

• Task Completion Decline: Task completion rates dropped by 36% on average compared to original requests.
• Error Source Analysis: 59.9% of errors were due to the agent misunderstanding user intent, while 11.2% were due to user-requested stops being incorrectly marked as failures.
• Low User Satisfaction: Even when tasks were completed, users gave the highest satisfaction rating in only 62.7% of cases, compared to 84.1% for original requests.

Technical Mechanisms and Engineering Trade-offs

PersonaCUABench's design focuses on simulating the complexity of real user needs rather than simple task completion. Its core mechanisms include:

• Role-Driven Instruction Rewriting: Leveraging natural language generation to integrate user persona traits into task instructions.
• Dialogue-Driven Evaluation Process: Agents make decisions based solely on dialogue content, avoiding reliance on user information.

While this design increases evaluation complexity, it more closely reflects real-world applications, providing stricter testing standards for agent development.

Developer Recommendations

1. Prioritize User Personalization: Developers should incorporate user roles and personalized needs into agent training data to improve performance in real-world scenarios.
2. Enhance Contextual Understanding: Agents need stronger natural language understanding capabilities to accurately capture user intent and respond appropriately.
3. Incorporate Multimodal Interaction: Combining visual, voice, and other modalities can further enhance agents' ability to understand user needs.

Conclusion

The release of PersonaCUABench marks a significant advancement in AI agent evaluation methods. It not only highlights the shortcomings of existing agents in handling user personalization but also provides stricter testing standards and directions for future agent development.

---

Source: Hugging Face Daily Papers (2026-10-16)

— END —

Tags: #AI Agents Follow #Benchmarking Follow #User Personalization Follow #MatrAIx AI Follow #PersonaCUABench Follow