arXiv Releases MemU-Bench: Benchmarking GUI Agents on Long-Horizon User Interactions
Summary:arXiv introduces MemU-Bench, a novel benchmark framework for evaluating GUI agents in handling vague and history-dependent user instructions. The benchmark consists of 118 tasks grounded in 29 long-horizon user trajectories, aiming to assess agents' ability to distill interaction histories and retrieve missing context. Evaluations show that state-of-the-art systems struggle under such ambiguity, with the best-performing Gemini-3.1-Pro achieving only 16.4% SPL. However, the introduction of the Ap
Background and Challenge
In everyday user interactions, instructions are often vague and history-dependent, with users expecting agents to recover missing context from long interaction histories without explicit details. However, existing GUI benchmarks predominantly rely on precise and stateless instructions, making it difficult to comprehensively evaluate agents in real-world scenarios.
MemU-Bench Framework
To address this gap, arXiv introduces MemU-Bench, a benchmark framework comprising 118 tasks grounded in 29 long-horizon user trajectories. Task outcomes are verified against serialized application databases to ensure evaluation accuracy. Additionally, MemU-Bench introduces two new metrics:
- Interaction Efficiency (IE): Measures the number of interaction steps required to complete a task.
- Success weighted by Path Length (SPL): Weighs the success rate by the path length to expose behaviors that rely on exhaustive searching rather than genuine memory recall.
Experimental Results and Findings
Experiments show that state-of-the-art systems struggle with vague instructions, with Gemini-3.1-Pro achieving only 16.4% SPL. However, the introduction of the AppMem framework boosts GPT-5.5's SPL to 66.9%, demonstrating a significant performance improvement. This enhancement not only highlights the effectiveness of AppMem in handling vague instructions but also provides new insights for future agent development.
Engineering Trade-offs and Practical Applications
The strength of the AppMem framework lies in its ability to achieve efficient context recovery without training, making it highly practical for resource-constrained environments. However, its performance relies on accurate modeling and inference of user history data, which may pose higher demands on data preprocessing and inference speed.
Developer Recommendations
For developers, the AppMem framework offers a pathway to enhance agent performance without complex training. It is recommended for use in the following scenarios:
- Resource-constrained devices: Such as mobile devices or embedded systems.
- Tasks requiring long-horizon user interactions: Such as intelligent assistants or automated customer service systems.
- Applications with high real-time requirements: Such as autonomous driving or robot control.
— END —Source: ArXiv AI (cs.AI) (2026-11-03)
Tags: #arXiv #GUI Agents #Long-Horizon Interaction #Vague Instruction Processing #AppMem
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments