ZICQ
中 Log in / Sign up
Newsroom Agentic #GMA #Intelligent Agents #Multimodal #Benchmark #Open Source AI

GMA Benchmark Released: Evaluating General Mobile Assistants in Real-World Scenarios

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 4 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:The GMA (General Mobile Assistants) benchmark has been released to evaluate multimodal AI agents in challenging real-world scenarios. Built on seven open-source projects spanning domains like lifestyle sharing and travel planning, GMA introduces 300 tasks across four difficulty tiers, from atomic actions to complex multi-step workflows. Evaluation of eight frontier models reveals a substantial decline in performance as task complexity increases, highlighting the gap between current agents and re


Overview

The GMA (General Mobile Assistants) benchmark has been released to address the limitations of existing mobile agent evaluation benchmarks in terms of application coverage and task design. Built on seven open-source projects, GMA spans domains like lifestyle sharing and travel planning and introduces 300 tasks across four difficulty tiers, from basic actions to complex multi-step workflows.

Technical Highlights

  1. Multimodal Task Design: GMA incorporates multimodal interaction tasks that simulate real-world complexity, including text, image, and audio inputs.
  2. Difficulty Tiers: Tasks are categorized into four difficulty levels, providing a comprehensive evaluation framework.
  3. Open-Source Integration: The platform's foundation on open-source projects ensures its openness and extensibility.
  4. Performance Evaluation: Evaluation of eight frontier models shows a significant decline in performance as task complexity increases, indicating a gap in handling realistic user requirements.

Ablation Studies

GMA also conducts ablation studies on agent harness choices, such as context retention and explicit state tracking. The results demonstrate that appropriate harness design can meaningfully improve performance on demanding workflows, though the effectiveness of specific designs varies across foundation models.

Industry Impact

The release of GMA provides a more challenging testing environment for agent development and promotes research into the reliability of agents in complex mobile workflows. Its open-source nature allows researchers and developers to extend and customize test tasks, thereby accelerating advancements in agent technology.

Developer Recommendations

  • Focus on Multimodal Interaction: Developers should prioritize the design of multimodal interaction tasks to enhance agents' adaptability in real-world scenarios.
  • Optimize Harness Design: Properly applying constraints like context retention and state tracking can significantly improve model performance.
  • Engage with the Open-Source Community: Actively participate in the GMA open-source community to share test results and optimization strategies, fostering collective progress in agent technology.

Source: ArXiv AI (cs.AI) (2026-09-01)

— END —

Tags: #GMA #Intelligent Agents #Multimodal #Benchmark #Open Source AI

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…