AutoGUIWorld Released: Synthesizing GUI Interaction Trajectories Without Software Deployment

Summary:AutoGUIWorld is an innovative data generation framework that synthesizes high-quality GUI interaction trajectories without deploying or running the corresponding software environments. By combining the visual priors of image generators with the task knowledge of a planner, it generates GUI scenes and actions iteratively. Experimental results demonstrate that fine-tuning Qwen3.5-35B-A3B on trajectories generated by AutoGUIWorld improves the mean task score on OSWorld from 33.0% to 40.8% and the t

---

Core Breakthrough

AutoGUIWorld is a novel data generation framework designed to address the challenge of insufficient interaction trajectory data in GUI agent training. Its key features include:

• No Need for Actual Software Environments: By leveraging the synergy between image generators and planners, AutoGUIWorld can generate high-quality GUI interaction trajectories without deploying or running the target software.
• Multi-Platform Support: The framework supports major operating systems and browsers such as Ubuntu, Windows, macOS, and Chrome, providing a wide range of application scenarios.
• High-Quality Data Generation: The generated trajectories undergo meticulous quality filtering and spatial annotation, ensuring the quality and reliability of the training data.

Technical Highlights

1. Integration of Image Generation and Planning: AutoGUIWorld combines the visual priors of image generators with the task knowledge of planners, achieving a complete closed loop from scene generation to action planning.
2. Multi-Platform Compatibility: It supports various operating systems and browsers, offering extensive application scenarios.
3. High-Quality Training Data: The generated trajectory data is meticulously quality-filtered and spatially annotated, ensuring the quality and reliability of the training data.

Applications and Impact

The release of AutoGUIWorld provides a new data generation method for GUI agent training, significantly enhancing the performance of agents in real desktop and scientific tasks. Specific application scenarios include:

• Automated Testing: By generating rich interaction trajectories, AutoGUIWorld can be used for automated testing, simulating user operations, and detecting software defects.
• Intelligent Assistant Development: It provides richer training data for intelligent assistants, enhancing their interaction capabilities in GUI environments.
• Human-Computer Interaction Research: It offers new data sources and methods for scholars researching human-computer interaction, promoting the development of related fields.

Developer Recommendations

• Try Fine-Tuning: Developers are advised to use the trajectory data generated by AutoGUIWorld to fine-tune existing GUI agent models to improve their performance.
• Explore Multi-Platform Applications: Utilize the multi-platform support feature of AutoGUIWorld to explore applications in different operating systems and browsers.
• Combine with Other Technologies: Combine AutoGUIWorld with other AI technologies such as reinforcement learning and imitation learning to explore more complex application scenarios.

Conclusion

The release of AutoGUIWorld provides a new solution for GUI agent training, showcasing the significant potential of generated trajectories in enhancing agent performance. As technology continues to advance, AutoGUIWorld is expected to play an important role in areas such as automated testing, intelligent assistant development, and human-computer interaction research.

---

Source: Hugging Face Daily Papers (2026-10-01)

— END —

Tags: #AutoGUIWorld #GUI Agents #Data Generation #Image Generation #Multi-Platform Support