ZICQ
中 Log in / Sign up
Newsroom Agentic #WebArena-Pro #Web Agents #Multimodal Evaluation #Reproducible Benchmarks #Audio and Video

WebArena-Pro Benchmarks Web Agents Across 300 Tasks, 20 Apps, and Multimodal Content

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:WebArena-Pro introduces a reproducible multimodal benchmark for web agents: 300 tasks across 20 self-hosted applications in six domains. Text, vision, audio, and video are treated as first-class observations, with dedicated extraction actions for media. Tasks execute in isolation and can be evaluated in parallel; authoring is supported by a dedicated interface, LLM-assisted triage, and final human validation. The top reported systems are Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5


WebArena-Pro extends web-agent evaluation beyond text and screenshots to browser tasks containing audio and video, while expanding coverage to 300 tasks, 20 self-hosted applications, and six domain categories. Its main contribution is not another aggregate leaderboard; it places web state, media understanding, agent actions, and isolated execution inside one repeatable evaluation stack.

Benchmark composition: from site count to task heterogeneity

The 300 tasks span 20 self-hosted web applications and six domain categories. They differ not only in page layout but also in interface conventions, business workflows, and data models. The benchmark therefore tests whether an agent can transfer its operating capabilities across heterogeneous web applications instead of memorizing DOM paths or interaction patterns for a small set of sites.

Live benchmarks can cover real websites more easily, but they are exposed to page updates, injected ads, changing business data, and server-side drift. Self-hosting gives WebArena-Pro control over these variables and allows each task to run repeatedly in a more stable environment. Self-hosting alone, however, does not guarantee reproducibility: browser versions, container images, database seeds, extensions, media assets, and retry policies must still be pinned. Otherwise, application upgrades can invalidate historical results in the same way that live-site changes do.

Technical mechanism: audio and video enter the action loop

A typical web-agent loop consumes observations, produces actions such as clicks or text input, and receives a new environment state. WebArena-Pro treats text, visual pages, audio, and video as core observations and provides dedicated actions for extracting information from media.

The design avoids serializing complete media streams into every model context. An agent can request specific evidence from a media object and use the extracted result in later decisions. The underlying implementation would typically rely on transcription, audio-event recognition, key-frame analysis, or specialized media parsers. This can bound context size and inference cost, but it creates another failure chain: transcription mistakes, omitted events, or incorrect visual parsing can propagate through subsequent actions. Audio and video tasks therefore need modality-specific scores rather than relying only on final task success.

Reproducibility engineering: isolated runs and human validation

Each task runs in isolation, and the benchmark supports parallel evaluation. Tasks are authored through a dedicated annotator interface, filtered with LLM assistance, and finally validated by humans before release. This pipeline addresses two separate risks: isolation limits cross-task state contamination, while human validation catches malformed, ambiguous, or unexecutable annotations that automated filtering may miss.

LLM-assisted triage improves annotation throughput, but it may favor fluent, conventional, or model-aligned tasks. Human review reduces this bias without eliminating it. Exact reproduction also requires the container-image digest, browser build, database reset procedure, media-file checksums, executor version, scoring model, prompt configuration, and retry policy. The supplied summary does not specify these execution details.

Reported results: framework differences remain a major confound

| Agent configuration | Reported score | |---|---:| | Codex with GPT 6.1 Sol | 62.3% | | Claude Code with Claude Opus 5.5 | 62.3% | | OpenCode with Kimi 3.1 Pro | 60.7% | | GenericAgent with Gemini 3.1 Pro omni | 30.3% | | GenericAgent with Gemma4 31B | 30.3% |

The tied 62.3% results show that coding-oriented harnesses such as Codex and Claude Code currently form the strongest end-to-end systems on WebArena-Pro. They should not, however, be read as a clean ranking of the underlying models. Codex, Claude Code, OpenCode, and GenericAgent differ in tool access, observation serialization, planning loops, context budgets, retry mechanisms, and maximum action budgets. In particular, the 30.3% GenericAgent results do not separate model limitations from differences in the surrounding agent scaffold.

The 1.6-point gap between the first and second configurations is also too small to establish a meaningful advantage from the summary alone. If the 300 tasks were approximately 300 independent binary outcomes, the standard error at p=0.623 would be sqrt(0.623×0.377/300)=2.8 percentage points, implying a rough 95% interval of plus or minus 5.5 points. Moreover, 62.3%×300=186.9 is not an integer number of successful tasks, implying that the final metric may involve partial credit, weighted aggregation, or averaging across runs. The scoring formula, number of repetitions, and confidence intervals are needed before the ranking can be interpreted.

Engineering trade-offs: reproducibility, coverage, and cost pull against one another

  • Self-hosted coverage versus maintenance: Twenty applications broaden task diversity, but they also create an ongoing burden of browser compatibility, dependency upgrades, database migrations, and state resets. Any application change can break historical comparability.
  • Multimodal completeness versus inference cost: Audio and video consume more encoding, caching, and inference resources than text. On-demand extraction limits context consumption, but information not covered by the extractor is lost.
  • Task isolation versus parallel efficiency: One environment per task improves determinism while introducing container scheduling, temporary storage, and state-cleanup overhead. Poor infrastructure can make queueing and reset time dominate evaluation time.
  • Automated triage versus task representativeness: LLM-assisted annotation scales quality control, but filtering bias may make the suite resemble the task types preferred by those models.
  • Static environments versus real-world realism: Fixed sites control variables but cannot fully reproduce login risk controls, network latency, advertising, third-party scripts, or frequent production redesigns.

Practical guidance for developers

  1. Pin the runtime before comparing models: Standardize application images, browser builds, database seeds, reset logic, and action logs that capture DOM state, screenshots, media extraction, and every issued action.
  2. Publish a multidimensional scorecard: Report text, vision, audio, and video results separately, alongside token cost, end-to-end latency, action count, retries, and human-intervention rates.
  3. Use a factorial evaluation design: Cross multiple models with multiple agent harnesses while holding observation interfaces, context budgets, tool permissions, and action limits constant. This separates model gains from scaffold gains.
  4. Preserve hidden tasks and versioned fixtures: Maintain separate development, hidden test, and regression splits so prompts and routing policies cannot overfit to the 300 public tasks.
  5. Use the benchmark as a CI gate: Track success-rate regressions, trajectory quality, and cost changes rather than treating a leaderboard position as the sole release criterion.

Source: Hugging Face Daily Papers (2026-11-13)

— END —

Tags: #WebArena-Pro #Web Agents #Multimodal Evaluation #Reproducible Benchmarks #Audio and Video

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…