ZICQ
中 Log in / Sign up
Newsroom Agentic #AI Agents #Web Agents #Agent Evaluation #Live-Web Verification #Deliverable Assessment

EverydayBench Evaluates Evidence-Backed Completion of Live Web Tasks

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:EverydayBench evaluates whether AI agents can complete live-web tasks satisfactorily, rather than merely retrieving relevant information. Its 77 human-authored tasks cover eight everyday domains and define success criteria before execution without prescribing either the answer or the workflow. Evaluation combines deterministic constraint checks, verification of claims against cited pages retrieved at evaluation time, and rubric-based assessment of deliverable quality. Across 23 reported agent-m


Core Finding

EverydayBench shifts the unit of agent evaluation from retrieving relevant information to delivering an outcome that satisfies explicit constraints, remains supported by evidence, and meets a quality threshold. The benchmark contains 77 human-authored tasks across eight everyday life and work domains. Success criteria are fixed before execution, while neither the answer nor the workflow is prescribed, allowing agents to select their own products, itineraries, research conclusions, or artifacts.

This is closer to the behavior required from production agents. Users care whether the final result is actionable and complete, not how many relevant pages an agent encountered during the process.

Reported Results

| Evaluation | Reported result | Engineering interpretation | |---|---:|---| | Highest strict pass rate | Claude Opus 5.5 and Claude Fable 5.1 both reach 54.55% | Even the best configurations complete only about half of the tasks, leaving substantial room for improvement | | GPT-6 Sol reasoning-effort sweep | 29.87% at low effort and 45.45% at maximum effort | More reasoning adds 15.58 percentage points but does not surpass the best overall configurations | | Lookup tasks | GPT-5.6 Sol leads | Capability varies materially by task type, so one aggregate score is insufficient for model selection | | Planning tasks | Fable 5.1 leads | Retrieval quality and task decomposition are distinct capabilities | | Comparison tasks | Opus 5.5 and Fable 5.1 tie | Stronger general reasoning does not produce a winner across every task family | | Customized deliverables | GPT-6 Astra and Sonnet 5.5 tie | Producing usable artifacts requires a different capability profile from retrieving information |

Using 77 tasks as a common denominator, 54.55% corresponds to approximately 42 tasks, 45.45% to 35 tasks, and 29.87% to 23 tasks. Even under a simplified independent-Bernoulli interpretation, the Wilson 95% interval for 42 out of 77 is approximately 43.5% to 65.2%. Repeated executions, task correlation, live-web variation, and scorer noise would add further uncertainty. The supplied abstract does not report repetition counts, task-level distributions, variance, token usage, latency, or tool cost, so these figures should be treated as directional evidence rather than a stable model leaderboard.

How the Evaluation Works

1. Deterministic Constraint Checks

The first layer checks explicit constraints declared before task execution, such as counts, ranges, required fields, or other programmatically verifiable properties. Machine-checkable facts should not be delegated to a subjective judge, reducing hallucination and prompt sensitivity.

The tradeoff is higher task-authoring cost. Each condition must be sufficiently precise and executable without collapsing the benchmark into exact-answer matching. Production implementations need dedicated parsers, failure reasons, and exception handling for every constraint; otherwise the evaluator itself becomes a source of hidden errors.

2. Evaluation-Time Web Evidence Verification

The second layer retrieves the pages cited by the agent at evaluation time and checks whether they support its claims. This is stronger than checking whether a URL opens: a reachable page may no longer contain information that justifies the original statement.

This design introduces live-web realism but also creates a temporal-consistency problem. Prices, inventory, travel conditions, or policies may change between agent execution and evaluator retrieval. The paper separates this source of variation from scorer instability by repeating scoring under fixed evidence and uses controlled grounding failures to test the behavior of the evidence chain when grounding is deliberately corrupted.

If strict passing uses the usual conjunction of all three layers, an implementation can be represented as:

StrictPass(i) = ConstraintCheck(i) ∧ EvidenceCheck(i) ∧ DeliverableQuality(i)

The abstract does not disclose the exact aggregation formula, thresholds, or missing-value rules. Reimplementations should not silently assume a particular interpretation.

3. Rubric-Based Deliverable Assessment

The third layer uses rubrics to assess whether the final artifact is complete, clear, usable, and aligned with the user's request. This captures experience dimensions that hard constraints and factual verification cannot measure, but it also introduces scorer drift, incomplete rubric coverage, and subjective bias.

If a model acts as the assessor, its model version, prompt, temperature, and structured-output schema should be pinned and calibrated against blinded human judgments. If humans perform the scoring, the benchmark should report reviewer counts, agreement statistics, and review cost. The supplied material does not establish which implementation was used.

Why the Protocol Matters

Registering success criteria before execution reduces the evaluator's freedom to move the goalposts after seeing an answer. Omitting a prescribed answer or workflow allows multiple valid strategies and measures outcome selection rather than benchmark memorization. Repeated scoring under fixed evidence further separates changing web content from scorer noise, giving the protocol a basic diagnostic structure.

Together, these choices extend agent evaluation beyond static question answering. The agent must not only find information; it must submit a deliverable that satisfies the request, remains supported by evidence, and is genuinely usable.

Engineering Trade-offs

| Design choice | Benefit | Cost and mitigation | |---|---|---| | Use the live web | Closely approximates production agents and tests current information | Pages, ads, paywalls, and rendering state can drift; preserve fetch times, snapshots, and content hashes | | Permit agent-selected outcomes | Measures strategy choice and completion rather than answer matching | The valid-answer space is wider, making comparison more rubric-dependent; fix success criteria rather than exact answers | | Report strict pass rates | Hard-constraint and evidence failures cannot be hidden | Nearly complete work may still score zero; also publish constraint, evidence, and quality component scores | | Assess artifacts with rubrics | Captures usability and completeness | Sensitive to judge version, prompt, and domain-expert calibration | | Use only 77 tasks | Keeps human authoring and protocol validation manageable | Too small for stable estimates of domain-specific behavior and long-tail failures | | Aggregate into one leaderboard | Enables simple comparison | Hides task heterogeneity; use domain-level results and cost alongside the aggregate metric |

Prompt injection is another production concern. A cited page may contain text instructing the evaluator to ignore its rules or alter a score. The evidence retriever must treat pages as untrusted data, isolate embedded instructions, and prevent web content from controlling judge prompts, tool permissions, or evaluation records.

Deployment Guidance

  1. Use a two-track evaluation setup. Run CI against timestamped, content-hashed page snapshots for stable regression testing, then run a live-web canary before release to detect changes in inventory, prices, and policies.
  2. Pin the entire evaluation stack. Freeze the task set, success criteria, constraint parsers, rubrics, judge model, retrieval region, locale, user agent, and fetch timestamp so environmental drift is not mistaken for model progress.
  3. Retain the complete evidence chain. Store agent outputs, cited pages, fetched content, claim-to-evidence mappings, component scores, and failure reasons rather than only the final strict-pass flag.
  4. Expose component metrics. Report constraint pass rate, valid-citation rate, evidence-support rate, deliverable-quality score, strict pass rate, token use, latency, and external-tool cost.
  5. Use paired statistics. Comparisons on the same tasks should rely on paired bootstrap, McNemar tests, or stratified analysis instead of treating each model's task outcomes as independent samples.
  6. Keep human audits. Sample evidence checks and rubric scores independently, report inter-rater agreement and disagreement cases, and pay particular attention to high-value, high-risk, or time-sensitive tasks.
  7. Expand task coverage. Beyond the 77 tasks, add realistic failure distributions, long-horizon tasks, cross-site operations, dynamic pages, adversarial webpages, and impossible requests, while separating model failures from tool and network outages.

EverydayBench's primary contribution is an evaluation protocol rather than a new model architecture. It places open-ended outcomes, dynamic evidence, and hard constraints into one auditable loop. The benchmark is useful for selecting agent stacks intended to perform real web tasks, but it should be combined with static browser sandboxes, business regression suites, cost measurements, and human acceptance testing rather than used as a standalone procurement or deployment score.


Source: ArXiv AI (cs.AI) (2026-11-20)

— END —

Tags: #AI Agents #Web Agents #Agent Evaluation #Live-Web Verification #Deliverable Assessment

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…