FINCOSMOS Exposes Unreliable Financial Agents: None of 150 Sessions Passed
Summary:FINCOSMOS introduces an open-world evaluation paradigm for financial agents operating on live Web data. Agents must discover data, reconstruct a user's hidden strategy, build point-in-time-correct backtests, and deliver independently verifiable results. Grading relies on executable checks rather than a reference answer: the system recomputes reported metrics, validates the hidden strategy specification, and rejects runs that fail any of 15 veto checks. In 150 sessions involving four coding agen
Why financial-agent evaluation cannot rely on polished reports
FINCOSMOS shifts financial-agent evaluation from judging whether an answer looks plausible to determining whether the result can be independently verified. In finance, an agent may collect data from the open Web while misunderstanding a strategy the user never explicitly specified, or leak future information into a backtest. A fluent report can therefore package a fundamentally flawed analysis.
Conventional benchmarks usually fix the task, the data, or the timeline, thereby removing some of the failures that matter in deployment. FINCOSMOS deliberately preserves three sources of uncertainty: data must be acquired from the Web, the strategy exists partly in the user's head, and correctness depends on temporal constraints.
Evaluation design: hidden strategies, executable grading, and veto checks
A compositional grammar generates hidden strategies that are held by a simulated user. Agents must actively ask questions to recover the missing details rather than receive a complete specification or prepared dataset.
An accepted run must complete an end-to-end workflow:
- Acquire the required market or company data from the open Web.
- Reconstruct the intended strategy from user feedback instead of applying a generic investment playbook.
- Enforce an explicit data cutoff so that backtests cannot read future information.
- Produce an executable backtest and submit its results.
- Submit to program-level validation of temporal correctness, strategy consistency, and reported metrics.
FINCOSMOS does not require a reference answer for every task. Executable validators apply point-in-time rules, recompute reported returns, risks, and other metrics, and compare the delivered implementation with the hidden strategy specification. A run is accepted only when the simulated user is satisfied and none of 15 veto checks fails. User approval is therefore necessary but cannot override hard failures such as data leakage, inconsistent metrics, or strategy substitution.
Results: plausible reports still fail audit
The study evaluated four coding agents and one model without tools across 150 sessions. No session was accepted. More strikingly, every run explicitly approved by the simulated user still failed at least one program check, showing that natural-language satisfaction cannot substitute for executable auditing.
Agents proactively surfaced only about 54%—64% of the details they had not been told. They frequently continued under unresolved requirements while presenting assumptions as conclusions. Human experts attributed 88.33% of the failures to the agents themselves, with the remainder involving the simulated users, data environment, or task setup to a greater extent.
The feedback loop nevertheless showed promise: an agent with persistent memory learned all ten tasks after receiving FINCOSMOS feedback. This suggests the central problem is not simply missing financial knowledge. Current agents lack stable mechanisms for attributing failures, preserving experience, and applying corrections at the task level.
Engineering implications: add an auditable execution layer
FINCOSMOS's main engineering contribution is an executable and rejection-capable verification chain for financial agents. Research teams and platform providers can adapt it into the following production gates:
- Data-lineage checks: Record the timestamp, source, and transformations for every Web extraction.
- Look-ahead detection: Prevent backtests from accessing observations beyond the cutoff.
- Metric recomputation: Re-run strategy code in a sandbox instead of trusting model-generated metrics.
- Specification matching: Compare the hidden strategy, user approval record, and actual execution logic.
- Veto-based acceptance: Reject data leakage, strategy replacement, and irreproducible output regardless of user confidence.
- Persistent error memory: Store failure causes as retrievable strategy-level lessons rather than preserving only final answers.
This design fits research assistants, financial-analysis agents, and automated investment-research pipelines. Models can propose strategies, gather evidence, and generate code, while deterministic programs constrain execution. Once model persuasion is separated from policy enforcement, even a highly capable model cannot bypass data and temporal audits.
Limitations and next steps
The findings require replication across more models, real financial websites, and longer time horizons. The framework currently relies on simulated users and synthetic strategies, so the grammar's coverage may constrain generalization. Live Web data also introduces dynamic-page, copyright, site-change, and provider-terms issues.
Moreover, the thresholds and false-positive rates of the 15 veto checks have not been fully reported. Programmatic validators can themselves miss subtle leakage or ambiguous strategy interpretations. The claim that a persistent-memory agent learned all ten tasks also needs separation into benchmark memorization, prompt-level adaptation, and genuine cross-task generalization.
Even with these caveats, FINCOSMOS establishes a more deployment-relevant direction for financial-agent evaluation: the agent must prove that its data is trustworthy, its timeline is correct, its strategy matches the request, and its metrics are reproducible. If any link fails, neither user approval nor the model's own narrative is sufficient.
— END —Source: Hugging Face Daily Papers (2026-11-05)
Tags: #Financial Agents #Open-World Evaluation #Verifiable Backtesting #Agent Auditing #Persistent Memory
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments