AI Agents Audit 324 Empirical Finance Papers Across 62,409 Specifications
Summary:Researchers used AI agents to reconstruct data for 13 reused natural experiments, attempt to reproduce the main estimates in 324 empirical finance papers, and re-estimate those claims across 62,409 defensible specifications. Under the papers' original setups, only 42% of signed main estimates were reproduced. Across the full grids, an average of 25% of specifications preserved the published sign and reached p<0.10, exposing substantial sensitivity to defensible analytical choices. Regression-di
What the study found
The research applies agentic coding to a reproducibility audit of empirical finance: 13 heavily reused natural experiments, 324 papers, and 62,409 defensible specifications. The objective is to measure how much published conclusions depend on data reconstruction, model specification, and other analytical choices.
| Measure | Reported result | Interpretation | |---|---:|---| | Corpus | 324 papers across 13 natural experiments | Reused identification designs provide a useful setting for testing robustness across outcomes | | Specification grid | 62,409 estimates | Agents extend one-off replication into large-scale specification search | | Reproduction under original specifications | 42% | Most signed main estimates were not reproduced even under the papers' own setups | | Full-grid robustness | 25% on average | Only one quarter of specifications preserved both the published sign and p<0.10 | | Design comparison | RDD was more robust than DiD on average | The association is suggestive, but the abstract does not establish a causal advantage of one design | | Other correlates | First-order effects, larger t-statistics, available data or code, top-journal papers, and experiments with fewer earlier papers were more robust | Open materials, significance, and publication tier may jointly capture research quality and selection effects |
The 42% reproduction rate must not be read as an automatic falsification rate for the remaining claims. Missing data versions, dependency drift, undocumented preprocessing, and incorrect agent-generated code can all cause a replication to fail. A defensible interpretation of robustness requires the agent to reconstruct the data, verify the estimand, and validate the generated implementation before varying the specification grid.
How agentic multiverse analysis works
The abstract does not disclose the agent model, prompts, orchestration, or toolchain, so the paper's exact implementation cannot be reconstructed. At the capability level confirmed by the abstract, however, the pipeline must perform the following sequence:
- Parse the paper and estimand: Extract the sample, treatment group, time window, fixed effects, and standard-error definition, then translate the natural-language research design into machine-checkable estimation targets.
- Rebuild the data: Recover the analysis sample from raw sources, author code, or public archives while recording provenance, versions, transformations, and row-level lineage.
- Reproduce the main estimate: Execute agent-generated code under the original specification and validate equivalence through variable dimensions, observation counts, coefficients, standard errors, and other invariants.
- Construct a specification grid: Vary defensible dimensions such as sample windows, controls, fixed effects, functional form, timing, and clustering. The abstract does not identify the grid actually used, so these examples should not be presented as the paper's confirmed design.
- Run estimates and recover from failures: Isolate each specification, capture logs, dependency versions, resource use, and exceptions, and permit only bounded retries. The agent must not silently alter the statistical estimand while repairing code.
- Aggregate robustness: For each paper, combine original-specification reproduction, specification curves, and the proportion of estimates that preserve sign and statistical significance, then compare results across designs and paper-level strata.
The abstract's central robustness rate can be approximated as ρ_i = (1/|G_i|) Σ_g 1[sign(β_g)=sign(β_pub) and p_g<0.10], where G_i is the defensible grid for paper i, β_pub is the published estimate, and β_g is a grid estimate. The metric is useful as a description of dependence on analytical choices. It does not measure effect-size bias, confidence-interval coverage, or estimator differences, and it should not replace full specification curves.
Engineering trade-offs
The primary benefit of agents is throughput, not automatic statistical correctness. Four trade-offs matter:
- Speed versus provenance: Agents can rapidly generate, rewrite, and parallelize analysis code, but every data transformation needs a structured manifest and content hash. Otherwise, a single coding error can contaminate thousands of estimates.
- Autonomy versus estimator equivalence: R, Python, and Stata can differ in missing-value handling, weights, clustered standard errors, and fixed-effect implementations. Reproduction should be accepted only after golden tests or independent implementations satisfy explicit coefficient, standard-error, sample-size, and invariant tolerances.
- Grid breadth versus multiple testing: Wider specification spaces expose analytical sensitivity but also increase computation. If an agent expands the grid after seeing results, researcher degrees of freedom can become an opaque, machine-driven form of p-hacking.
- Throughput versus total cost: Total expenditure can be approximated as
C_total ≈ Σ_i C_rebuild(i) + Σ_i C_agent(i) + Σ_g C_est(g). Parallel containers reduce wall-clock time, but they do not remove the cost of data reconstruction, model calls, or estimator execution.
Evaluation credibility and limits
The paper reports corpus scale, reproduction rates, and several robustness associations, but the abstract provides no non-agent baseline. It is therefore impossible to isolate whether the scale comes from LLM planning, template-based code generation, container orchestration, or human review. The rate of agent intervention, compilation success, data-reconstruction success, and stage-by-stage failure attribution are also missing.
Thirteen natural experiments cannot represent every financial causal-inference setting. The RDD-versus-DiD difference may also reflect sample granularity, scoring rules, treatment timing, and data quality. Similarly, the greater robustness associated with top-journal papers or available author materials may reflect selection effects rather than a direct causal benefit of journal prestige or open science. The 25% average also does not indicate whether multiplicity was corrected inside each grid, so it should not be interpreted as a confirmatory probability after adjustment.
Deployment guidance for developers and research institutions
- Put provenance before autonomy: Preserve hashes of raw files, acquisition dates, cleaning scripts, environment locks, and sample sizes. Agents should access data only through controlled interfaces.
- Use dual-channel verification: Run an agent-generated implementation alongside a fixed template or human implementation. Mark a reproduction successful only when coefficients, standard errors, sample sizes, and key invariants agree within explicit tolerances.
- Pre-register the grid and metric: Fix the defensible specifications, significance thresholds, and aggregation rules in advance, and report sensitivity at p<0.01, p<0.05, and p<0.10.
- Publish per-paper distributions: Report the full distribution of reproduction rates, median effect sizes, specification curves, failure categories, and the number of specifications actually executed for each paper.
- Constrain agent permissions: Default to read-only data, block arbitrary network access, and separate file writes, dependency installation, and estimator execution into auditable tasks. Deterministic validators or human approval should guard high-risk actions.
- Pilot before scaling: Calibrate the agents on a small set of papers with original code, then expand to projects with missing data or historically fragile dependencies. This keeps infrastructure failures separate from disputes over scientific conclusions.
The paper's central contribution is evidence that agentic coding can move multiverse analysis from a handful of hand-built workflows toward tens of thousands of automated specifications. The current evidence supports feasibility more strongly than replacement of researchers: the full method, non-agent baseline, failure decomposition, runtime cost, and reproducibility bundle determine whether this workflow can become routine research infrastructure.
— END —Source: ArXiv AI (cs.AI) (2026-11-20)
Tags: #AI Agents #Multiverse Analysis #Empirical Finance #Reproducible Research #Agentic Coding
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments