Testing LLMs on a 102-Paper Scientific Review Benchmark
Summary:This study evaluates six AI tools across an end-to-end scientific literature-review workflow: article discovery, interpretation, cross-paper synthesis, and research-question proposal. The benchmark uses a published review of the Spanish plume weather pattern and its corpus of 102 articles. Institutional library access materially improved recall, raising Codex from 42 to 82 retrieved articles and Claude Code from 15 to 79. Yet, when given the complete corpus, three tools found all 18 articles th
Four stages of an end-to-end review workflow
Rather than testing isolated literature-related questions, this study uses a published review of the Spanish plume weather pattern and its 102-paper corpus to evaluate six AI tools across four stages: discovering relevant articles, interpreting them, synthesizing across them, and proposing future research.
This separation matters because search recall, evidence classification, synthesis quality, and research ideation fail in different ways. A system may retrieve the right papers while misclassifying many irrelevant mentions, or produce a broadly accurate summary while overstating consensus.
Access to institutional libraries materially changes recall
During initial searches, the tools collectively identified 57 articles, with the best individual tool finding 42. With institutional library access, Codex increased from 42 to 82 retrieved articles, while Claude Code rose from 15 to 79.
The key engineering lesson is that access, index coverage, and metadata quality are data-plane constraints, not problems solved merely by selecting a stronger model. LLMs can reformulate queries and explore candidate results, but they cannot overcome subscription barriers, catalog differences, or missing database indexes.
An AI literature-review system should therefore expose and distinguish:
- Open-web search for discovering accessible material;
- Institutional databases and library APIs for licensed literature;
- Local PDF analysis for legally obtained full texts.
If the interface only provides a search box while hiding connectors, authorization scope, coverage dates, and deduplication rules, apparent disciplinary coverage often reflects query generation rather than reliable evidence recall.
Finding core papers is not equivalent to assessing evidence relevance
When given all 102 PDFs, three tools correctly identified the 18 articles that substantially discussed the Spanish plume. However, those tools classified 40% to 79% of 84 passing mentions—references that mentioned the phenomenon without substantively examining it—as substantial discussions.
This exposes a gap between full-text retrieval and evidence grading. Keyword occurrence answers whether a target concept appears, not whether the paper actually investigates the mechanism. Directly promoting retrieval relevance into evidence importance produces systematic false positives.
A robust pipeline should distinguish at least four states:
retrieved: returned by search or retrieval;topically_relevant: related to the topic;substantively_related: provides substantive evidence;methodologically_relevant: supports the inference under evaluation.
The final two labels should not be assigned from semantic similarity alone. Section location, study design, research-question alignment, citation direction, and expert rules can be combined with the model, with confidence intervals or multi-level labels replacing a binary decision.
Accurate synthesis is not necessarily insightful synthesis
Two domain experts blindly evaluated the syntheses produced by the same three tools. Most content was accurate, but the outputs offered limited new insight and exhibited unsupported claims, quotation errors, and exaggerated statements of consensus.
This is more than generic hallucination. LLMs favor coherent narratives, and coherence can conceal missing evidence. Once a subset of papers enters the context window, a model may construct a plausible causal account around it and fill gaps with unsupported assertions.
A more reliable synthesis stage begins with a structured evidence table rather than direct prose generation. For each paper, it should capture the research question, sample and method, principal findings, limitations, direction of evidence, and source pages. Generated claims should then be linked back to that table through deterministic validation. Unsupported claims should be removed, downgraded to hypotheses, or explicitly marked as unverified.
Research proposals may drift beyond the evidence boundary
All three tools proposed research directions broader than the Spanish plume literature, although two moved toward questions already identified in the benchmark review. Broadening is not inherently wrong: researchers may legitimately ask questions beyond an existing review. The problem arises when the system does not distinguish gaps directly connected to the evidence from generic extrapolation.
A safer procedure first extracts contradictions, sampling gaps, measurement limitations, and unresolved mechanisms from the evidence table. The model may then expand questions only from those gaps. Suggestions outside the corpus boundary should be separated and labeled as speculative extensions rather than conclusions supported by the literature review.
No single tool wins every stage
Perhaps the most important result is that no tool performed best across all tasks. Search depends on connectors and corpus access; evidence interpretation depends on full-text reasoning and structured outputs; reliable synthesis depends on claim-level provenance and deterministic checks; research planning still depends on domain judgment.
Selecting the best LLM is therefore not a complete workflow design. A more practical architecture assigns each component a bounded responsibility and introduces human approval at high-cost decision points.
Practical recommendations for research teams
First, maintain an auditable retrieval log containing data sources, queries, timestamps, access type, duplicates, and exclusion reasons.
Second, never replace source verification with generated summaries. Important claims should be checked against each paper's research question, methods, results, and limitations.
Third, separate topical relevance from evidential importance and evaluate recall and false-positive rates independently.
Fourth, validate references programmatically by checking metadata, citation locations, and quoted text.
Fifth, ask models to label evidence strength, disagreements, and unknowns explicitly rather than presenting uncertainty through vague wording.
Sixth, retain expert authority over research questions, inclusion criteria, and final conclusions. LLMs are best used to expand search coverage, screen candidates, organize material, and draft first-pass summaries—not to replace scholarly judgment.
The practical value of these systems is not one-click review generation. It is the ability to move researchers away from manually scaling search and initial organization while making the remaining evidence-grade decisions—classification, conflict resolution, and boundary control—explicitly human-owned.
— END —Source: ArXiv AI (cs.AI) (2026-11-05)
Tags: #LLM Literature Review #102-Paper Benchmark #Academic Search #Evidence Synthesis #Human-AI Collaboration
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments