PRISM Exposes Cross-Topic Name Collisions in Scientific Retrieval
Summary:PRISM analyzes 564,861 AI-related arXiv records and finds that shared scientific names increasingly collide across research topics. Cross-topic collision volume grows 4.33-fold relative to a pooled 2020–2022 baseline, compared with only 1.12-fold within topics. Even after research context is supplied, title-based BM25 retains a 25.7-point matched-identification deficit, while a title-plus-abstract E5 model still shows a 6.9-point deficit. Listwise language-model ranking reaches 89.2% when candid
Core finding: scientific names increasingly create retrieval ambiguity
PRISM studies a failure mode that directly affects scientific search and research assistants: the same scientific names are increasingly reused across unrelated topics. A name-only query can therefore return multiple plausible papers, and adding topic context or document fields may not fully resolve the target's identity.
The study covers 564,861 AI-related arXiv records, using a pooled 2020–2022 set as the reference for comparison with 2026. Cross-topic name collisions grow 4.33-fold relative to that baseline, while collisions within a topic grow only 1.12-fold. The primary expansion therefore comes from shared nomenclature across research communities rather than simple term proliferation inside a single discipline.
A second sentence encoder reproduces the trend, and name-group resampling quantifies uncertainty. Fixed word-frequency bands locate much of the increase in intermediate-frequency vocabulary: terms that are neither highly specialized nor dominant enough to serve as stable discriminators, making them especially reusable across fields.
Separating the stages of retrieval failure
Rather than attributing fuzzy retrieval to a generic weakness of dense models, the paper separates three stages: candidate competition, document-field usage, and contextual ranking. These correspond directly to engineering layers. Recall determines which same-name papers enter the candidate pool; document construction determines what evidence the encoder sees; reranking uses topic context to resolve candidates.
Under the same three-term query construction, shared-name targets retain matched-identification deficits of 25.7 percentage points with title-based BM25 and 6.9 points with a title-plus-abstract E5 model. Adding abstract evidence therefore helps materially but does not eliminate ambiguity. As the candidate pool grows, early recall errors also increase downstream reranking cost and the risk of a research assistant citing the wrong paper.
When candidates are fixed to the correct same-name set, listwise language-model ranking reaches 89.2% using only the query and candidate titles. This points to a substantial recall-stage bottleneck: a reranker cannot recover a target that never enters its candidate window. Engineering evaluations should report candidate recall and within-candidate identification separately rather than relying on a single end-to-end Top-k number.
Why BM25 and dense retrieval still fail
Title-based BM25 relies on lexical matching and inverse document frequency. When a name occurs across many disciplines, its term-level discriminative power declines. Paper titles are short and often lack stable clues such as method, dataset, author, and venue, leaving same-name documents nearly tied during lexical retrieval.
E5-style dense encoders map queries and title-plus-abstract text into a shared semantic space and can exploit contextual clues better than BM25. Their ceiling nevertheless depends on candidate recall, field quality, and training-distribution coverage. If a shared name is treated as a strong topical signal, or generic abstract prose overwhelms useful evidence, semantically distinct papers may still collapse into one neighborhood.
Listwise language-model reranking has a different advantage: it compares candidates jointly rather than scoring each item independently. It can use the query, differences among candidate titles, and their relative ordering. However, the reported 89.2% comes from a fixed same-name candidate set. It does not represent end-to-end performance on open corpora, where systems must also handle candidate expansion, index noise, and prompt-length constraints.
Engineering trade-offs
The findings suggest a layered scientific-retrieval pipeline: use authors, topics, abstracts, and citation neighborhoods to form a broad high-quality candidate set, then apply listwise reranking for fine-grained disambiguation. This is not free. More candidates increase the compute cost of query-document cross-encoding or listwise inference. Moving full abstracts into the context also raises KV Cache memory usage and inference latency as token counts expand.
A practical deployment can filter high-confidence fields first, then add titles, abstracts, and citation context during reranking. Compared with sending every same-name paper to a generative model, this design is easier to audit and less dependent on long-context inference.
Fair comparisons must also control the available fields. Comparing title-only BM25 with title-plus-abstract E5 measures both model differences and information differences. PRISM improves this analysis through fixed queries and explicit ablation of candidate competition and field contribution. External validation should still cover different disciplines, languages, preprint versions, and database coverage.
Implications for research agents
When a research agent passes a scientific name as a tool parameter, it risks treating a merely related paper as the intended target. A safer API should return structured candidates and uncertainty instead of a single title. Results can include collision count, author evidence, date range, topic labels, version information, and source links for an explicit disambiguation stage.
The results also caution against asking a language model to reconstruct missing identities from memory. Fluent generation can combine conflicting candidates into a nonexistent citation. The reliable sequence is to retain multiple candidates during recall, disambiguate using verifiable metadata, and bind the final answer to a paper identifier or canonical source.
Deployment recommendations
- Report layered metrics: Measure same-name candidate recall, within-candidate identification, and end-to-end citation correctness separately.
- Control fields: Compare title, title-plus-abstract, and extended metadata under identical queries and candidate sets.
- Preserve disambiguation metadata: Keep names, authors, years, topics, and versions as structured constraints for reranking.
- Budget reranking explicitly: Cap candidates for high-collision names and select abstract spans according to query-document cost.
- Verify citations: Check identifiers, versions, and source links before an agent emits a reference; never cite from a generated title alone.
PRISM's main contribution is not a new leaderboard winner. It turns scientific nomenclature into a measurable ambiguity object, separating collision scale, candidate recall, field contribution, and contextual ranking. For teams building scientific search and research-agent systems, that diagnostic framework is more actionable than adding another generic embedding model.
— END —Source: Hugging Face Daily Papers (2026-11-01)
Tags: #Scientific Retrieval #Name Disambiguation #BM25 #Dense Retrieval #Research Agents
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments