FinDendro Exposes Temporal Knowledge Leakage in LLM Financial Backtests
Summary:FinDendro benchmarks temporal self-knowledge using 2,226 dated U.S. facts spanning 1999–2026, including macroeconomic series, month-end stock prices, and Federal Reserve decisions. By combining value, recall, prediction, and historically dated prompts, it separates factual accuracy from the model's claims about what it knows. The results undermine common leakage-control practices. GPT-5 answered 76% of controlled facts correctly while claiming knowledge of only 38%, and it correctly answered 61
The Core Finding: Correct Answers and Accurate Self-Knowledge Do Not Coincide
FinDendro is designed to answer a fundamental question about LLM financial backtests: when a model predicts a historical outcome, is it genuinely reasoning about the scenario, or has it already encountered the answer during training?\n The benchmark contains 2,226 dated U.S. facts from 1999 through 2026, covering monthly macroeconomic series, month-end stock prices, and Federal Reserve decisions. Each fact receives four prompt types: a direct value question; Recall and Predict questions that pair a candidate answer with a claim of knowing or guessing; and another Predict question with an explicitly declared historical date.
Persistence and dominant-outcome controls reduce chance accuracy to approximately 0.27. This prevents a model from scoring well merely by recognizing common values, persistent trends, or skewed answer distributions. The central target is not factual accuracy alone, but the reliability of the model's claims about the scope of its own knowledge.
GPT-5's Claims and Behavior Diverge Sharply
GPT-5 answered 76% of the controlled facts correctly but claimed knowledge of only 38%. More strikingly, it answered 61% of the facts it explicitly said it did not know.
Two common safeguards consequently fail:
- Asking a model what it knows systematically understates its training-data coverage.
- Splitting evaluation periods using a vendor-stated cutoff does not prove that the test interval is free from leakage.
Knowledge claims can also extend as far as seven quarters beyond the measured boundary. “The model says it does not know” is therefore not evidence against leakage, and “the model says it knows” cannot automatically be interpreted as memorization.
How Four Prompt Families Separate Knowledge from Meta-Knowledge
FinDendro does more than construct a financial timeline. It separates factual knowledge from statements about the model's own knowledge.
Direct value questions establish baseline accuracy. Recall and Predict questions present the same candidate answer while eliciting either a knowledge claim or a guess. The historically dated Predict question tests whether an explicit temporal instruction changes the model's behavior. This permits a comparison among three distinct quantities: whether the model knows the answer, whether it believes it knows the answer, and whether prompting changes its output.
The experiments show that declaring a historical date removes most claims of prior knowledge while leaving accuracy largely unchanged. This suggests that date instructions primarily affect metaknowledge expressions or answer strategy rather than removing information already encoded in model parameters. If a historical answer appeared in training data, instructing a model to behave as if it were in the past still does not create a trustworthy data-isolation boundary.
Why Historical Backtests Are Especially Vulnerable
Financial backtests typically use continuous time series. Macroeconomic indicators, policy decisions, and price movements are highly related. Once an event within a test interval has entered the training corpus, a model may generate abnormal predictive performance through recall even when the interval as a whole sits before a vendor's stated knowledge cutoff.
This contamination is difficult to detect for three reasons:
- The memorized form is often indirect. Training material may contain a value through news reports, databases, or research summaries rather than the canonical benchmark row.
- The boundary is gradual. A cutoff date is a data-governance property, not a sharp behavioral boundary. Different facts are learned at different times.
- Backtest pipelines amplify small leaks. A single unexpected lookup can be amplified through signal generation, parameter optimization, and repeated experiment selection, making leakage resemble genuine alpha.
Splitting the data by event date therefore reduces risk but does not provide audit-grade assurance that a backtest is leakage-free.
A Practical Backtest Auditing Workflow
FinDendro suggests a stronger procedure: estimate the model's factual knowledge boundary through behavioral probes, then select evaluation periods accordingly.
1. Probe Candidate Historical Facts in Both Directions
Test whether the model answers correctly, whether it claims knowledge, and whether an explicit historical date changes its output. The probe is not merely another quiz; it is a search for behaviorally observable memory.
2. Remove Data Near Demonstrated Knowledge Boundaries
If a class of facts exhibits reliable memorization along the timeline, the corresponding interval should not be used directly for predictive-return evaluation. Macro data and policy events require finer segmentation because different sources, revisions, and publication dates can produce distinct boundaries.
3. Separate Probes, Backtests, and Reporting Data
The main backtest set, behavioral probes used to estimate boundaries, and final reporting set should be managed independently. If boundary estimation shares the same examples used for strategy tuning, selection bias remains.
4. Report Leakage Evidence, Not Merely Cutoff Dates
A reproducible report should include the vendor cutoff, per-fact behavioral probes, uncertainty intervals, excluded ranges, model version, prompt templates, sampling settings, and answer-parsing rules. Better historical-date prompting does not prove that the model lacked prior exposure.
5. Audit Ingestion Time, Not Just Event Time
A macroeconomic value may be republished in news, databases, and analyst reports after its initial release. Whether it entered a model's training corpus depends on collection and training dates, not the date on which the economic event occurred. Event-time splits do not replace ingestion-time audits.
Evaluation Limits and Appropriate Use
FinDendro focuses on U.S. financial facts, primarily monthly data and discrete decisions. It does not cover every asset class, forecasting target, or real trading decision. Behavioral knowledge is not identical to trading skill, and the benchmark cannot detect every form of indirect contamination introduced through multi-step reasoning or tool retrieval.
Probes can also be affected by prompt sensitivity, output-parser errors, and sampling variance. A robust implementation should use repeated samples, randomized answer ordering, multiple prompt templates, confidence intervals, and manual review of samples classified as leaked.
Conclusion
An LLM can correctly answer questions it says it does not know and claim knowledge of facts beyond its measured temporal boundary. For financial backtesting, vendor cutoffs, self-reporting, and historical-date instructions are all weak safeguards.
The stronger approach is to measure temporal knowledge behaviorally, exclude facts the model demonstrably knows, and make leakage auditing part of the backtest protocol rather than an appendix. FinDendro turns that previously implicit assumption into a reproducible benchmark.
— END —Source: Hugging Face Daily Papers (2026-11-04)
Tags: #FinDendro #Financial Backtesting #Temporal Knowledge Boundary #Data Leakage #Model Evaluation
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments