Apertus-8B Retains 90.7% of Early News After Four Continual-Learning Nights
By Mr.Xu Community Post
Published:
Summary:Fintan Labs ran a four-day continual-learning experiment on Apertus-8B, updating the model with each day's news and testing all previously learned days the following morning without external prompts. The first day's knowledge remained at 90.7% after four updates, compared with 26.7% for a continually trained LoRA baseline. GSM8K performance was nearly unchanged at 69.2% versus 69.6% for the released model, while the LoRA baseline collapsed to 4.4%. The experiment used predefined pass thresholds
Early News Knowledge Survives Four Updates
Fintan Labs conducted a four-day continual-learning experiment on Swiss AI's Apertus-8B. Each night, the model learned that day's news; the following morning, it answered questions about every earlier day without retrieving the source material from the prompt. The hard part was not the first update but the second and subsequent ones: by the fourth night, the LoRA baseline retained only 26.7% on the first day's questions, while the continual-learning system retained 90.7%.
The reported results were:
| Metric | Continual-learning model | Nightly LoRA baseline | Released Apertus-8B | |---|---:|---:|---:| | Day-1 knowledge after four nights | 90.7% | 26.7% | — | | GSM8K after four nights | 69.2% | 4.4% | 69.6% | | Related knowledge from Days 2 and 3 | About 70% | — | — |
The key signal is not merely that the model learned news. Its general reasoning capability remained nearly intact: GSM8K declined by only 0.4 percentage points relative to the released model. The nightly LoRA baseline, by contrast, lost both old knowledge and general mathematical reasoning.
How the Evaluation Attempts to Control Leakage
The author says that pass thresholds for each day were recorded before the corresponding questions were created. The next morning, another model generated 295 questions, which were then evaluated by a strict scorer. The question-answer data and an interactive benchmark were published for inspection.
This is a useful first step, but it is not the same as a fully independent human evaluation. A question generator may already know the news, follow similar stylistic patterns, or create answers with ambiguous wording. A closed scorer may also reward lexical overlap rather than genuine recall. Stronger replication should use human-authored and human-reviewed questions, vary the generator, prompt format, and scorer, and test whether the same questions remain valid under semantic paraphrasing.
Plausible Mechanisms Behind the Result
The core method has not been released, so its architecture cannot be verified. Plausible mechanisms include parameter isolation, importance constraints, replay, or gradient projection that reduces interference between updates. Each carries different deployment costs:
- Parameter isolation preserves earlier capabilities but increases storage or requires modular inference.
- Gradient projection can reduce overwrite with little inference overhead, although it may require baseline-parameter sensitivity estimates.
- Replay is straightforward but raises licensing, privacy, and long-term corpus-maintenance concerns.
- Weight merging or orthogonal updates can support gradual task expansion, but accumulated drift remains possible.
The LoRA collapse illustrates why continuously stacking low-rank adapters is not sufficient for multi-day updates. The low-rank subspace has limited capacity, and successive writes may overwrite older directions. Without carefully controlled data mixtures and optimization, the model can also drift away from the base model's general capability distribution.
Recurring News Exposes Semantic Interference
Related Day-2 and Day-3 questions fell to roughly 70% when the same stories reappeared in later coverage. This suggests that the model does not simply preserve independent atomic facts inside its parameters. When an event returns with different wording, summaries, or updated details, semantic overwriting and contradiction remain visible.
Unlike conventional training corpora, a news stream is non-stationary: stories evolve, the same fact is rewritten repeatedly, and old claims can become invalid. A production system therefore needs more than a dated list of learned facts. It needs stable event identifiers, versioned claims, source confidence, effective dates, and explicit invalidation rules.
A practical design could keep volatile facts in a versioned retrieval store while using weight updates for durable behavioral patterns. For model-internal knowledge, evaluation should include unchanged facts, revised facts, expired claims, paraphrased questions, and entity combinations absent from training.
Limits of the Current Evidence
This is an author-reported, short-duration experiment whose central method is still private. The training data, compute budget, memory footprint, and reproducibility package cannot yet be audited. Four days, four news sets, and 295 generated questions cannot establish performance over weeks or months of updates, nor generalization to other architectures.
News QA is also vulnerable to lexical shortcuts. A question closely resembling a training article could be solved through surface matching rather than generalizable memory. More credible evidence would include held-out formulations, updated-event questions, corpus-overlap analysis, human audits, and scaling curves for compute, latency, and storage.
Deployment Guidance
For systems that must absorb daily news, the safest architecture today is to separate rapidly changing facts from stable model capabilities. Keep the base model stable, place current information in a versioned external memory or retrieval index, and use controlled weight updates only when durable behavioral adaptation is required.
For weight-based continual learning, start with one model, one stream, and rollback checkpoints. Measure old-knowledge recall, general-capability regression, fact-expiration accuracy, and update latency after every round. Until the method, full data, and cost profile are public, this experiment is best treated as a promising short-term signal rather than a deployment-ready solution.
— END —Source: Reddit r/MachineLearning (2026-11-07)
Tags: #Apertus-8B #Continual Learning #Catastrophic Forgetting #LoRA #News QA
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments