Research Reveals Catastrophic Forgetting in Scientific Foundation Models During Sim-to-Real Adaptation
Summary:A new arXiv study delves into the catastrophic forgetting problem faced by Scientific Foundation Models (SciFMs) during sim-to-real adaptation. Using the Walrus-1.3B model as a case study, researchers found that finetuning on simulated data rapidly degrades the model's performance on original pretraining datasets, with one-step errors increasing up to 35 times after just 20 epochs. Further adaptation to real data exacerbates the issue. The study proposes that replaying as little as 1% of the sim
Background and Motivation
Scientific Foundation Models (SciFMs) have demonstrated strong capabilities in handling complex scientific problems such as Partial Differential Equations (PDEs). However, like all neural network architectures, SciFMs face the risk of catastrophic forgetting when adapting to new tasks, where adapting to new tasks degrades the model's memory of prior knowledge. This study examines the catastrophic forgetting problem in the context of sim-to-real adaptation using the Walrus-1.3B model as a case study.
Methodology and Findings
The study employs a two-stage migration approach: first finetuning on simulated flow data around a cylinder, and then adapting to experimental data of the same system. The results show that finetuning on simulated data rapidly degrades the model's performance on the original pretraining datasets, with one-step errors increasing by 35 times after just 20 epochs. Further adaptation to real data exacerbates the issue, further eroding the model's performance on the simulated task.
The researchers analyzed the potential drivers of forgetting, including dataset dependence and task representation similarity. They proposed a lightweight replay strategy: replaying a small portion (1%) of the simulated data during real-data finetuning. This strategy effectively recovers most of the lost simulation performance while maintaining effective adaptation to the experimental dynamics.
Engineering Trade-offs and Practical Applications
- Efficiency of Replay Strategy: Replaying a small amount of data is a computationally inexpensive and effective solution, avoiding the high overhead of full retraining.
- Applicable Scenarios: This method is particularly suitable for resource-constrained environments where knowledge retention is critical, such as scientific computing and engineering simulation.
- Limitations: The replay strategy requires storing some simulation data, which demands additional storage space. Moreover, determining the optimal replay data proportion requires further research.
Developer Recommendations
- Data Management: When designing SciFM applications, it is advisable to retain some key simulation data for subsequent finetuning.
- Model Evaluation: Regularly evaluate the model's performance on the original task during transfer learning to monitor the degree of forgetting.
- Optimization Strategies: Combine the replay strategy with other regularization methods (such as L2 regularization, Dropout) to further enhance knowledge retention.
— END —Source: ArXiv Machine Learning (cs.LG) (2026-11-02)
Tags: #Catastrophic Forgetting #Scientific Foundation Models #Sim-to-Real Adaptation #Finetuning #Replay Strategy
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments