Frozen Forecasting Models Underperform Strong Time-Series Classifiers
Summary:This controlled benchmark evaluates Reverso and Chronos-Bolt as frozen feature extractors for time-series classification across six MONSTER datasets. Although pretraining improves accuracy over random-weight versions on five of six datasets, with gains reaching 23 percentage points, the frozen representations never outperform the strongest non-pretrained baseline: they lose in 15 of 17 settings and tie in the other two. Restoring amplitude information recovers part of the gap, while fine-tuning
Core Finding
The pretrained weights in Reverso and Chronos-Bolt do encode useful information, but that information is not enough to make frozen feature extraction consistently beat conventional time-series classifiers. Across six MONSTER datasets and three label regimes—1%, 10%, and 100%—the frozen representations recorded zero wins against the strongest non-pretrained baseline, 15 losses, and two ties.
This does not mean pretraining is useless. Against random-weight versions of the same forecasting architectures, pretraining improved accuracy on five of the six datasets, with gains reaching 23 percentage points. The challenged assumption is narrower but important: pretrained foundation-model representations are not automatically superior to task-specific models.
How the Benchmark Isolates Pretraining
The study evaluates Reverso and Chronos-Bolt as frozen feature extractors on MONSTER. It compares them with four established time-series classification methods and with random-weight versions of the same forecasting models.
The random-weight control is essential. A comparison only between a pretrained model and a conventional classifier can conflate learned representations with the expressive capacity of the architecture, feature dimensionality, or classifier head. The random-weight baseline separates these effects:
- Pretrained versus random weights tests whether forecast training learned transferable structure;
- Pretrained versus strong task-specific baselines tests whether that structure adds practical value for classification.
The experiments cover six large datasets and label subsets at 1%, 10%, and 100% of the available training data. Low-label regimes did not consistently reverse the result. Pretraining may help in some configurations, but it did not produce a robust advantage over the strongest non-pretrained method.
Why Pretraining Helps Yet Still Loses
Forecasting pretraining and classification optimize different objectives. A forecasting model learns temporal evolution, dependencies, and predictable patterns, but a classifier needs discriminative class boundaries. A frozen representation can contain forecast-relevant information without being arranged into a class-separable space.
Strong task-specific baselines directly optimize a classification objective such as empirical risk minimization, margin maximization, or a statistical representation designed for temporal structure. In other words, a pretrained model may learn useful structure in a broader prediction space that a small classifier head cannot fully exploit when labels are scarce or the downstream task differs from pretraining.
Random-weight results add another nuance. Even without learned temporal structure, projections from the same architecture can provide nonlinear expressive capacity. The pretrained versions improve on these controls by as much as 23 percentage points, demonstrating that training contributes something real. The issue is that this contribution is not large enough to defeat a model already optimized for classification.
Amplitude Recovery and Adaptation
The authors restore amplitude information to test whether normalization or representation extraction discards scale information required for classification. Recovering amplitude closes part of the gap, indicating that preprocessing and representation design matter. The repair is partial rather than a systematic reversal of the ranking.
Fine-tuning and forecast-error features also fail to close the gap. Fine-tuning is constrained by overfitting and optimization cost in low-label settings. Forecast-error features connect forecasting to classification, but they still depend on a correlation between prediction residuals and class membership that is not guaranteed across datasets.
The practical implication is straightforward: connecting a foundation model and fine-tuning its head should not be the default replacement for a strong classical pipeline. Random-weight controls, strong baselines, amplitude recovery, and adaptation strategies should all be evaluated before deployment.
Engineering Trade-offs
Frozen extraction has a real advantage: the forecasting model does not need to be updated, so training can be limited to a lightweight classifier head. This is attractive when labels are scarce, compute is constrained, or the foundation model is already deployed through an API or checkpoint service.
The total inference path may nevertheless be more expensive than direct classification. If every example must pass through a large forecasting model to obtain hidden states, latency, memory use, and batch-processing complexity can outweigh the benefit of a small trainable head.
Representation dimension, sequence length, and classifier capacity must also be selected together. Higher-dimensional representations may preserve more information while increasing classifier parameters and overfitting risk. Longer contexts add prefill work and KV Cache costs.
Finally, the result covers two forecasting models, six MONSTER datasets, and the tested label regimes. It should not be generalized into a universal claim that pretrained representations are unsuitable for time-series classification. Data-generating processes, class predictability, and the relationship between forecasting and classification objectives can all change the outcome.
A defensible deployment workflow is:
- Establish strong task-specific baselines, including linear models, statistical-feature methods, and established time-series classifiers.
- Create random-weight controls with the same architecture to test whether pretraining contributes beyond model capacity.
- Evaluate 1%, 10%, and 100% label regimes instead of reporting only the easiest full-data setting.
- Test normalized representations, amplitude recovery, fine-tuned heads, and forecast-error features separately and in combination.
- Include foundation-model forward passes in the comparison and calculate the accuracy benefit against end-to-end latency and training cost.
Limits of the Evidence
The experimental design is stronger than a conventional leaderboard, particularly because of the random-weight controls and multiple label regimes. However, the public abstract does not specify all four non-pretrained baselines, the classifier-head architectures, dataset domain coverage, confidence intervals, or stability across random seeds.
The result is therefore strong evidence that frozen representations should not be adopted without rigorous comparison, but not proof that forecasting foundation models have no value for classification. The defensible engineering position is narrower: pretraining is a hypothesis to validate, not a default upgrade path.
— END —Source: ArXiv Machine Learning (cs.LG) (2026-11-06)
Tags: #Time-Series Classification #Forecasting Foundation Models #Frozen Representations #Chronos-Bolt #Random-Weight Control
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments