fNIRSAtlas Standardizes fNIRS Classification and Exposes Deep-Learning Generalization Gaps
Summary:fNIRSAtlas consolidates 19 public functional near-infrared spectroscopy datasets into 23 standardized classification tasks spanning motor execution, motor imagery, mental arithmetic, working memory, and emotion recognition. It provides an automated pipeline covering raw-signal preprocessing, model training, and evaluation, with explicit protocols for both cross-subject and within-subject classification. The benchmark's baseline experiments show that performance depends strongly on task and data
Nineteen Public Datasets and Twenty-Three Classification Tasks
fNIRSAtlas reorganizes 19 public fNIRS datasets into 23 classification tasks covering motor execution, motor imagery, mental arithmetic, working memory, and emotion recognition. It defines both cross-subject and within-subject protocols. Cross-subject evaluation tests transfer to previously unseen participants, while within-subject evaluation measures generalization across sessions or experimental runs for the same participant.
This distinction is critical for fNIRS. The signals combine hemodynamic responses with physiological noise, motion artifacts, and instrument-related variation. Differences in experimental paradigms, subjects, and acquisition sessions can dominate the apparent classification performance. A randomly split window-level benchmark may therefore reward leakage of participant, session, or preprocessing fingerprints rather than transferable cognitive-state representations.
An Automated Pipeline for Reproducibility
The project provides an automated path from raw data through preprocessing, model training, validation, and evaluation. Its value is not merely convenience: it turns filtering decisions, channel handling, normalization, and train-test construction into executable and repeatable procedures.
Automation does not automatically remove bias. Motion correction, bad-channel rejection, short-separation-channel handling, baseline correction, and physiological-noise regression can materially change results. A single fixed preprocessing policy may improve comparability while embedding a particular modeling assumption. Configurable components, versioned artifacts, and ablation studies remain necessary.
Deep Learning Does Not Generalize Reliably Across Tasks
The benchmark evaluates baselines drawn from fNIRS, neuroimaging, and time-series machine learning. Its central result is not the superiority of one architecture, but strong dependence on task and dataset scale. Methods validated on limited data often fail outside their original evaluation setting, with deep-learning approaches serving as a prominent example.
The cause is closely related to the structure of fNIRS recordings. Although a recording may contain thousands of temporal windows, the number of genuinely independent experimental observations is usually much smaller. A high-capacity network can memorize participant or session signatures unless the split operates at the subject or run level. Strong regularization, early stopping, domain adaptation, and subject-wise cross-validation are therefore substantive evaluation controls rather than optional refinements.
Cross-subject evaluation introduces additional distribution shifts. Individual differences in anatomy, physiology, hair condition, electrode placement, and instrument settings can all change the observed signal. A single aggregate accuracy conceals these effects. More informative reporting includes subject-wise variance, confidence intervals, macro-F1, balanced accuracy, and worst-task performance.
Engineering Trade-offs: Standardization Versus Fixed Assumptions
A common benchmark saves researchers from repeatedly implementing data acquisition and evaluation logic. It also makes comparisons more transparent and supports cumulative evidence about methods that fail under broader conditions.
The cost is increased computational and maintenance burden. Multiple public datasets introduce dependency, caching, licensing, and version-control requirements. Data corrections or withdrawals can invalidate earlier results. If every method receives a different tuning budget, or if test sets influence preprocessing choices, nominal comparability will erode.
Confidence in the benchmark ultimately depends on isolation of test data, subject-level statistical units, documented licensing, exact configuration capture, and independent verification. New datasets should include provenance, ethics information, content hashes, and explicit versioned preprocessing configurations.
Practical Adoption
A useful first step is to reproduce three settings: cross-subject classification, a strict within-subject split, and the commonly used random-window split used in prior work. The gap among them can reveal whether a method generalizes across people or depends on session-level leakage.
Keep tuning and compute budgets comparable across methods. Report class-sensitive metrics in addition to accuracy, and include parameter count, training time, and pretraining requirements. For real-world brain-computer interfaces, evaluate cross-day and cross-device transfer, signal-quality rejection, motion robustness, calibration drift, and confidence calibration. A benchmark score is an entry criterion for further testing, not evidence of deployment readiness.
A Shared and Falsifiable Experimental Foundation
fNIRSAtlas contributes infrastructure rather than a new state-of-the-art model. Its 19 datasets and 23 tasks widen the scope of generalization tests, while the automated pipeline lowers the cost of reproduction. The most consequential result is negative: deep models that appear strong on limited evaluations do not consistently retain that advantage across tasks and datasets.
For fNIRS and brain-computer-interface developers, the benchmark is best treated as a model-screening and regression-testing layer. Its practical value will grow as new devices, populations, and online recordings are added, and as failed runs, compute costs, and statistical uncertainty remain openly documented.
— END —Source: ArXiv Machine Learning (cs.LG) (2026-11-10)
Tags: #fNIRS #Brain-Computer Interface #Classification Benchmark #Neuroimaging #Generalization Evaluation
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments