ZICQ
中 Log in / Sign up
ZICQ Info Research & Papers #Paper Ablations #Meta-Science #LoRA #Experimental Design #Negative Results

SCIME Atlas Audits 126,758 Papers: 31% of ML-Venue Changes Report Null or Negative Effects

Avatar of Mr.Xu

By Mr.Xu Community Post

Published: · 1 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:Latentic Labs' SCIME Atlas mines full text from 126,758 papers published across ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV, ECCV, TPAMI, IJCV, and JMLR between 2016 and 2026. It records 621,064 added, measured changes from 110,393 papers and classifies each by the outcome reported by the authors: 69% helped, 10% had no clear effect, 16% hurt, and 5% were mixed. The project reports that 66% of papers contain at least one unhelpful change, while null or negative outcomes account for 31% of changes at M


Core Findings

SCIME Atlas converts 126,758 papers published from 2016 through 2026 across ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV, ECCV, TPAMI, IJCV, and JMLR into an evidence table linking added changes to the outcomes reported by their authors. The 621,064 records are not the result of one team rerunning every experiment under a unified protocol; they inherit each paper's metrics, baselines, compute budget, and statistical criteria.

| Metric | Reported result | Correct interpretation | |---|---:|---| | Corpus | 126,758 papers | Ten ML and computer-vision venues and journals spanning 2016—2026 | | Extracted changes | 621,064 | Drawn from 110,393 papers and covering added layers, losses, data, learning rates, and other interventions | | Reported direction | 69% helped, 10% unclear, 16% hurt, 5% mixed | Based on the original papers' own metrics rather than a standardized rerun | | Paper-level result | 66% contain at least one unhelpful change | Expected when one paper evaluates many separate interventions | | Venue comparison | 31% null or negative at ML venues versus 22% at vision venues | Descriptive only; not evidence that one research community is inherently more reliable | | Frequently unhelpful | Higher sampling temperature 52%, larger LoRA rank 46%, larger learning rate 46% | The temperature result covers 122 papers; denominators for the other two are absent from the collected text | | Least often unhelpful | Contrastive loss 4% | Denominator is 1,317 papers, but applicability remains task-dependent |

Two useful quantities can be derived directly from the disclosed numbers. Papers with extractable changes account for approximately 87.1% of the corpus, leaving about 12.9% unexplained. More importantly, each contributing paper supplies roughly 5.6 extracted changes. The 621,064 rows are therefore strongly clustered by paper, team, year, and task and cannot be treated as 621,064 independent statistical samples.

How the Atlas Is Constructed

At least four processing layers can be inferred from the project description, although the implementation is not disclosed:

  1. Corpus assembly: collect full text from the selected venues, including appendices and potentially experimental tables.
  2. Change unit extraction: identify components or training settings added relative to a baseline and actually measured by the authors, such as layers, losses, additional data, or larger LoRA ranks.
  3. Outcome normalization: map reported results into helped, no clear effect, worse, or mixed categories.
  4. Cross-paper aggregation: aggregate by change type, paper, and venue to produce the public table of commonly unhelpful interventions.

The critical bottleneck is not the final aggregation but extraction and label assignment. The collected information does not specify the parser or classification model, whether humans audited the output, how appendix tables were handled, how preprint and camera-ready duplicates were reconciled, or how inconsistent terminology was normalized. It also provides no extraction precision, recall, or inter-annotator agreement. Latentic Labs built the project and disclosed that relationship; that does not invalidate the dataset, but it increases the need for transparent methodology and independent validation.

Engineering Mechanisms Behind the Results

  • Higher sampling temperature: Among 122 papers reporting this change, 52% were classified as unhelpful. Temperature changes the sampling distribution through p_i(T) = exp(z_i / T) / Σ_j exp(z_j / T). Increasing T flattens probabilities, raises entropy, and makes long-tail samples more likely. During inference, this can improve diversity while reducing factual precision, instruction adherence, or reproducibility. During training-time sampling, distillation, or data augmentation, the consequences differ. Because the collected summary does not identify the exact setting, the result cannot be generalized to every generative workload.

  • Larger LoRA rank: Forty-six percent of the associated papers reportedly obtained no benefit. For a d_out × d_in weight matrix, a LoRA adapter contains approximately r(d_in + d_out) trainable parameters. Increasing r therefore expands adapter parameters, gradients, and optimizer-state memory roughly linearly. Doubling rank is not automatically better: it can add unnecessary overfitting freedom on small datasets, while multi-task adaptation may genuinely require more capacity. Many implementations also scale updates using alpha/r, so changing rank alters both capacity and effective update magnitude. Merging the adapter into the base weights avoids meaningful inference overhead; retaining a separate adapter branch adds low-rank matrix multiplications and memory-bandwidth pressure.

  • Larger learning rate: Forty-six percent of the associated papers reportedly did not benefit. A larger learning rate adds no parameters, but it can amplify gradient noise, seed sensitivity, and divergence, particularly with small batches or LoRA fine-tuning. Warmup, gradient clipping, and a stable baseline learning-rate search remain necessary. Conversely, a larger rate can help underfitting or plateau regimes, so the finding is conditional rather than directional.

  • Contrastive loss: Only 4% of 1,317 papers were classified as unhelpful, suggesting constructive results where suitable semantic relationships exist. Contrastive training requires reliable positive and negative pairs, suitable augmentations, an appropriate temperature or margin, and controls for false negatives. It also adds representation computation, negative sampling, and training-memory costs. The low failure rate does not imply universal benefit because many tasks lack valid pair construction.

Why 31% and 66% Are Not Direct Failure Rates

  1. Multiple trials naturally increase the chance of at least one failure. With about 5.6 extracted changes per contributing paper, the 66% paper-level statistic does not mean that 66% of research failed or that the main conclusions were invalid.
  2. Direction labels do not use standardized effect sizes. One paper may call a 0.1-point gain an improvement, while another labels a genuine gain as mixed because of variance. Accuracy, FID, perplexity, and human ratings are not directly comparable.
  3. Outcomes are self-reported. Unified reruns could control data splits, seeds, training budgets, and statistical tests, but would require substantial reproduction resources. The atlas preserves ecological realism at the cost of causal comparability.
  4. Selective reporting can suppress negative outcomes. Experiments that failed are less likely to appear in the main text, and extraction cannot recover experiments that were never reported.
  5. Venue comparisons are confounded. ML and vision papers differ in task mix, baseline tuning, statistical practice, and year composition. Without denominators, effect sizes, and confidence intervals, 31% versus 22% describes the sample rather than explaining the difference.
  6. Extraction error is unquantified. Full-text coverage does not guarantee accurate item-level extraction, particularly for tables, footnotes, and evidence distributed across sections.

A Practical Playbook for Developers

  • Treat the atlas as a prior, not a configuration answer. Larger rank, learning rate, or temperature are unhelpful only under particular baselines, tasks, and compute budgets. Use the data to narrow search spaces and flag risky changes, not to bypass local validation.
  • Tune the baseline before adding a component. When comparing a layer, loss, or adapter capacity, hold the data, initialization, training steps, and inference configuration fixed. Otherwise the result cannot distinguish the intervention from accidental baseline gains.
  • Run small, staged ablations. For LoRA, inspect rank, alpha, and target-module choices separately before a limited joint search. Sweep learning rate around a stable region, and evaluate sampling temperature independently of training hyperparameters.
  • Use task-specific metrics for directional parameters. Temperature experiments should measure both task success and diversity, repetition, or factuality. LoRA studies should track convergence, final quality, training memory, and adapter storage. Learning-rate studies need multi-seed means, variance, and failure rates.
  • Keep a negative-result registry. Record every attempted intervention and why it was rejected. Adding dataset size, model size, compute budget, seed count, and effect size would enable stronger future meta-analysis.
  • Use clustered statistics. If record-level data become available, treat paper as the clustering unit and use hierarchical models or cluster-robust standard errors across venue, year, task, and model family. Before adopting the atlas internally, manually audit a random sample for change-identification accuracy, outcome-label confusion, and duplicate-paper handling.

Bottom Line

The main contribution of SCIME Atlas is not a universal proof that a hyperparameter or loss function fails. It converts negative results dispersed across papers, appendices, and tables into a searchable engineering prior. Its 621,064 changes and 126,758-paper corpus provide an unusually broad view, but author-reported outcomes, nested observations, and undisclosed extraction error make the atlas an experiment-planning aid rather than a finished recommendation engine. Extraction methods, confidence intervals, human audits, and independent replication are required before it can become a dependable automated ablation layer.


Source: Reddit r/MachineLearning (2026-11-17)

— END —

Tags: #Paper Ablations #Meta-Science #LoRA #Experimental Design #Negative Results

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…