ZICQ
中 Log in / Sign up
Newsroom Research & Papers #Tabular Classification #Data Leakage #ROC-AUC #Controlled Benchmark #Statistical Audit

Not All Data Leakage Is Equal: Target Encoding Inflates Tabular ROC-AUC by 2.42 Points on Average

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 2 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:This controlled study separates five leakage mechanisms by their effect on held-out ROC-AUC in tabular classification. It covers six public datasets, two synthetic stress tests, three model families, and five repeated stratified splits, producing 600 paired clean-versus-leaked comparisons. Dataset-level cluster bootstrap confidence intervals and Holm-corrected one-sided Wilcoxon tests reduce pseudo-replication and multiplicity concerns. Global target encoding, supervised feature selection befor


Core Finding

Rather than introducing a new classifier, this paper turns a vague concern about touching the test set into a measurable evaluation bias. For each split, the authors run a clean pipeline and a pipeline contaminated by exactly one leakage mechanism, then measure ΔROC-AUC = leaked ROC-AUC - clean ROC-AUC. A larger positive value means that offline model selection is more likely to overstate real generalization.

The results form a clear severity hierarchy:

  • Global target encoding raised ROC-AUC by 2.42 percentage points on average, with a 95% cluster-bootstrap confidence interval of 0.62–4.89.
  • Global supervised feature selection before splitting produced a 1.93-point increase, with an interval of 0.63–3.39.
  • Duplicate contamination produced a 1.52-point increase, with an interval of 0.40–2.80.
  • All three effects remained significant after multiplicity correction. Global scaling and global median imputation had near-zero mean effects in the studied IID settings.
  • A synthetic stress test showed that target-encoding leakage becomes substantially worse as categories become sparse.

Label-dependent transformations and direct observation reuse therefore deserve the highest audit priority. Feature-only dependence still requires scenario-specific analysis rather than a universal exemption.

How the Five Leakage Mechanisms Reach the Test Set

Global Scaling and Global Median Imputation

Global standardization estimates the mean and standard deviation from all rows before splitting, typically as z = (x - μ_all) / σ_all. Global median imputation similarly uses test-feature distributions to determine replacement values. Neither mechanism reads labels directly, so under IID sampling and reasonably sized datasets, their expected influence on a training-only transformation is small. That is consistent with the near-zero average effects observed in the paper.

They nevertheless create held-out-data dependence. The effect can change with tiny test sets, temporal drift, strong group structure, or curated test distributions. It is especially ambiguous when a production system explicitly performs online or batch distribution adaptation: allowing test statistics may change the estimand being evaluated. The near-zero IID result therefore does not establish universal harmlessness.

Supervised Feature Selection Before Splitting

Supervised feature selection uses labels to rank or filter candidate variables. If this operation runs before the train/test split, test labels directly determine which features the final model receives. The leakage path is explicit. With few samples or many candidate features, label-driven selection variance can be amplified by the fixed test result.

Feature selection must therefore remain inside the training fold. If feature-selection thresholds and model hyperparameters are tuned together, an inner cross-validation loop is required rather than repeated reuse of the final test set.

Duplicate Contamination

When a duplicated record appears in both training and test data, a model can memorize the training instance and its label instead of learning a generalizable boundary. This is direct data reuse. The problem is not limited to large duplicate ratios: even a small number of repeated users, devices, or transaction sequences can make the test set fail to represent genuinely unseen cases.

Global Target Encoding

Target encoding replaces a categorical feature with label-derived statistics. A typical global estimator can be written as e_global(c) = mean(y_j for j in D where category_j = c), where D contains both training and test rows. A held-out label can therefore alter a feature observed by the model, and a test row can also be influenced by labels from other test rows in the same category.

When the category count n_c is small, one test label carries substantial weight. The encoded value becomes unstable while directly incorporating held-out information. The paper’s sparse-category stress test is consistent with this mechanism. Smoothing and minimum-frequency thresholds may reduce variance, but they do not remove the underlying label dependence.

Benchmark Design and Statistical Strength

The study covers six public datasets, two synthetic stress tests, three model families, and five repeated stratified splits, yielding 600 paired clean-versus-leaked comparisons. Pairing holds the data split and underlying model constant, reducing variance caused by differences in sample difficulty.

Inference is performed at the dataset-effect level rather than treating the 600 comparisons as independent observations. Cluster bootstrap confidence intervals reflect that hierarchy, while Holm-corrected one-sided Wilcoxon tests target the directional hypothesis that leakage inflates performance. The one-sided design matches the question being asked, and the Holm adjustment controls the family-wise error rate across the tested mechanisms.

This is considerably stronger than reporting only a pooled mean difference, but it has boundaries. The reported dataset-and-test environment yields a small number of inferential clusters, so bootstrap coverage and test power still deserve scrutiny. An average can also conceal substantial variation across datasets, model families, and category cardinalities. In addition, the supplied abstract does not identify the datasets, model families, category thresholds, or code status, making external reproduction difficult.

The direct conclusions are limited to tabular classification, ROC-AUC, and the studied IID conditions. The study does not demonstrate that global scaling and imputation are harmless under temporal drift, group shift, or online normalization, and it does not assess calibration, threshold-dependent precision and recall, or business cost. One-sided tests are also not designed to characterize settings in which leakage might reduce a metric.

Engineering Trade-offs

  1. Define the split unit before preprocessing. Use temporal splits for time-dependent data and entity-level splits for users, devices, or transactions. Deduplicate first while retaining row identifiers, group keys, and record fingerprints so the same observation cannot cross the boundary.

  2. Fit every stateful transformation on training data only. Scalers, imputers, supervised selectors, target encoders, and category vocabularies should follow fit(train) → transform(validation/test). A category first seen at evaluation time should be treated as unknown rather than triggering a new statistic computed from test data.

  3. Use out-of-fold target encoding. Each training row should receive an encoding derived from other training folds, while test rows use an aggregate learned exclusively from the training set. This increases encoding cost and requires consistent mappings at inference, but it removes the direct self-label path.

  4. Place feature selection inside the training fold. If feature selection, threshold tuning, and model selection are combined, use nested cross-validation. Strict isolation reduces effective sample size and increases experimental cost, but that cost is preferable to silently converting the test set into a validation set.

  5. Audit leakage mechanisms separately. A single Boolean leakage flag is insufficient. Track label-dependent transformations, duplicated observations, global-statistic dependencies, their measured effects, confidence intervals, and per-dataset results. Worst-case effects are more useful for release gating than a pooled average alone.

  6. Keep a locked final test set. Any model, feature, or threshold change made after viewing test results turns that test set into a validation set. Encapsulate splitting, preprocessing, feature selection, and final scoring in one pipeline, and use data-lineage controls to prevent full-data fits from crossing the split boundary.

The paper provides a severity ranking, not a universal pass-or-fail threshold. The safest operating rule remains: split first and fit second; let training labels influence only training-side statistics; isolate repeated observations by group; and apply the strictest automated audits to label-dependent transformations and observation reuse.


Source: ArXiv Machine Learning (cs.LG) (2026-11-17)

— END —

Tags: #Tabular Classification #Data Leakage #ROC-AUC #Controlled Benchmark #Statistical Audit

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…