ZICQ
中 Log in / Sign up
Newsroom Research & Papers #VideoLLM #Supervised Fine-Tuning #Orthogonal Projection #Textual Shortcut #Out-of-Distribution Generalization

VETO Suppresses Text-Only Shortcuts in Video-LLM Fine-Tuning via Orthogonal Projection

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:VETO addresses a subtle failure mode in supervised fine-tuning for video-language models: a model can improve its training-distribution accuracy by exploiting textual regularities in the supervision data while bypassing the video entirely. Rather than cleaning or reweighting the dataset, VETO calibrates the activation subspace occupied by text-only inputs in the base model and projects the fine-tuning update onto its orthogonal complement. The paper reports results across seven Video-LLMs and s


Higher SFT Accuracy Does Not Prove Visual Grounding

During supervised fine-tuning, a video-language model learns the conditional distribution p(y|q,v), where q is the question and v is the video. Yet question wording, answer templates, and video categories may be correlated in the training set. The model can therefore learn a text-only shortcut s(q) without adequately learning the visually dependent function f(q,v).

Within the training distribution, both s(q) and f(q,v) can improve accuracy, so conventional metrics cannot distinguish them. When question, answer, or visual distributions change, the textual correlation breaks while visual reasoning remains weak. This produces the familiar pattern of higher in-distribution accuracy but stagnant or degraded out-of-distribution performance. VETO attempts to separate these two capabilities during optimization.

How VETO Changes the Fine-Tuning Update

Instead of cleaning data, down-weighting text loss, or masking the question during inference, VETO intervenes in the base model's activation space. It first calibrates the subspace occupied by text-only inputs. If the columns of U form an orthonormal basis for that subspace, so that UᵀU = I, the conceptual projection of a fine-tuning direction Δz is:

Δz⊥ = Δz - U(UᵀΔz) = (I - UUᵀ)Δz

The term U(UᵀΔz) is the component of the update lying in the text-only subspace. Removing it leaves an update in the orthogonal complement, preventing the newly written fine-tuning direction from directly reinforcing the identified shortcut and encouraging the model to use evidence from the combined question-video input.

The abstract does not specify whether the projector acts directly on a parameter task vector, layer-wise gradients, or an equivalent surrogate in activation space. The implementation therefore cannot be reduced to a single multiplication over raw model parameters; it must establish a valid correspondence between parameter updates and the activation subspace. The public Youngwoo-git/VETO repository is the primary source for resolving this detail.

Why an Orthogonal Constraint

Reducing text-token loss or rebalancing examples only discourages shortcuts indirectly: the model may discover alternative linguistic regularities. VETO instead constrains the geometry of optimization, preventing new updates from being written along the identified text subspace.

The trade-off is that a text subspace is not purely nuisance. Subtitle interpretation, commonsense reasoning, and answer-format learning may also occupy language-related directions. An overly broad estimate can suppress legitimate language capability, while an overly narrow estimate can leave alternative shortcuts untouched. If the estimated basis is not orthonormal, the corresponding projector is U(UᵀU)⁻¹Uᵀ; numerical stability and rank selection then become material training concerns. The supplied abstract does not disclose the relevant ablations.

Empirical Claims and Evidence Boundaries

The abstract reports two levels of coverage: VETO preserves standard SFT gains within the training distribution across seven Video-LLMs, and on six benchmarks under distribution shift it performs better than SFT. The authors attribute the latter result to greater reliance on the complete input rather than the shortcut.

This supports the method's central hypothesis, but the magnitude cannot yet be assessed. The abstract provides no model list, benchmark names, absolute accuracy, random seeds, variance, significance tests, subspace rank, or projection-layer selection. Coverage across seven models and six shifted benchmarks is useful, but it does not replace effect sizes and statistical confidence. It also remains unclear whether the shifts cover question rewrites, answer-template changes, visual-domain shifts, and temporal shifts, or whether VETO wins consistently across all of them.

Engineering Costs and Deployment Trade-offs

  • Calibration cost: Estimating the text-only activation subspace requires collecting activations under text-only inputs. With n samples in d-dimensional activation space, covariance estimation and eigendecomposition have approximate costs of O(nd²) and O(d³); storing a rank-r basis requires about O(dr) memory. Actual costs depend on sampling, depth, and numerical routines.
  • Training cost: Activation-level projection adds roughly O(dr) work per update. A parameter-level task-vector implementation needs an additional mapping between the two spaces. This is likely cheaper than rebuilding datasets or repeating SFT, but GPU throughput and peak-memory measurements are still necessary.
  • Capability trade-off: Removing a text subspace may also weaken linguistic priors. Teams should monitor the energy of projected updates by layer rather than choosing one global rank without analysis.
  • Inference cost: As described, VETO acts mainly during training and should not require a changed inference architecture. Whether an auxiliary calibration branch or base-model refresh is needed at runtime must still be confirmed from the code and configuration.
  • PEFT compatibility: LoRA or QLoRA updates are not necessarily aligned with the text subspace. Projecting the low-rank factors, the accumulated update, or the gradient may produce materially different results, so these variants require separate experiments.

Reproduction and Deployment Guidance

  1. Measure the shortcut first: Add blank, mismatched, shuffled, and counterfactual-video ablations, and report the accuracy gap between full input and text-only input rather than only final task scores.
  2. Validate the projection site: Compare activation-level, gradient-level, and parameter-task-vector implementations. Sweep subspace rank, projection layers, and projected update energy to ensure the method is not collapsing into ordinary gradient clipping.
  3. Separate shift types: Test question rewriting, answer-format changes, visual-domain transfer, and temporal distribution shift independently so that one shift cannot conceal a failure mode.
  4. Track quality and systems metrics: In addition to accuracy, report visual reliance, confidence calibration, training and inference latency, peak memory, and variance across random seeds.
  5. Use strong controls: Compare vanilla SFT with text decorrelation, resampling, text-loss down-weighting, gradient projection, and data augmentation to determine whether VETO offers more than generic regularization.

VETO reframes robust multimodal post-training from a data-cleaning problem into an optimization-geometry problem: which representation directions are allowed to receive new updates. If its projection cost, subspace stability, and distribution-shift results can be independently reproduced, it could become a broadly reusable reliability layer for Video-LLM fine-tuning.


Source: Hugging Face Daily Papers (2026-11-19)

— END —

Tags: #VideoLLM #Supervised Fine-Tuning #Orthogonal Projection #Textual Shortcut #Out-of-Distribution Generalization

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…