VidDub Distills Replacement-Aware Video Dubbing into Two Diffusion Steps

Summary:VidDub introduces replacement-aware supervision for offline video dubbing. Paired-audio samples retain full-frame reconstruction, while cross-audio samples mask the conflicting source-mouth region and train lip motion through audio-visual synchronization alone, preserving identity, pose, and background. A full-context diffusion teacher is distilled into a two-step student using matched trajectory regression and distribution matching distillation. The paper reports a 0.833 improvement in cross-au

---

VidDub frames video dubbing as a conditional-replacement problem. The target speech changes, while the source mouth movements still encode the original utterance. Continuing to apply pixel-level reconstruction to the entire frame therefore pushes the model toward reproducing source mouth shapes that conflict with the replacement audio, producing weaker audio-visual alignment.

Replacement-Aware Supervision

VidDub uses two training paths according to the audio pairing relationship:

• Paired-audio samples retain full-frame reconstruction targets, teaching appearance invariance for identity, head pose, background, and original phonemes.
• Cross-audio samples mask reconstruction losses over the source mouth region and apply audio-visual synchronization supervision only to that region. Unmasked facial and background regions still receive reconstruction supervision.

The design separates two types of learning signals. Full-frame reconstruction is useful for appearance preservation, whereas audio-visual synchronization directly constrains articulation under the new speech track. Across three training seeds, the paper reports a 0.833 improvement in cross-audio Sync-C after excluding the conflicting source-mouth reconstruction target.

Distilling the Teacher into Two Steps

VidDub first trains a full-context diffusion teacher. Its CFG-guided trajectories can use the complete source video to maintain identity and background consistency, but iterative sampling remains expensive. The distillation stage combines:

1. Matched trajectory regression, which transfers the teacher's CFG-guided denoising trajectories to the few-step student.
2. Distribution matching distillation, which aligns student and teacher output distributions to mitigate out-of-distribution trajectories, texture degradation, and local artifacts.

The final student requires only two denoising function evaluations, or 2 NFE. This can reduce serial diffusion overhead in offline dubbing, but it is not automatically an equal reduction in end-to-end latency. VAE encoding and decoding, cross-attention, synchronization front ends, resampling, and video encoding remain in the pipeline.

Engineering Trade-offs

VidDub benefits from a specific offline setting: both the complete source video and the target speech are available in advance. That enables full-context processing and long-range identity consistency, but does not directly transfer to streaming dubbing where future frames and the complete target track may be unavailable.

Few-step diffusion also creates a quality trade-off. Two denoising steps can preserve broad motion and identity, yet subtle articulation, teeth, tongue motion, and high-frequency textures are more vulnerable to approximation error. The teacher’s CFG behavior also does not necessarily transfer perfectly to a two-step student.

The paper reports strong lip synchronization, visual quality, identity preservation, and visual speech recognition across four benchmarks. However, the available summary does not disclose complete baselines, parameter counts, peak memory, video resolution, frame rate, or end-to-end latency. Sync-C and similar automatic metrics can reward mechanical alignment without fully measuring perceptual acceptability, so evaluation should also include human MOS, phoneme-error analysis, extreme pose tests, occlusion handling, multi-person scenes, and long-video drift.

Data and Reproducibility

The authors plan to release a curated training dataset, VidDub-Bench, fixed paired- and cross-audio evaluation manifests, preprocessing outputs, and unified evaluation code. Fixed manifests and consistent preprocessing are important in video-editing research, where differences in audio segmentation and metric implementation can obscure real model improvements.

A stated future release is not equivalent to currently available artifacts. Independent reproduction will depend on resource availability, code completeness, base-model licenses, and the rights associated with the training data. A stronger generalization test would use speech-video combinations unseen during training and vary speakers, recording conditions, and languages.

Deployment Guidance

The most practical initial targets are post-production dubbing for film, education, and localized content, where the source video and target speech are already available. Teams should treat VidDub as an offline editing component rather than a real-time digital-human system.

Before deployment, measure:

• Full-pipeline latency and memory use, not denoiser timing alone;
• Identity drift and background flicker over long clips;
• Lip stability when replacement speech differs in timbre, pace, or emotion;
• Human acceptability in addition to synchronization metrics;
• Commercial-use compatibility of datasets, weights, and code.

If the promised artifacts are released, the most transferable idea may be the conditional switching of reconstruction supervision, rather than the two-step distillation alone. The same principle could be applied to expression transfer, motion retargeting, and other generation-editing tasks where stale visual state conflicts with a replacement condition.

---

Source: Hugging Face Daily Papers (2026-11-02)

— END —

Tags: #Video Dubbing Follow #Diffusion Models Follow #Model Distillation Follow #Lip Synchronization Follow #Diffusion Editing Follow