Kezek Opens Two-Speaker Speech Separation for Full-Duplex Dialogue Data
Summary:Kezek addresses a data bottleneck in full-duplex speech-model training: most conversations are mono mixtures, while turn-taking, overlap, and backchannel supervision require per-speaker tracks. Kezek first makes recording-level timing and identity decisions, then runs a conditioned time-domain separator whose two equal-length outputs sum back to the input. It is evaluated on short-reply retention, cross-track leakage, channel swaps, and coverage rather than standard separation scores alone. Aga
The real bottleneck is the training data
Kezek splits a recording containing exactly two people into two equal-length, time-aligned speaker tracks. Its objective is not merely to raise an average separation score, but to prevent short replies, cross-track leakage, and channel swaps from corrupting full-duplex training data.
Full-duplex speech models need isolated channels to learn turn-taking, overlapping speech, and backchannels such as “yeah” and “mm-hm,” yet most real-world conversations exist only as mono mixtures. Kezek is therefore a data-preparation layer rather than a dialogue model. It attempts to recover per-speaker tracks suitable for training or evaluation from a mixed recording. Its scope is equally important: the system is designed for exactly two speakers and does not replace general multi-speaker activity detection and clustering.
Two-stage design: consistent decisions, then waveform separation
Stage 1: recording-wide timing and identity decisions
The first stage combines a two-voice activity model with a phase-insensitive speaker fingerprint to answer both when speech is active and who is speaking. Making these decisions with whole-recording context avoids assigning each short window independently, which can cause identity flicker and channel swaps.
A phase-insensitive fingerprint may reduce identity drift caused by channel differences or phase perturbations. The available description does not reveal whether the representation is built from magnitude spectra, time-frequency statistics, or another training objective, so its noise-robustness mechanism cannot be assumed. Discarding phase cues can also remove useful identity information, particularly for speakers with similar timbres, leaving the model to rely on stronger vocal characteristics.
Global consistency is both the main advantage and the principal failure mode. If one region is incorrectly assigned to speaker B, the downstream separator usually executes that decision instead of correcting it using local evidence. The supplied material does not disclose the activity model's architecture, parameter count, training loss, or memory behavior on long recordings, all of which determine whether the approach scales to multi-hour meetings.
Stage 2: decision-conditioned time-domain separation
The conditioned time-domain separator receives the activity and identity decisions from Stage 1 and performs waveform separation. If the mixture and outputs are represented as:
x[n] = y1[n] + y2[n]
the construction constrains the two tracks to reconstruct the original mixture. This has two practical benefits. First, it avoids silently dropping difficult regions, as some post-processing pipelines do. Second, the outputs remain directly alignable with frame-level activity labels, ASR timestamps, and the source recording.
Conservation does not imply correct attribution. Energy from the wrong speaker can still be placed in a track without vanishing. During overlapping speech, the second stage still has to determine which waveform fragments belong to which speaker, but it executes rather than revisits the identity decision.
Results: coverage explains much of the headline gap
Rather than relying only on conventional metrics such as SDR or SI-SDR, the paper evaluates short-reply retention, leakage during another speaker's solo turns, channel swaps, and coverage.
| Evaluation | Kezek | pyannote→DialogueSidon | |---|---:|---:| | Short-reply retention on Open Yap | 0.83 | 0.69 | | Retention on Open Yap audio returned by both systems | 0.859 | 0.865 | | Short-reply retention over the other speaker on unseen CallHome English telephone speech | 0.80 | 0.71 | | Throughput | About 230 audio-hours per GPU-hour | Roughly one-quarter to one-fifth of Kezek |
Kezek also reports lower leakage and fewer swaps, although exact values are not provided in the available results. The critical qualification is that on the Open Yap region returned by both systems, DialogueSidon retains short replies at 0.865, slightly above Kezek's 0.859. The overall 0.83-versus-0.69 gap therefore cannot be attributed entirely to better conditional separation; the external pipeline's partial audio coverage is a major cause.
That distinction matters for evaluation. If the goal is to compare usable data produced by each system, Kezek's coverage advantage remains valuable. If the goal is to measure separation quality alone, both systems must be evaluated on their common time intervals or coverage will confound the comparison. Results on CallHome English provide limited evidence of generalization to an unseen telephone domain, but the reported comparator set is limited to pyannote→DialogueSidon and does not cover a broad set of modern separation systems.
Remaining failure mode and reproducibility gaps
The paper also reports negative results and an ideal-mask ceiling, locating the main residual error in quiet replies underneath a louder voice. Such fragments contain little energy while overlapping another speaker. Stage 1 may miss their activity, and Stage 2 may lack sufficient acoustic evidence to reconstruct the waveform even after receiving a decision.
Several reproducibility gaps remain. The available material does not specify the GPU model, batch size, sample rate, single-stream latency, peak memory, or absolute throughput. It also omits most conventional absolute separation scores, variance, and confidence intervals. Stability across recording durations is also unstated. The figure of 230 audio-hours per GPU-hour should therefore be treated as an offline throughput signal rather than proof of real-time inference capability.
Engineering trade-offs
- Global consistency versus local correction: Recording-wide identity decisions suppress frame-level jitter and swaps, but one erroneous decision can contaminate a long segment.
- Phase robustness versus identity discrimination: Ignoring phase can reduce channel-induced drift, but it may discard cues needed to distinguish similar voices.
- Conservation versus leakage: Reconstructing the input from two tracks prevents dropped audio, but does not prevent speaker A's energy from being assigned to speaker B.
- Full-recording context versus streaming: Long-range context favors stable identity, while the described design is closer to offline processing. Chunked streaming still requires boundary-artifact and identity-drift testing.
- Two-person scope versus generality: Restricting the task to two speakers simplifies the activity and identity space, but leaves three-person meetings, multi-party overlap, and crosstalk outside the supported envelope.
- Throughput versus reproducibility: The reported 4–5× speed advantage is substantial, but without hardware and memory details it cannot be extrapolated directly to consumer GPUs.
Deployment guidance
- Start with offline corpus preparation: Run Kezek over complete recordings to create tracks for full-duplex training, overlap annotation, or data augmentation. Do not place it directly into a live call path before validating streaming behavior.
- Gate on speaker count: Detect whether an input contains exactly two people before calling Kezek. Use a multi-speaker pipeline for three or more participants instead of forcing a two-track mapping.
- Conserve quiet and non-speech regions: Aggressive VAD can remove backchannels before separation. Preserve the original timeline and remove only clear clipping, device faults, or unrecoverable spans.
- Validate with dialogue-specific metrics: In addition to SDR-type scores, measure short-reply retention, leakage during solo speech, recording-level swap rate, output coverage, and track-sum reconstruction error, stratified by SNR, overlap ratio, and telephone codec.
- Use equal-coverage comparisons: Compare separation quality only on time regions returned by both systems. Measure dropped duration and coverage separately when assessing data-production value.
- Treat outputs as auditable pseudo-labels: Inspect low-energy overlap regions and acoustically similar speakers, then filter systematically swapped intervals rather than allowing incorrect tracks to enter a full-duplex training set.
- Benchmark the target hardware: Measure first-package latency, sustained throughput, memory use, and long-recording stability on the intended GPU, then verify the repository's actual license, weight format, and inference dependencies.
— END —Source: Hugging Face Daily Papers (2026-11-12)
Tags: #Full-Duplex Dialogue #Two-Speaker Speech Separation #Backchannel Retention #Dialogue Data Pipeline #Open-Source Weights
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments