SD-ViT Enables Budget-Aware Early Exit for Face Recognition Without Gallery Rebuilds

Summary:SD-ViT applies stochastic deep supervision to make intermediate Vision Transformer blocks produce directly usable identity embeddings. During each training iteration, one intermediate block is sampled and optimized through the same embedding head and classifier as the final layer, using a weighted margin-penalty softmax loss. This aligns representations across depths in a shared embedding space, allowing operators to select a fixed early exit according to their compute budget while comparing pro

---

Core Idea: Early Exits That Produce Usable Identity Embeddings

SD-ViT addresses fixed-depth inference in Vision Transformer face recognition. A conventional ViT backpropagates the final loss through intermediate blocks, but those blocks are never directly optimized to produce embeddings suitable for identity similarity matching. An intermediate exit may therefore exist structurally while still producing representations that are unusable for recognition.

SD-ViT adds stochastic deep supervision. At each training iteration, it samples an intermediate block and passes its output through the same embedding head as the final layer. A weighted margin-penalty softmax loss then directly optimizes that output. Because the intermediate and final branches share both the embedding head and classifier, their representations are constrained to a common identity space.

At inference time, an operator selects a fixed exit depth according to the available compute budget. The resulting probe embedding can be compared directly with unchanged full-depth gallery embeddings, avoiding gallery recomputation and adaptation layers.

The method does not dynamically decide where each individual sample should exit. Instead, it trains several budget-selectable, fixed-depth exits. This converts early-exit routing into a deployment configuration that can be determined ahead of time.

Training Mechanism: Cross-Depth Alignment Through a Shared Classifier

In a standard ViT, intermediate representations are constrained only indirectly through backpropagation from the final objective. Different depths may carry related semantics, but they are not explicitly required to form embeddings that can independently support recognition.

For iteration (t), SD-ViT samples an intermediate block (h_t) and applies a loss of the form:

[ L_{SD}=\sum_t p_t\left[\alpha L_{CE}(f_e(h_t),y)+\beta L_{Margin}(f_e(h_t),y)\right] ]

Here, (f_e) is the shared embedding head, (L_{CE}) is the standard identity classification loss, and (L_{Margin}) is a margin-penalty softmax objective. The final layer continues to use the original training objective, preserving full-depth performance.

Stochastic sampling provides two practical benefits:

• The system does not need to compute a full auxiliary loss at every intermediate depth in every forward pass, limiting training memory and compute overhead.
• Every depth repeatedly receives direct supervision, reducing the risk that one fixed auxiliary branch dominates the optimization or acquires an undesirable training-order bias.

The shared head and classifier are essential. If each exit had its own classifier or projection layer, the training objective could push different depths into incompatible embedding spaces and break cross-depth gallery matching.

Gallery Consistency: More Than a Latency Saving

Face-recognition galleries normally store embeddings extracted from enrollment images rather than retaining the raw images for repeated full-depth inference. Most early-exit approaches therefore face a gallery-domain shift: probe embeddings produced at shallower depths are not directly comparable with full-depth gallery embeddings. Common remedies include rebuilding galleries, introducing distillation losses, or inserting adapter networks into the query path.

SD-ViT uses the shared classification objective to align intermediate embeddings with full-depth embeddings. Migration can be relatively direct: keep the existing full-depth gallery, generate a probe embedding at the selected depth, and compute cosine similarity without re-enrolling all subjects or adding a runtime adapter.

This matters for large galleries. Rebuilding a gallery consumes GPU inference time and can trigger data-access, privacy, and operational review procedures. Those costs may exceed the savings from a faster online query path.

Performance and Computational Trade-offs

On established benchmarks including IJB-C, the paper reports that full-depth SD-ViT remains within 0.4 IJB-C points of a conventionally trained model with the same backbone. The more important result concerns the exit points:

• At approximately 50% of the computational cost, early-exit probes recover more than 96% of their own full-depth performance.
• A single verification threshold calibrated at full depth remains usable from quarter depth onward.
• The reference gallery does not need to be recomputed, and early-exit embeddings remain directly comparable with existing full-depth embeddings.

The reported 50% computational cost should not automatically be interpreted as an exact 50% wall-clock latency reduction across every platform. Transformer execution is approximately linear in depth, but measured latency also depends on batch size, operator fusion, memory bandwidth, branching overhead, and accelerator-specific kernels. Production evaluation should separately measure end-to-end P50 and P95 latency, throughput, and peak memory.

Evaluation Limitations and Reproduction Checks

The results come from author-run experiments on established benchmarks and do not directly establish gains for real-world access control, mobile deployments, or very large search systems. A stronger evaluation should examine:

1. Verification, false-accept, and false-reject performance at every exit depth, rather than only relative performance recovery.
2. Whether one threshold remains stable across demographic groups, capture devices, and environmental noise.
3. End-to-end cost compared with dynamic early exit, distillation-based exits, and gallery rebuilding.
4. Whether stochastic deep supervision requires additional training epochs and whether results remain stable across random seeds.
5. Whether the similarity distribution between shallow probes and full-depth galleries remains stable as gallery size, enrollment age, or template-update policies change.

The reported recovery of more than 96% of full-depth performance is a relative measure. If the full-depth model's absolute accuracy is already insufficient for a deployment, a high recovery ratio may still be operationally inadequate. Production thresholds should therefore be recalibrated against the cost of false accepts and evaluated separately from closed-set verification performance.

Deployment Guidance

For teams balancing recognition quality and inference cost, SD-ViT offers a relatively low-friction migration path: retain a standard full-depth gallery, train multiple usable exits in a shared embedding space, and assign fixed depths to different hardware tiers instead of regenerating a separate gallery for each deployment profile.

A practical rollout can proceed as follows:

• Reproduce the depth-versus-performance curve under the target backbone, sampling probability, and loss weights.
• Export full-depth gallery embeddings together with model version, preprocessing parameters, and normalization rules.
• Measure end-to-end latency, throughput, and memory for full, half, and quarter depth on the target accelerator.
• Recalibrate the verification threshold with independent business data and test its stability across depths and devices.
• Re-extract templates and galleries uniformly at full depth whenever the model version changes.

The main contribution of SD-ViT is not a new peak face-recognition score. It connects intermediate-layer supervision, budgeted inference, and reuse of existing galleries through a shared interface, making early exit more practical to deploy.

---

Source: Hugging Face Daily Papers (2026-11-02)

— END —

Tags: #SD-ViT Follow #Face Recognition Follow #Vision Transformer Follow #Stochastic Deep Supervision Follow #Early Exit Follow