SAGE-CP: Addressing Route-Conditional Failures in Adaptive VLM Ensembles

Summary:SAGE-CP is a novel adaptive Vision-Language Model (VLM) ensemble pipeline designed to address the issue of route-conditional failures caused by insufficient marginal coverage in traditional methods. By freezing routing choices and calibrating separately for early exits and escalated inputs, SAGE-CP significantly improves the worst-route coverage (from 0.623 to 0.887) across multiple benchmarks while reducing the route-coverage gap from 0.324 to 0.025. This approach maintains low computational co

---

Background and Problem

Adaptive Vision-Language Model (VLM) ensembles save computational resources by exiting early on easy images and escalating to more complex models for harder ones. However, a critical issue arises: even if the final predictor is marginally valid, specific computational routes may still fail severely. This failure stems from the mismatch between marginal coverage and conditional coverage: if a route has prevalence pi, marginal 1-alpha coverage allows its conditional coverage to drop to max{0, 1-alpha/pi}.

SAGE-CP Solution

To address this, SAGE-CP introduces a new ensemble pipeline:

1. Router Learning and Freezing: The routing choices are learned on development data and then frozen to ensure route stability.
2. Calibration Step: A separate calibration is applied for early exits and escalated inputs. Specifically, SAGE-CP applies Mondrian split conformal prediction to calibrate the operational groups created by the router.
3. Route-Level Reliability Guarantee: Under exchangeability, the resulting prediction sets achieve finite-sample coverage within each route.

Experimental Results

Across 10 benchmarks, 4 frozen VLMs, and 5 aligned splits, both SAGE-CP and global calibration achieve the 0.900 marginal coverage target, but their route behavior differs significantly:

• Worst-Route Coverage: SAGE-CP raises the average worst-route coverage from 0.623 to 0.887.
• Route-Coverage Gap: SAGE-CP reduces the average route-coverage gap from 0.324 to 0.025.
• Prediction Set Size: Although the prediction set size increases from 1.893 to 2.212, it remains smaller than that of the full ensemble (2.486), and the number of source calls per image is reduced by 70.6%.

Furthermore, SAGE-CP maintains route-level reliability across 6 fixed query fractions, while finer exit-depth groups indicate the need for additional calibration data on rare routes.

Performance and Efficiency

Across 5 datasets, SAGE-CP reduces measured batch-one latency by 58.8% to 63.4% and gross GPU energy by 55.7% to 57.5% relative to full inference. This demonstrates that routing provides computational savings and that route-stratified calibration attaches coverage guarantees to the operational paths created by the router.

---

Source: ArXiv Machine Learning (cs.LG) (2026-10-21)

— END —

Tags: #Sagemath Follow #VLM Follow #Adaptive Ensemble Follow #Route Calibration Follow #Computational Efficiency Follow