arXiv Releases MOSAIC: Novel Attribution Method for Vision-Language Models

Summary:arXiv introduces MOSAIC, a novel method for explaining vision-language model (VLM) predictions. MOSAIC learns a shared sparse image-text latent space and employs EK-FAC decomposition to attribute training sample influence to specific text phrases and image regions. Experiments on Qwen and Gemma demonstrate that MOSAIC provides more localized and coherent feature attribution without degrading task performance. This research offers a new technical pathway for the interpretability of multimodal mod

---

Technical Mechanism Analysis

MOSAIC (Multimodal Orthogonalized Sparse Autoencoders for Influence Characterization) is a novel method for attributing training sample influence in vision-language models (VLMs). Its core mechanisms include:

1. Shared Sparse Latent Space Learning: MOSAIC learns a shared sparse image-text latent space, ensuring that concepts within each modality are sparse, image regions are contiguous, descriptive phrases are aligned, and the space is approximately decorrelated.

2. EK-FAC Decomposition: Using EK-FAC, MOSAIC decomposes training sample influence through these concepts, attributing the influence to specific text phrases and image regions.

3. Feature Attribution and Model Behavior Connection: MOSAIC not only provides more localized and coherent feature attribution but also demonstrates how visual evidence influences model behavior while preserving the shared latent structure.

Engineering Trade-offs and Performance

• Advantages:

  • Offers finer-grained attribution capabilities, localizing influence to specific text and image regions.
  • Demonstrates stable performance across multiple tasks (e.g., captioning, chest X-ray understanding, visual question answering) without degrading task performance.
  • More effective in changing model outputs compared to attribution methods based on activation or frequency.

• Challenges:

  • The learning of the sparse latent space and the EK-FAC decomposition introduce significant computational overhead, potentially impacting the training and inference speed of large-scale models.
  • Requires thorough training on multimodal datasets to ensure the accuracy and consistency of attribution.

Developer Implementation and Deployment Recommendations

• Application Scenarios:

  • Suitable for multimodal model applications that require high interpretability, such as medical diagnostics, autonomous driving, and intelligent customer service.

• Deployment Recommendations:

  • In practical applications, it is recommended to combine MOSAIC with existing model interpretation tools to provide a more comprehensive explanation of model behavior.
  • To address computational overhead, consider using distributed computing or hardware acceleration technologies to optimize the training and inference process.

• Future Directions:

  • Further research on the applicability of MOSAIC to different types of multimodal models and tasks.
  • Exploration of more efficient computational methods to reduce the attribution cost of MOSAIC.

---

Source: ArXiv AI (cs.AI) (2026-10-14)

— END —

Tags: #MOSAIC Follow #Multimodal Models Follow #Model Interpretability Follow #arXiv Follow