arXiv Introduces MOSAIC: Revolutionizing Cross-Modal Training Sample Attribution for Enhanced VLM Interpretability

Summary:arXiv introduces MOSAIC, a novel method for cross-modal training sample attribution aimed at enhancing the interpretability of Vision-Language Models (VLMs). MOSAIC learns a sparse image-text latent space and employs the EK-FAC algorithm to decompose training sample influence into specific text phrases and image regions, enabling more precise attribution. Experiments demonstrate that MOSAIC excels in multiple tasks, significantly improving the interpretability of model outputs concerning trainin

---

Technical Mechanism Analysis

MOSAIC (Multimodal Orthogonalized Sparse Autoencoders for Influence Characterization) achieves cross-modal training sample attribution through the following steps:

1. Sparse Latent Space Learning: MOSAIC learns a shared sparse image-text latent space where concepts are sparse within each modality, supported on contiguous image regions, aligned with the phrases that describe them, and approximately decorrelated.

2. EK-FAC Algorithm Application: The EK-FAC algorithm is employed to decompose the influence of training samples through these concepts and localize their contributions to specific text phrases and image regions.

3. Attribution and Enhanced Interpretability: This decomposition allows MOSAIC to more precisely explain model behavior and quantify the impact of different training samples on model outputs.

Engineering Trade-offs and Performance

• Advantages:

  • Achieves finer-grained attribution by localizing influence to specific text phrases and image regions.
  • Demonstrates stable performance across multiple tasks (e.g., captioning, chest X-ray understanding, visual question answering) without negatively impacting task performance.
  • Attribution results are more interpretable, and ablation experiments show that MOSAIC-based attribution changes model outputs more effectively than attribution based on activation or frequency.

• Challenges:

  • Higher computational complexity, requiring additional computational overhead during training.
  • Higher requirements for the quality of multimodal data alignment; imperfect alignment may lead to attribution bias.

Developer Implementation and Deployment Recommendations

• Application Scenarios:

  • Suitable for multimodal model applications that require high interpretability, such as medical diagnostics, autonomous driving, and intelligent customer service.

• Deployment Recommendations:

  • Deploy in resource-abundant environments to accommodate additional computational overhead.
  • Combine with other attribution methods (e.g., gradient-based attribution) for cross-validation to enhance the reliability of attribution results.

• Future Outlook:

  • Future research could further optimize MOSAIC's computational efficiency and explore its application to other types of multimodal data (e.g., video, audio).

Conclusion

MOSAIC provides a new approach to cross-modal training sample attribution, enhancing the transparency and reliability of vision-language models through more precise attribution and higher interpretability.

---

Source: ArXiv AI (cs.AI) (2026-10-15)

— END —

Tags: #MOSAIC Follow #Cross-Modal Follow #Training Attribution Follow #Vision-Language Models Follow #arXiv Follow