ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical Vision-Language Models
Summary:ModaLens is an innovative paired image-swap audit method designed to measure the image sensitivity of report-conditioned medical Vision-Language Models (VLMs). Experiments on the MedGemma-27B model using 3,199 paired MIMIC-CXR cases show that with the report, the model's answer changes on 4.26% of trials, while without the report, the change rate jumps to 20.94%, indicating that report availability significantly reduces image-swap sensitivity. This research sheds light on the dependency of medic
Background and Motivation
In the medical field, Vision-Language Models (VLMs) are widely used to extract information from medical images and generate clinical reports. However, evaluating whether these models truly utilize the image information has been a challenge. ModaLens offers a novel evaluation method through paired image-swap auditing to measure the impact of report availability on the model's image sensitivity.
Methodology and Experiments
The research team used the MedGemma-27B model to conduct experiments on 3,199 paired MIMIC-CXR cases from 293 patients. Each case included 14 questions (13 finding-specific and one composite), and the model's sensitivity was tested by replacing images with those from another study of the same patient. The results showed that with the report, the model's answer changed on 4.26% of trials, while without the report, the change rate was 20.94%, indicating that the report significantly reduces image-swap sensitivity.
Key Findings
- Impact of Report: The presence of the report reduces the model's image-swap sensitivity by 16.7 percentage points.
- Model Consistency: The directional results are consistent across two different model lineages, suggesting the method's generalizability.
- Limitations: Due to labels derived from reports, conclusions about visual correctness are limited.
Technical Highlights
- Innovative Method: ModaLens introduces a new paired image-swap auditing method for evaluating medical VLM image sensitivity.
- Data Scale: The experiments used a large amount of real-world medical imaging data, ensuring the reliability of the results.
- Multi-Model Validation: The consistency of results across multiple model lineages enhances the study's credibility.
Industry Impact and Developer Recommendations
The findings of ModaLens have significant implications for the application of medical VLMs. Developers can use this method to improve the interpretability and reliability of their models. Additionally, the study provides new insights for future medical AI research, such as how to better combine image and textual information to enhance model performance.
— END —Source: Hugging Face Daily Papers (2026-09-14)
Tags: #Medical AI #Vision-Language Models #Image-Swap Auditing #MIMIC-CXR #Model Evaluation
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments