ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #Multimodal Large Language Models #Inference Optimization #Visual Token Pruning #Attention Prediction

MAP: Enhancing Multimodal Large Language Model Efficiency via Sample-Specific Middle-Layer Attention Prediction

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 8 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:The paper introduces Middle-layer Attention Prediction (MAP), a novel approach to improve the efficiency of Multimodal Large Language Models (MLLMs). MAP employs a Question Contrastive Teacher Selection mechanism to dynamically identify a sample-specific teacher layer and distills its attention into a lightweight predictor for estimating visual token importance during inference. Experimental results on LLaVA-NeXT-7B demonstrate that MAP retains 97.5% of the unpruned model performance while proce


Background and Challenges

Multimodal Large Language Models (MLLMs) demonstrate strong performance in vision-language tasks, but their inference efficiency is limited by the high cost of processing numerous visual tokens. Visual token pruning is an effective optimization strategy, but it requires accurate estimation of token importance. Current methods typically use attention from a predefined middle layer to select visual tokens to retain, but face two main challenges:

  1. The layer most responsive to the question varies across samples, making a fixed layer suboptimal.
  2. Obtaining attention from the appropriate middle layer requires processing numerous visual tokens through several language model layers, leading to significant computational overhead.

Methodology and Innovation

To address these issues, the paper proposes Middle-layer Attention Prediction (MAP), featuring the following innovations:

  • Question Contrastive Teacher Selection: Dynamically identifies a sample-specific teacher layer by contrasting attention under the original and reference questions.
  • Attention Distillation and Lightweight Predictor: Distills the selected layer's attention into a lightweight predictor that estimates visual token importance from multimodal input features.
  • Inference-Time Pruning: During inference, MAP combines the predicted importance scores with a diversity criterion to prune visual tokens before the first language model layer.

Experimental Results and Performance

MAP demonstrates excellent performance on LLaVA-NeXT-7B across ten benchmarks:

  • Performance Retention: Retains 97.5% of the unpruned model performance.
  • Visual Token Usage: Processes only 5.56% of the visual tokens.
  • Speedup: Achieves a 3.09x end-to-end speedup.

Technical Highlights

  • Dynamic Adaptability: MAP can dynamically select the most suitable teacher layer for different samples, overcoming the limitations of fixed-layer methods.
  • Efficiency: The lightweight predictor allows MAP to avoid generating attention maps during inference, making it compatible with existing inference acceleration techniques.
  • Scalability: The method can be applied to various multimodal large models, offering broad application potential.

Industry Impact and Developer Recommendations

MAP provides a new approach to optimizing the inference efficiency of multimodal large models, particularly in resource-constrained environments such as mobile devices or edge computing. Developers can leverage MAP's methodology to design more efficient visual token pruning strategies and explore its application in other domains.

Future Directions

Future research could further optimize the predictor architecture of MAP and explore its performance in larger models and more complex tasks. Additionally, MAP can be combined with other inference acceleration techniques to achieve more comprehensive performance improvements.

Original Source

Source: arXiv:2608.06411v1


Source: ArXiv AI (cs.AI) (2026-08-10)

— END —

Tags: #Multimodal Large Language Models #Inference Optimization #Visual Token Pruning #Attention Prediction

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…