Skip to main content
ZICQ

Topics Multimodal & Video Generation

Multimodal & Video Generation

Image, video, and speech generation plus production practice: Sora, Veo, Kling, Suno, and the open-source multimodal stack.

· Sep 15, 2026 #Multimodal AI#video generation#Multimodal#image generation#speech generation#voice cloning#Sora#Veo#Kling#Suno · 30 / 638 articles

News · 2026-08-27

ArXiv Proposes RFQ Framework: Addressing Activation Outliers in Low-bit Quantized Multimodal LLMs

ArXiv has released a study on low-bit quantization for Multimodal Large Language Models (MLLMs), introducing a lightweight activation reconstruction framework called Residual Fallback Quantization (RFQ). The research identifies activation quantization as the primary source of performance degradation in aggressive 4-bit quantization schemes, while 8-bit quantization achieves near-lossless performance. RFQ supplements the primary ultra-low-bit activation representation with an auxiliary quantized

News · 2026-08-27

ArXiv Introduces SAGE: A CLIP-Based Framework for Variate-Wise Semantic Augmentation in Vision-Language Time Series Fore

ArXiv introduces SAGE (Seeing and Augmenting with Grounded Encoding), a novel framework addressing the lack of semantic knowledge in time series forecasting. Built on the CLIP model, SAGE jointly models temporal, cross-variable, textual, and visual information, significantly improving prediction accuracy. It employs frequency-enhanced text encoding and variable-specific descriptor injection mechanisms, alongside a frozen CLIP vision encoder for multimodal alignment. Experimental results demonstr

News · 2026-08-27

ArXiv Proposes MedREAL: Grounded Medical Multimodal LLMs for Precise Pixel-Level Localization

ArXiv introduces MedREAL, a novel framework designed to address the lack of precise pixel-level grounding in Medical Visual Question Answering (Med-VQA) by Multimodal Large Language Models (MLLMs). MedREAL incorporates Seg Anchored Reasoning Pooling (SARP) and Reasoning-to-Visual (R2V) fusion to align linguistic reasoning with spatial localization seamlessly. Extensive experiments demonstrate that MedREAL significantly outperforms existing state-of-the-art methods, achieving higher accuracy and

News · 2026-08-27

ArXiv Proposes Pixel-level Table Compression for Improved Multi-Table Document QA Efficiency

ArXiv introduces a novel pixel-level table compression method for multi-table document question answering (QA). This approach represents tables as images to reduce token costs while preserving critical information. Experiments show that highly compressed table images allow models to identify relevance effectively and process them at native resolution for reasoning, resulting in a 41% reduction in total tokens and a 7% increase in QA accuracy across two benchmarks.

News · 2026-08-27

ArXiv Releases Multi-Modal AI Framework for Optimizing Real-Time Queue Management in Intelligent Border Control

ArXiv introduces a novel multi-modal AI framework tailored for intelligent border control systems. By integrating heterogeneous data sources and employing LSTM networks for real-time queue prediction, scheduling optimization, and resource management, the framework significantly enhances border control efficiency. Experimental results demonstrate a 35% reduction in queue prediction error, a 30% decrease in average waiting time, and nearly a 20% increase in average throughput compared to tradition

News · 2026-08-27

ArXiv Releases OCSD Model: Breakthrough in Multi-Person Human Motion Forecasting in Complex Scenes

ArXiv has introduced Object-Conditioned Social Diffusion (OCSD), a conditional diffusion model for accurate multi-person motion forecasting in complex scenes. The model integrates motion history, multi-person interactions, and object cues into a unified framework, enabling fine-grained human-object reasoning and sampling of multiple plausible futures. Experiments show that OCSD achieves state-of-the-art results on the Humans in Kitchens (HiK) and HOI-M3 benchmarks, reducing the two-second path e

News · 2026-08-27

ArXiv Introduces OmniUE: The First Omni-Interactive Universal Embedder

ArXiv has introduced Omni-Interactive Universal Embedder (OmniUE), the first model to learn a unified embedding space across text, video, and audio modalities while supporting omni-interactive querying in the form of text, visual regions, and audio spans. OmniUE demonstrates significant performance improvements across diverse modalities, outperforming state-of-the-art baselines by up to 83.7% in visual-interactive benchmarks. This research paves the way for more versatile and powerful multimodal

News · 2026-08-27

ArXiv Releases ContraTalk: A Benchmark for Audio-Grounded Dialogue Reasoning with Focus on Cross-Modal Disagreements

ArXiv has released ContraTalk, an audio-grounded dialogue reasoning benchmark designed to address the shortcomings of existing dialogue models in handling cross-modal disagreements. The benchmark identifies text-biased interpretations and converts disagreement regions into conflict QA examples to evaluate model performance in processing inconsistent acoustic signals. Experiments show that current models struggle with cross-modal inconsistencies, while the Audio Twin framework significantly impro

News · 2026-08-27

ArXiv Releases MM-Spectrum: A Sparse Mixture-of-Experts Framework for Multimodal Molecular Structure Elucidation

ArXiv has released MM-Spectrum, a novel multimodal molecular structure elucidation model leveraging a sparse Mixture-of-Experts (MoE) framework. The model introduces an explicit modality-aware routing mechanism and heterogeneous expert capacities to address the heterogeneity and imbalance in multispectral signal integration. MM-Spectrum demonstrates substantial performance improvements across full-modality, bimodal, and missing-modality settings, validated by ablation studies and interpretabilit

News · 2026-08-27

ArXiv Releases PAWBench: A Benchmark for Probabilistically Aligned World Modeling in Video Generation

ArXiv has released a study on the probabilistic alignment capabilities of video generation models and introduced PAWBench, a benchmark for evaluating these models as stochastic samplers of world dynamics. The research highlights that existing evaluations primarily focus on the plausibility of individual videos rather than the alignment of the distribution of possible behaviors. PAWBench addresses this gap by converting repeated video rollouts into empirical distributions of physical behaviors. E

News · 2026-08-27

CLAP Framework Released: Cross-Embodiment Video World Models Enable Zero-Shot Physical Simulation

CLAP is a novel cross-embodiment video generation framework capable of training on diverse, internet-scale videos across human and robotic agents. Its key innovation lies in reconciling disparate action spaces using end-effector poses, language instructions, and latent actions, and introducing a curriculum-based learning approach to learn foundational physical priors from unlabeled videos. CLAP matches or surpasses state-of-the-art single-embodiment video models in challenging environments like

News · 2026-08-27

arXiv Proposes CAT-GS: Addressing Instability in Multimodal Neural Network Training

arXiv introduces CAT-GS (Calibrated, Adaptive, Thresholded Gating with Fusion Surgery), a novel neural dynamics-based optimization controller designed to address three core challenges in multimodal neural network training: modality imbalance, unstable gating, and fusion interference. CAT-GS operates during backpropagation, utilizing temperature scaling and EMA smoothing for teacher reliability calibration, and employs a margin-thresholded policy alongside fusion-only PCGrad to stabilize gradient

News · 2026-08-27

arXiv Releases Comprehensive Survey on Multi-Modal Anomaly Detection: Addressing Core Challenges and Future Directions

arXiv has released a comprehensive survey on Multi-Modal Anomaly Detection (MMAD), systematically addressing the field's core challenges and future directions. The paper categorizes existing methods into two paradigms—normality-assumption and anomaly-assumption—and explores how foundation models are reshaping MMAD through scalable pretraining, flexible cross-modal transfer, and emerging reasoning capabilities. Additionally, the survey compiles representative benchmarks and evaluation protocols,

News · 2026-08-26

DeepSeek Launches V4 Pro: Outperforms Fable 5 with 3x Lower Cost per Task

DeepSeek has launched its latest language model, V4 Pro, which excels in SWE-Bench and LiveCodeBench benchmarks, outperforming Fable 5 and reducing the cost per task by threefold. The model features a 1-million-token context window, native tool calling, and preserved reasoning history, targeting long-horizon agentic workloads. While strong in Rust and C++, V4 Pro lags slightly in Java compared to Fable 5. Fireworks AI offers full training and deployment support, enabling rapid domain adaptation

News · 2026-08-26

ArXiv Proposes Hierarchical Multimodal MoE for Improved ILD Diagnosis

The ArXiv team introduces a Hierarchical Multimodal Mixture-of-Experts (MoE) model for classifying Interstitial Lung Disease (ILD). This model integrates a pre-trained imaging expert with structured Electronic Health Records (EHR) through a two-stage gating mechanism, enabling patient-specific weight assignment and clinically defined feature grouping. Experimental results demonstrate that the model achieves the highest mean AUC (0.8750 ± 0.0443) among evaluated methods under strict patient-level

News · 2026-08-26

ArXiv Introduces WAVE Framework: Revolutionizing Coarse-to-Fine Guided Depth Super-Resolution

The ArXiv team introduces WAVE, a novel method that leverages Multi-Level Discrete Wavelet Transform (ML-DWT) as an explicit feature-control mechanism to enable coarse-to-fine depth super-resolution reconstruction. By processing sub-bands and semantic tokens in reverse order, WAVE effectively reduces artifacts and blurred edges, outperforming existing methods in multiple benchmarks, particularly in high upsampling scenarios. This research offers a new technical pathway for depth image processing

News · 2026-08-26

ECNU-RAIL Releases OmniPhys: First Multimodal Benchmark for Physics Understanding and Generation

ECNU-RAIL introduces OmniPhys, a large-scale multimodal benchmark for physics understanding and reasoning, covering middle school to university-level problems from Chinese educational corpora. OmniPhys consists of 15,246 questions and 19,850 images, accompanied by detailed annotations for fine-grained analysis of reasoning processes and knowledge usage. Beyond conventional evaluation, OmniPhys systematically assesses models' ability to generate structured physics diagrams, which are fundamental

News · 2026-08-26

ArXiv Releases VGA-BenchV2: Enhanced Benchmark for Video Generation and Aesthetic Quality Evaluation

ArXiv has released VGA-BenchV2, an enhanced human-aligned benchmark and optimization framework for jointly evaluating and improving video generation quality and aesthetic value. Building upon the original VGA-Bench, VGA-BenchV2 expands the taxonomy with 52 sub-dimensions and introduces 36,000 new task-level annotations, including 16,200 for aesthetic quality, 13,200 for aesthetic tagging, and 6,600 for generation quality. This substantial increase in labeled data provides a more comprehensive ev

News · 2026-08-26

ArXiv Releases 4DStreamCtrl: Real-time Controllable 4D Video Generation Model

ArXiv introduces 4DStreamCtrl, an innovative video generation model that enables joint control of camera motion, object trajectories, and depth in real-time streaming generation. By unifying 3D point-track representations with a lightweight Geometric Motion Head and distilling the model into a causal streaming student, 4DStreamCtrl achieves interactive 4D-controllable video generation for the first time. It operates at 20 FPS on a single high-end GPU for 480p video and maintains temporal coheren

News · 2026-08-26

ArXiv Releases MMJailBench: A Benchmark for Evaluating Multimodal Jailbreak Vulnerabilities

ArXiv introduces MMJailBench, a benchmark designed to evaluate the jailbreak vulnerabilities of Multimodal Large Language Models (MLLMs). By systematically varying and combining factors such as harmful intent, prompt framing, visual semantics, and instruction carriers, MMJailBench enables fine-grained analysis and attribution of vulnerabilities. Experiments reveal significant heterogeneity in jailbreak vulnerabilities across models, with prompt framing being the dominant factor. The benchmark al

News · 2026-08-26

ArXiv Proposes AdaVDR: Revolutionizing Video Deep Research with Adaptive Tool Invocation and Reflection

The ArXiv team introduces AdaVDR, a novel adaptive video deep research agent that addresses the challenges of inappropriate tool invocation, high error rates, and reasoning latency in existing video deep research systems. AdaVDR selects tools based on task requirements and its own capabilities, backtracking only when unreliable intermediate results necessitate correction. The research develops a video deep research data construction pipeline and constructs the VDR-EE benchmark covering entity-ce

News · 2026-08-26

ArXiv Introduces EgoArgus Benchmark: Assessing VLMs as Situational Assistants

The ArXiv team introduces EgoArgus, a benchmark dataset designed to evaluate Vision-Language Models (VLMs) as situational assistants in daily scenarios. The dataset focuses on five dialogue-video settings, testing the models' ability to arbitrate between visual evidence and user dialogue when they are helpful, irrelevant, or conflicting. Results show that current VLMs struggle with identifying trustworthy modalities and deciding when to intervene, highlighting the need for further advancements.

News · 2026-08-26

ArXiv Introduces Modality Maturity Index (MMI): A Benchmark for Evaluating Multimodal Capabilities of Large Language Mod

ArXiv introduces the Modality Maturity Index (MMI), a benchmark designed to evaluate the multimodal capabilities of large language models across five modalities (text, image, audio, video, and document) and combinations of up to three modalities in both inputs and outputs. The benchmark consists of 893 questions that assess a model's ability to understand multiple input modalities and generate responses in various output formats. Initial results show that current models struggle with multimodal

News · 2026-08-26

ArXiv Releases Comprehensive Review of Multimodal AI Research: Covering Vision-Language Models, Robotics, and Intelligen

ArXiv has recently published a series of multimodal AI research papers, spanning vision-language models, robotics, intelligent agents, and cross-modal reasoning. These studies introduce new frameworks to enhance robot manipulation robustness, optimize long video generation with memory modeling techniques, and propose novel benchmarks for evaluating medical vision-language models and intelligent agents in complex urban environments. The research highlights the significant potential of AI in cross

News · 2026-08-26

Hugging Face Releases RetrievalRouter: Intelligent Routing for Optimized Document Retrieval

Hugging Face has introduced RetrievalRouter, a lightweight query-aware routing tool designed to address the trade-off between accuracy and latency in modern document retrieval systems. By learning from query text, RetrievalRouter dynamically selects the most suitable retrieval pipeline, demonstrating superior performance across multiple benchmarks. Compared to the best static baseline, RetrievalRouter achieves a 2.5% increase in accuracy while being 12.4 times faster. Its innovation lies in expo

News · 2026-08-26

Hugging Face Releases VBVR-Pro: A Scalable Native Visual Reasoning Testbed

Hugging Face introduces VBVR-Pro, a closed-loop testbed for native visual reasoning that addresses bottlenecks in scalable training, reliable feedback, and controlled comparisons across generative substrates. VBVR-Pro features a controlled task space of 300 procedurally generated tasks and provides verifiable reward scorers for task-grounded evaluation. Its mechanism studies, spanning over 30 image, video, and interleaved generators, highlight the strengths of video generation for persistent spa

Info · 2026-08-26

Meta Releases Muse Image and Muse Video: AI-Powered Tools for Image and Video Generation

Meta has officially launched two AI-driven tools for content generation: Muse Image and Muse Video. Muse Image focuses on generating high-quality images based on user input, delivering realistic and creative visuals. Muse Video, on the other hand, specializes in video generation, enabling rapid transformation from text to video with support for various styles and customizable scenes. These tools mark Meta's continued expansion in AI-generated content, providing powerful resources for creative pr

News · 2026-08-26

Hugging Face Releases Training and Finetuning Techniques for Multi-Vector Embedding Models

Hugging Face has introduced a novel training and fine-tuning technique for multi-vector embedding models based on Sentence Transformers. This advancement aims to improve AI models' performance in handling complex semantics and cross-modal data by leveraging multi-vector embeddings to enhance semantic understanding and reasoning capabilities. The release provides developers with a powerful tool to tackle challenges in multimodal AI applications, including natural language processing, image recogn

Info · 2026-08-26

Zai Releases GLM-5.3-Flash: Multimodal AI Model with Breakthrough Performance

Zai has announced the release of GLM-5.3-Flash, a multimodal AI model capable of handling a 1 million token context window and achieving approximately 63% performance in the DeepSWE benchmark. Confirmed by Roman Chernin to be the mysterious Ox Alpha model, GLM-5.3-Flash will be available on the Nebius TF platform. This release signifies a significant advancement in AI's multimodal processing and long-context understanding capabilities, providing developers with a more powerful tool.

News · 2026-08-26

ArXiv Research: LLM Agents Perform Controlled Experiments with Simulation Models

ArXiv introduces a novel multi-agent framework that enables Large Language Models (LLMs) to conduct controlled experiments using scientific simulation models. This framework constructs structured task representations, designs experiments, executes comparative simulations, interprets outcomes, and synthesizes evidence-based recommendations for process parameter optimization. The research demonstrates improved specificity and user-rated accuracy in industrial applications, highlighting its potenti