Skip to main content
ZICQ

Topics Multimodal & Video Generation

Multimodal & Video Generation

Image, video, and speech generation plus production practice: Sora, Veo, Kling, Suno, and the open-source multimodal stack.

· Sep 15, 2026 #Multimodal AI#video generation#Multimodal#image generation#speech generation#voice cloning#Sora#Veo#Kling#Suno · 30 / 638 articles

News · 2026-09-07

LlamaIndex Launches Agentic Document Extraction: Revolutionizing Document Processing

LlamaIndex has introduced Agentic Document Extraction, a novel AI-driven approach that enhances document processing accuracy and automation by understanding document structure, inferring context, and verifying its own outputs. This technology leverages visual grounding and bounding boxes to address the limitations of traditional OCR systems, particularly in handling complex layouts and tables. It is especially effective for high-stakes applications like medical forms, where accuracy is critical.

News · 2026-09-07

Kimi Releases WorldVQA: A New Benchmark for Evaluating Atomic Visual Knowledge in Multimodal LLMs

Kimi has released WorldVQA, a novel benchmark designed to assess the factual correctness of Multimodal Large Language Models (MLLMs) in understanding visual world knowledge. The benchmark consists of 3,500 high-quality image-question pairs and challenges models to demonstrate accurate recognition of specific entities rather than relying on visual pattern-based hallucinations. Initial experiments reveal that even state-of-the-art models struggle with long-tail visual knowledge, often achieving ac

News · 2026-09-07

Kimi Launches Kimi K2.5: Pioneering Visual Agentic Intelligence

Kimi has launched Kimi K2.5, the latest open-source model that builds on extensive vision-text joint pre-training to deliver state-of-the-art capabilities in visual coding and agentic intelligence. The model introduces a novel Agent Swarm paradigm, enabling up to 100 sub-agents to execute parallel workflows, significantly reducing task completion time. Kimi K2.5 excels in coding, visual reasoning, and front-end development, showcasing groundbreaking advancements in visual-to-code generation and

Info · 2026-09-07

Alibaba Releases SAM-D2Q: A Novel Multimodal Doc2Query Framework for Enhanced E-commerce Search

Alibaba's research team has introduced SAM-D2Q, a multimodal Doc2Query framework designed to address vocabulary mismatch issues in e-commerce search. The framework enhances product title, image, and user query understanding through three stages: multimodal supervised fine-tuning, data augmentation, and reinforcement-learning-based preference alignment. Deployed in the AliExpress production system, SAM-D2Q has demonstrated significant improvements in business metrics, including a 3.38% increase i

Info · 2026-09-07

arXiv Introduces MURAL Framework: Breaking Barriers in Multimodal Recommendation Systems

arXiv introduces MURAL (Multimodal Uncertainty-aware Recommendation via Adaptive edge Learning), a novel framework addressing the limitations of current multimodal recommendation systems in terms of structural rigidity and semantic fragility. By leveraging adaptive edge learning and uncertainty-aware fusion, MURAL dynamically discovers latent item correlations and mitigates the impact of noisy modality signals. Experiments on large-scale benchmarks like TikTok and Amazon demonstrate that MURAL s

News · 2026-09-05

Hugging Face Releases Tri-PvP Benchmark: Exposing Modality Bias in Omni-Modal Large Language Models

Hugging Face's research team introduces Tri-PvP, a novel tri-modal conflict benchmark designed to expose modality bias in omni-modal large language models (OLLMs) when processing vision, audio, and text cross-modal conflicts. The benchmark consists of 8,000 samples, covering both perceptual and propositional forms of vision and audio. Evaluations of five OLLMs reveal a robust visual bias across most models and evidence-type conditions, along with a systematic asymmetry in evidence-form bias. Thi

Info · 2026-09-05

Microsoft Releases MAI-Image 2.6: Pushing the Frontier of Image Generation Quality and Cost Efficiency

Microsoft has released MAI-Image 2.6, a new version of its AI image generation platform that achieves a significant breakthrough in balancing image quality and cost efficiency. By optimizing the generation algorithms and resource scheduling mechanisms, MAI-Image 2.6 maintains high-quality image output while significantly reducing computational resource consumption. This advancement enhances the economic viability of AI image generation tasks and provides more efficient technical support for larg

News · 2026-09-05

CONFLICTGUI and CONFLICTGUARD: A New Framework for Enhancing Conflict-Aware Termination in GUI Agents

This research introduces CONFLICTGUI, a benchmark for studying conflict-aware termination in GUI agents when faced with infeasible instructions, and proposes CONFLICTGUARD, an inference-time framework that aligns an agent's feasibility awareness with its action generation. Experiments demonstrate that CONFLICTGUARD significantly improves the average success rate of conflict tasks while preserving normal GUI-task performance. This work provides a novel technical approach for optimizing agent beha

Info · 2026-09-04

Ling-3.0-flash-VL Released: AI Model with Integrated Visual Understanding and Agent Capabilities

Ling-3.0-flash-VL is a multimodal AI model built on Ling-3.0-flash, featuring capabilities in visual perception, STEM reasoning, document intelligence, multimodal agent tasks, frontend coding, and medical report interpretation. The model excels in tasks that require the integration of vision and language, providing developers with a powerful tool for a wide range of applications.

News · 2026-09-04

AWS Releases Multimodal WhatsApp Ordering Assistant Built on Amazon Bedrock AgentCore

AWS has launched a multimodal WhatsApp ordering assistant built on Amazon Bedrock AgentCore, enabling businesses to handle customer orders via text, voice notes, and real-time voice calls on a single business number. The solution leverages Amazon Nova 2 Lite for text processing, Amazon Nova 2 Sonic for voice interactions, and a shared memory system for cross-channel customer recognition. With an asynchronous processing mechanism, the system ensures quick responses and efficient order management,

Info · 2026-09-04

Wan 3.0 Released: 30-Second Native-Audio Video Generation from Text, Images, Docs, or Web

Wan 3.0 is an innovative AI tool that can convert text, images, documents, or web content into multimodal videos with native audio in just 30 seconds. This tool supports various input formats and leverages AI technology to automatically generate high-quality audiovisual content, offering content creators a more efficient and convenient way to produce multimedia materials. The release of Wan 3.0 marks a significant advancement in AI-driven multimedia content generation, providing users with a nov

News · 2026-09-03

FactoSR Framework Revolutionizes VLM Reasoning: Advances in 3D and 4D Spatial Understanding

FactoSR is a novel factorized reinforcement learning framework designed to address the spatial reasoning bottleneck in Vision-Language Models (VLMs). By decomposing world-consistent reasoning into three orthogonal geometric sub-objectives—planar correspondence (XY), depth consistency (Z), and temporal reversibility (T)—FactoSR optimizes high-dimensional reasoning capabilities. Experimental results demonstrate substantial performance improvements, with a 5.9% boost on VSI-Bench and a 4.5% increas

News · 2026-09-03

Hugging Face Publishes Study on Cross-Modal Attention in Audio-Video Models: Unveiling Semantic Leakage Issues

Hugging Face's research team delves into the cross-modal attention mechanisms within audio-video diffusion models, introducing the concept of the 'Attention Triangle' to analyze interactions between text, audio, and video streams. The study reveals that bidirectional influence between audio and video can lead to semantic leakage, where models may deviate from intended outputs and generate visually plausible but semantically incorrect results when faced with complex prompts. The researchers propo

News · 2026-09-03

Scal3R: Introducing Efficient Multi-Relative Pose Querying for Scalable Online 3D Reconstruction

Scal3R is an innovative online 3D reconstruction approach that addresses the performance degradation caused by global pose drift in long videos by reformulating the problem as multi-reference relative pose querying. It leverages lightweight learnable tokens injected into a frozen backbone via asymmetric attention and incorporates an online pose-graph optimization system with loop closure to suppress long-range drift. Experiments show that Scal3R reduces the average Absolute Trajectory Error (ATE

Info · 2026-09-03

IFM.AI Releases K2 Horizon: A Frontier-Performance Open Source AI Model

On September 3, 2026, IFM.AI unveiled its latest AI model, K2 Horizon, which stands out for its frontier performance and fully open-source nature. Designed to empower developers with robust AI capabilities, K2 Horizon supports a wide range of applications, from natural language processing to complex reasoning tasks. Its open-source nature not only lowers the barrier to entry for AI technology but also fosters a collaborative innovation platform for the global developer community.

Info · 2026-09-03

Google Releases Lyria 3.5: AI Music Generation Model Enters Public Preview

Google has launched the next generation AI music generation model, Lyria 3.5, and announced its entry into the public preview stage. This model focuses on the music creation domain, aiming to help users generate high-quality, creative music pieces through advanced AI technology. The release of Lyria 3.5 marks a further breakthrough in AI music generation, providing stronger tool support for music creators and demonstrating Google's continuous innovation in AI art creation.

Info · 2026-09-03

Microsoft Releases MAI-Transcribe-2: The World's Fastest, Most Accurate, and Most Affordable Speech Recognition Model

Microsoft has unveiled MAI-Transcribe-2, a next-generation speech recognition model that sets new benchmarks in speed, accuracy, and affordability. Leveraging advanced algorithmic optimizations and hardware acceleration, MAI-Transcribe-2 achieves industry-leading processing speeds while maintaining high recognition accuracy. Additionally, the model incorporates innovative cost-control strategies, making it highly competitive in terms of pricing. This release represents a significant advancement

Info · 2026-09-03

Kuben Developer Releases TikTok Videos Dataset: 4.5 Billion Posts with Engagement Metrics

Kuben Developer has released a new dataset on Hugging Face titled 'TikTok Videos: 4.5B Posts with Engagement Metrics.' This dataset includes over 4.5 billion TikTok videos and their engagement metrics, offering researchers and developers valuable resources for analyzing user behavior, trend prediction, and content recommendation in AI-related tasks. This release provides significant support for social media analysis and multimodal AI research.

Info · 2026-09-03

Google Releases Gemini 3.8: New AI Model Sparks Attention and Controversy

Google has officially released the latest AI model, Gemini 3.8, which aims to enhance multimodal processing capabilities and cross-task reasoning performance. However, its release has been met with immediate mockery and skepticism from competitors, primarily due to perceived discrepancies between its claimed capabilities and actual performance. Despite this, the release of Gemini 3.8 underscores Google's ongoing innovation and competitive drive in the AI space, offering new tools for developers.

News · 2026-09-03

AVERT: Novel Audio-Verified Approach for Spoken Dialogue State Tracking Released

arXiv has released a new study on spoken dialogue state tracking, introducing AVERT, a novel method that scores candidate values by combining cross-turn agreement with an audio-conditioned verifier. AVERT resolves inconsistencies, omissions, and unsupported audio errors using three operators: vote, add, and swap. On the SpokenWOZ benchmark, AVERT improved the base model's JGA from 33.04% to 40.13% without retraining, demonstrating its potential in handling ASR errors and enhancing dialogue state

Info · 2026-09-03

H3Max Released: Fast Audio-Visual Generation and Restoration with Video Model

H3Max is a novel video model capable of rapidly generating video clips with synchronized audio based on user descriptions. Leveraging advanced video generation algorithms and audio processing techniques, H3Max can convert text descriptions into audio-visual content within seconds. This breakthrough in video generation offers powerful tools for content creators and developers, particularly excelling in the rapid creation of high-quality video content.

Info · 2026-09-02

Developer Releases 5.94 Billion TikTok Videos Dataset for Free on Hugging Face

A developer has released a dataset containing 5.94 billion TikTok videos, along with user profiles, comments, hashtags, and audio data, on Hugging Face. The dataset was collected using a reverse-engineering method, and while the data is publicly accessible, accessing it in this manner may violate TikTok's terms of service. The release includes a step-by-step tutorial and code, providing AI researchers with a valuable resource for analysis and model training.

Info · 2026-09-02

Google Cloud Releases Antigravity Image Generation Skill: A New Breakthrough in AI Agent Applications

Google Cloud has recently released the image generation skill for its Antigravity agent, marking a significant advancement in AI agents' ability to handle multimodal tasks. This skill empowers agents to generate high-quality images and achieve smarter interactions by understanding context, providing developers with more powerful tools. The release not only enhances the performance of agents in complex tasks but also opens new possibilities for the application of image generation technology in th

News · 2026-09-02

UI-Venus-2 Released: A General-Purpose GUI Agent Framework for Cross-Platform Task Automation

UI-Venus-2 is a general-purpose foundation GUI agent framework designed for multimodal task automation, addressing the challenges of transitioning from benchmark models to real-world applications. It enhances environment coverage (supporting over 170 multilingual mobile apps and desktop operating systems), optimizes task construction (through function-grounded instruction generation), and improves verification mechanisms (using visual keypoints and multi-model voting) to ensure reliability and g

News · 2026-09-01

Hugging Face Releases Enoki: A Multi-Level Hallucination Detection Framework for Enhanced LLM Factuality

Hugging Face has released Enoki, an open-source Open Information Extraction framework for multi-level hallucination detection in large language models (LLMs). Enoki addresses the challenge of factuality in high-stakes scenarios by extracting text-anchored relational facts, verifying them against evidence, and projecting unsupported facts back to hallucinated spans. This shared representation enables claim-level verification and span-level localization without requiring separate alignment. Enoki

News · 2026-09-01

Google Unveils August AI Updates: Focus on Multimodal Agents and Cybersecurity

In August 2026, Google unveiled a series of AI updates spanning multimodal agents, document processing, and cybersecurity. Key highlights include the Agent Swarm system, which supports up to 100 sub-agents working in parallel for enhanced task efficiency, and Agentic Document Processing, an AI-driven approach for precise document understanding and automation. Additionally, the Fairwind program offers advanced cybersecurity tools to governments and trusted partners, underscoring Google's commitme

News · 2026-09-01

Pixel Linguist II Released: Breakthrough in Pixel Text Representation Learning

Pixel Linguist II is a novel vision encoder for pixel text representation learning, integrating principles such as variable resolution rendering, natural image-text pairs, layout-aware rendering, and a two-stage multilingual curriculum. It sets new state-of-the-art results on English, cross-lingual, and multilingual Visual STS and ViDoRe benchmarks, demonstrating strong robustness under 80% visual token compression. This model holds significant promise for optical context compression and enhance

News · 2026-09-01

Google DeepMind Introduces Agentic Video Understanding in Gemini

Google DeepMind has introduced agentic video understanding capabilities in its Gemini model, enhancing AI's ability to comprehend and process complex video content. This advancement enables AI to intelligently identify key information, actions, and contextual relationships within videos, leading to more accurate results in applications such as video analysis, automatic captioning, and content recommendation. The release highlights AI's progress in multimedia processing, providing more powerful t

News · 2026-09-01

GMA Benchmark Released: Evaluating General Mobile Assistants in Real-World Scenarios

The GMA (General Mobile Assistants) benchmark has been released to evaluate multimodal AI agents in challenging real-world scenarios. Built on seven open-source projects spanning domains like lifestyle sharing and travel planning, GMA introduces 300 tasks across four difficulty tiers, from atomic actions to complex multi-step workflows. Evaluation of eight frontier models reveals a substantial decline in performance as task complexity increases, highlighting the gap between current agents and re

News · 2026-09-01

Parametric Multimodal User Memory: A Breakthrough in AI Agent Perception

This research introduces a novel model called 'Parametric Multimodal User Memory' to address the limitations of AI agents in processing user information. Traditional methods rely on textual descriptions, which struggle to capture non-textual information such as voice, facial features, and other perceptual cues. By integrating a vision-language model (VLM) and a dedicated encoder, the study decomposes user identity into two subproblems: context grounding and identity key extraction. This approach