Skip to main content
ZICQ

Topics Multimodal & Video Generation

Multimodal & Video Generation

Image, video, and speech generation plus production practice: Sora, Veo, Kling, Suno, and the open-source multimodal stack.

· Sep 15, 2026 #Multimodal AI#video generation#Multimodal#image generation#speech generation#voice cloning#Sora#Veo#Kling#Suno · 30 / 638 articles

News · 2026-09-29

Qwen-Image Adopts IDSpect: Enhancing Structural Quality and Semantic Alignment in Chinese Text Rendering

The Qwen-Image team introduces IDSpect, a novel approach to address challenges in Chinese text rendering by enhancing structural quality and semantic alignment. Traditional OCR-based reinforcement learning rewards overlook the compositional nature of Chinese characters, leading to coarse feedback. IDSpect employs Ideographic Description Sequences (IDS) to decompose target text into IDS tokens and aligns visual predictions with the target sequence, providing fine-grained rewards for components an

News · 2026-09-29

PRISM Framework Released: Enabling Humanoid Loco-Manipulation via Counterfactual Video Generation

PRISM is a novel real-to-sim-to-real framework that addresses the data scarcity problem in humanoid robot loco-manipulation skill learning by generating a large, diverse set of counterfactual human-object interaction videos from a handful of real videos. The framework reconstructs human and object motions into physically plausible trajectories, enabling the training of a policy that generalizes to unseen scenarios. Experiments demonstrate that the PRISM framework successfully enables robots to p

News · 2026-09-29

CompVis Introduces Improved Distributional Diffusion Models: Breakthrough in Image Generation Efficiency and Quality

CompVis introduces an improved Distributional Diffusion Model (iDDM) that addresses key limitations in scaling diffusion models for modern image generation. By deferring particle expansion to later transformer layers and introducing time-dependent scoring rule schedules, the model achieves state-of-the-art performance on ImageNet-256² with 4.48 FID at 4 steps and 2.38 FID at 50 steps using DiT-XL/2. This advancement eliminates the need for teacher models, self-distillation, or Jacobian-vector pr

News · 2026-09-29

Hugging Face Releases SoL-Refiner: One-Step Video Refinement for Efficient 4K Video Generation

Hugging Face has introduced SoL-Refiner, a novel technology for efficient high-resolution video generation. This one-step denoising process transforms low-resolution video outputs into 4K videos, addressing the secondary sampling bottleneck in conventional multi-step refinement methods. SoL-Refiner combines high-resolution continual training, reinforcement learning post-training, and one-step distillation. It outperforms all external refiners at 2K resolution on the Refiner-Bench benchmark and s

News · 2026-09-29

Hugging Face Introduces SplitMoE: Breaking the Uniformity Trap in Video Diffusion Models

Hugging Face's research team introduces SplitMoE, a novel sparse architecture designed to address the uniformity trap in traditional vision generative models using Mixture-of-Experts (MoE). By explicitly bifurcating the expert pool into semantic and generic experts and leveraging prototype-guided routing and pull-push regularization, SplitMoE enables tokens to cluster naturally by semantic attributes. This results in superior video generation quality, faster speed of convergence, and routing coh

News · 2026-09-29

Hugging Face Releases LIFT: Layout-In-Future Video Generation with Large Viewpoint Changes

Hugging Face has introduced LIFT, a novel image-to-video generation framework that enhances camera control with Layout-In-Future control, allowing users to specify the content and spatial layout of future views. LIFT leverages the last-frame layout as an explicit control signal and employs On-Policy Self-Distillation (OPSD) to address the challenge of learning from sparse layout guidance. Experiments demonstrate that LIFT outperforms existing methods in video quality, future layout controllabili

News · 2026-09-29

Hugging Face Proposes Adversarial Training to Enhance Pixel Diffusion Models

Hugging Face has introduced an adversarial training approach for pixel diffusion models, which adds an adversarial loss to the predicted output at non-high-noise timesteps. This method enhances distribution fidelity, coverage, prompt alignment, and perceptual quality without altering the model architecture or sampling procedure. It is the first systematic study of adversarial post-training for pixel diffusion, demonstrating that the approach restores missing high-frequency content in natural ima

News · 2026-09-29

Hugging Face Releases LongLive-Plug: A Universal Distillation Framework for Video Generation

Hugging Face has introduced LongLive-Plug, a universal distillation framework for video generation tasks that enables training-free, plug-and-play deployment across multiple downstream models through a single training process. Leveraging LoRA technology, the framework learns reusable capabilities such as single-pass classifier-free guidance, few-step sampling, and long-context error correction for autoregressive generation on a base model. The adapters remain reusable even when downstream models

Info · 2026-09-29

Qwen-family LLMs Quietly Become the Backbone of Modern Audio Models

An analysis of audio.cpp reveals that Qwen-family LLMs have become the predominant architecture in the audio domain, with 32 audio model families adopting Qwen and 20 specifically using Qwen3 LLM. This trend extends beyond text-to-speech (TTS) to encompass speech synthesis, ASR/audio understanding, music generation, speech-to-speech, and audio/video models, highlighting Qwen's versatility and dominance in multi-modal AI tasks.

Info · 2026-09-29

Anthropic Releases GLM-5.3: Exploring Advanced Cyber Capabilities and AI Safety Challenges

Anthropic has released its latest large language model, GLM-5.3, and examined the spread of advanced cyber capabilities within AI systems and their potential implications. GLM-5.3 demonstrates significant improvements in complex task handling and decision-making abilities. However, its advanced capabilities also raise concerns about AI safety and potential misuse. This research underscores the importance of balancing technological advancements with security risks, providing new insights for deve

News · 2026-09-29

AWS Launches Multimodal Video Discovery Solution on Amazon Bedrock, Revolutionizing Content Retrieval for Condé Nast

AWS, in collaboration with Condé Nast, has developed a multimodal video discovery solution powered by Amazon Bedrock and Amazon OpenSearch Service. This solution leverages semantic search and multimodal understanding across visual, audio, and text to reduce video content retrieval time from an average of 250 minutes to under 2 minutes, significantly enhancing asset discoverability and achieving an estimated $800,000 in annual operational savings.

News · 2026-09-28

Hugging Face Releases QuantWM: Revolutionizing KV Cache Quantization for Video World Models

Hugging Face has introduced QuantWM, a novel 2-bit KV cache quantization framework designed for video world models. This framework employs two complementary techniques—Quantization-Sensitivity-Aware Clustering (QSAC) and Principal-Subspace Attention Compensation (PSAC)—to mitigate attention shifts during quantization, significantly improving the visual quality and temporal consistency of video generation. Experimental results demonstrate that QuantWM achieves up to 6.20x KV cache memory compress

News · 2026-09-28

Hugging Face Releases Persistence Forcing: Revolutionizing Feature Optimization in Pixel-Space Diffusion Models

Hugging Face introduces Persistence Forcing (PerF), a novel method for optimizing feature refinement in pixel-space diffusion models. PerF employs heterogeneous refinement, assigning distinct refinement budgets to different feature groups, resulting in ordered feature specialization where sparsely refined features encode global visual structure and frequently refined features focus on localized, high-frequency details. On ImageNet 256x256, PerF-L achieves an FID of 1.91, approaching JiT-H's 1.86

News · 2026-09-28

AWS Releases vLLM-Omni DLC: Enabling Real-Time Bidirectional Streaming for Multimodal AI Models

AWS has released the vLLM-Omni Deep Learning Container (DLC), a solution that enables real-time bidirectional streaming for multimodal AI models. Built on the vLLM framework, this DLC supports the generation, processing, and streaming of text, audio, images, and video. By integrating with Amazon SageMaker AI, users can deploy models like Qwen3-TTS to achieve real-time speech generation and bidirectional audio streaming. This technology opens new avenues for building real-time voice applications,

Info · 2026-09-28

SeedRouter Launches Unified API for LLM, Image, and Video Models

SeedRouter has launched the SeedRouter API, a unified interface designed to simplify the use of Large Language Models (LLMs), image, and video models. By abstracting the complexities of different models, this API offers developers a one-stop solution for integration, thereby enhancing development efficiency and reducing technical barriers. This innovation is expected to accelerate the deployment of AI applications in multimodal scenarios and provide developers with more convenient tools.

Info · 2026-09-28

Google Cloud Releases Guide on Building JEV-style Classifier with DiffusionGemma and vLLM

Google Cloud has released a comprehensive tutorial on building a JEV-style classifier using DiffusionGemma and vLLM. DiffusionGemma is a diffusion-based generative model, while vLLM stands for efficient variational Large Language Models. This guide aims to assist developers in combining these cutting-edge AI technologies to create sophisticated classification systems, enhancing AI's capabilities in multimodal data processing and classification tasks.

News · 2026-09-27

Hugging Face Releases StoryEngine: A Novel Framework for Coherent Video Storytelling

Hugging Face has released StoryEngine, a novel framework for agentic video storytelling that addresses the challenges of coherence and consistency in long-form video generation. By separating authoritative semantic plans from unreliable visual observations, StoryEngine maintains structured representations of entity states and propagates event-induced changes to ensure accurate visual realization. The framework incorporates canonical references, executable render plans, and a bounded evaluation-g

Info · 2026-09-27

MiMo-v2.6-Flash Released: Exploring the Intelligence/Price Pareto Frontier

MiMo-v2.6-Flash, the latest release in the MiMo model series, focuses on achieving an optimal balance between intelligence and affordability. By optimizing the model architecture and training methods, this version significantly enhances the decision-making capabilities and efficiency of the agent while maintaining low computational costs. This release demonstrates the potential of AI in various applications, providing developers with more flexible and efficient tools, particularly in resource-co

Info · 2026-09-27

Kimi K3 vs Claude Opus 5.5: A Comparative Analysis of Two Flagship LLMs in Building Pokémon Emerald

This article provides a detailed comparison between two flagship large language models, Kimi K3 and Claude Opus 5.5, focusing on their performance in building the Pokémon Emerald game. By evaluating their code generation, logical reasoning capabilities, and efficiency in multimodal tasks, the analysis highlights the strengths and weaknesses of each model in handling complex tasks. This study offers valuable insights for AI developers in model selection and showcases the innovative applications o

Info · 2026-09-27

Qwen 27B Demonstrates Local AI Capabilities: High-Quality Motion Graphics Generation on 4090 GPU

A Reddit user demonstrated the capabilities of Qwen 27B in generating motion graphics locally using a 4090 GPU. By analyzing Opus 5.5-generated motion graphic videos, Qwen 27B successfully replicated and enhanced the effects, showcasing its strong performance in multimodal tasks. This achievement highlights the potential of Qwen 27B in creative design applications and underscores the growing capabilities of local AI systems.

Info · 2026-09-27

Peekaboolean: Vision-Language Model for Image and Typed Question Answering Released

Peekaboolean, developed by Bykof, is a vision-language model designed to answer typed questions based on images. It incorporates mechanisms similar to Jev, including choice selection, scoring, and reasoning capabilities. The model’s technical highlights include efficient processing of combined image and text inputs and low-latency responses across various hardware environments, such as 400 milliseconds for six questions on an M1 Pro and about 60 milliseconds on a desktop GPU. The release of Peek

Info · 2026-09-27

PrivateMode Proposes Transforming GLM-5.3-Flash into a Jev-like System One Model

PrivateMode has proposed a method to transform the GLM-5.3-Flash model into a Jev-like System One model, aiming to enhance AI systems' capabilities in rapid decision-making and intuitive responses. This approach leverages System 1 thinking, using types instead of text as output responses to mimic human quick and intuitive decision-making, thereby reducing the complex reasoning delays of traditional text generation models. This research offers a new technical path for AI agents in real-time decis

News · 2026-09-26

Hugging Face Releases CUA-SWE: A New Benchmark for Integrating Computer-Use Agents with Software Engineering

Hugging Face has introduced CUA-SWE, a benchmark, environment, and evaluation pipeline designed to integrate computer-use agents with software engineering tasks. CUA-SWE addresses the gap in current research by simulating real-world development scenarios that involve GUI feedback, code modification, command execution, and visual feedback inspection. The benchmark spans four software engineering domains and includes task-specific deterministic tests to verify the correctness and behavioral consis

News · 2026-09-26

Hugging Face Releases VoxPolyMem: A Novel Interaction-Aware Multimodal Memory Framework for Multi-Party Conversations

Hugging Face has released VoxPolyMem, a novel multimodal memory framework designed for multi-party spoken conversations. VoxPolyMem addresses the limitations of existing research in long-term memory for multimodal interactions by integrating incremental speaker identification with a hierarchical memory structure (interaction memory, fact memory, and participant profiles). It employs an evidence-based sequential decision-making approach for retrieval. Experimental results on VoxPolyBench demonstr

News · 2026-09-26

Hugging Face Releases VoxMem: A Benchmark for Multimodal Memory in Conversational AI

Hugging Face has introduced VoxMem, a novel benchmark framework designed to evaluate the multimodal memory capabilities of large language models (LALMs). VoxMem assesses models across 34,743 spoken sessions (177 hours) spanning four acoustic evidence types—speech semantics, speaker identity, paralinguistic cues, and environmental sounds—and four memory operations—information extraction, multi-session reasoning, temporal tracking, and answer refusal. Evaluation of 15 LALMs shows that no model exc

Info · 2026-09-26

InternLM Releases Intern-Decision Series: A Leap in Multimodal Structured Decision-Making

InternLM has launched the Intern-Decision series, comprising 4B and 0.8B models. Fine-tuned from Qwen3.5-4B, this multimodal structured decision model accepts shared states, question schemas, and optional images, returning answer distributions for each question in a single forward pass. Its key innovation lies in structured candidate scoring, which streamlines decision-making by avoiding free-form text sampling. This model offers a novel approach to AI agent decision-making in complex tasks, par

Info · 2026-09-26

DeepSeek Releases DSec: Revolutionizing AI Infrastructure with Elastic Computing Architecture

DeepSeek has introduced the DeepSeek Elastic Compute (DSec), an innovative elastic computing architecture designed to optimize resource allocation and task scheduling for AI models. By dynamically adjusting computing resources to match the varying demands of different AI tasks, DSec significantly enhances the operational efficiency and resource utilization of AI systems. This release marks a significant advancement in elastic computing for AI infrastructure, providing more efficient and flexible

Info · 2026-09-26

Drawgent Released: Claude AI Assistant Integrated into Excalidraw

Drawgent is a new tool that integrates Anthropic's Claude AI assistant into the Excalidraw drawing platform. This integration allows users to leverage Claude's capabilities in real-time while creating drawings, such as generating text, suggesting design elements, or optimizing existing graphics, thereby enhancing productivity and creative output. The release of Drawgent represents a significant step in the fusion of AI with creative tools, providing designers and developers with smarter collabor

News · 2026-09-25

Hugging Face Introduces Diffusion Model-Based DSM Refinement for Enhanced 3D Map Accuracy

Hugging Face researchers have developed a novel diffusion model-based approach for refining Digital Surface Models (DSMs) by conditioning on both photogrammetric DSMs and Pléiades imagery. This method, built on a modified Stable Diffusion 3 architecture with a pruned text stream and patch-wise normalization, enables stable training on LiDAR data and transfers effectively from natural images to elevation maps. Experiments in French cities demonstrate significant improvements in elevation accuracy

News · 2026-09-25

Hugging Face Introduces FuseReg: A Novel Approach to Mitigate Reconstruction-Generation Gap in Representation Autoencode

Hugging Face introduces FuseReg, a novel regularization technique for representation autoencoders (RAEs) that replaces heuristic feature selection with training over random subsets of encoder layers. This approach mitigates the reconstruction-generation gap by penalizing sensitivity to cross-layer disagreement. Experiments on ImageNet-256 with DINOv3-L demonstrate that a single FuseReg decoder achieves higher PSNR and reduces unguided gFID by 27% without retraining the generator. When extended t