Skip to main content
ZICQ

Topics Multimodal & Video Generation

Multimodal & Video Generation

Image, video, and speech generation plus production practice: Sora, Veo, Kling, Suno, and the open-source multimodal stack.

· Sep 15, 2026 #Multimodal AI#video generation#Multimodal#image generation#speech generation#voice cloning#Sora#Veo#Kling#Suno · 30 / 638 articles

News · 2026-10-16

Hugging Face Releases MOSS-Image: Open-Source Text-to-Image Model with High-Fidelity Layout Control and Bilingual Suppor

Hugging Face has released MOSS-Image, an open-source text-to-image model featuring a 5B-parameter single-stream diffusion transformer (DiT) for high-fidelity visual synthesis and intent-aware layout control. The model supports bilingual long-form text rendering and introduces an intention-aware prompt enhancement (PE) module, significantly improving its ability to handle complex text and layouts. Evaluations on LongLongTextBench (LLTB) and RealLayoutArena (RLA) demonstrate that MOSS-Image achiev

News · 2026-10-16

Seerray Lab Releases Xiaomi-Lens-0: Enhancing Region Understanding for Multimodal Large Language Models

Seerray Lab has introduced Xiaomi-Lens-0, a 4B multimodal large language model (MLLM) built on Qwen3.5, designed to enhance fine-grained region understanding in images. By predicting masks from its own visual representations, the model leverages a lightweight decoder to combine vision-encoder patch features with the language model's mask-token states, eliminating the need for an additional segmentation encoder. Experimental results demonstrate that Xiaomi-Lens-0 outperforms Qwen3.5 by an average

Info · 2026-10-16

OpenMOSS Releases MOSS-Image: A New Open-Source Image Generation Model

OpenMOSS has released MOSS-Image, a new image generation model licensed under Apache. Compared to existing models like Qwen Image 2.1 and Flux Klein 3, MOSS-Image offers unique advantages in performance and application scenarios. Its open-source nature provides AI developers with more options, particularly in image generation and visual task processing.

News · 2026-10-16

Elorian Releases Glimpse: AI Model Surpasses GPT-6 Astra in Visual Reasoning Tasks

Elorian has released its first AI model, Glimpse, which outperforms GPT-6 Astra in scientific chart reasoning and short-form video captioning tasks. Additionally, Glimpse demonstrates competitive performance in other visual reasoning tasks. This release signifies a significant advancement in AI's multimodal reasoning capabilities, offering more efficient tools for scientific research and video content generation.

News · 2026-10-16

Reflection Releases Beam: Open-Weight Model for Software Engineering and Long-Running Agentic Tasks

Reflection has released its first open-weight model, Beam, designed for software engineering, reasoning, and long-running agentic tasks. The model is now freely available on Ollama for a limited time. Beam aims to empower AI developers with enhanced capabilities in handling long-context understanding and multi-step reasoning tasks, showcasing its potential in complex application scenarios.

News · 2026-10-16

Hugging Face Releases MeddeX: A Medical Report Analysis System Combining LLaMA-3 and Gemini Vision

Hugging Face has released MeddeX, a medical report analysis system that integrates a QLoRA fine-tuned LLaMA-3 (8B) model, Retrieval-Augmented Generation (RAG), and a multimodal Gemini Vision layer to deliver accurate, concise, and evidence-backed summaries of medical documents. MeddeX can handle text-native PDFs, scanned image-based PDFs, and direct radiological images through an intelligent hybrid extraction pipeline with a novel garbage-text detection mechanism. The system also features report

News · 2026-10-15

Hugging Face Releases ACEdit: Breakthrough in Stable Multi-Turn Image Editing

Hugging Face has released ACEdit, a novel model addressing the degradation problem in multi-turn image editing caused by distribution shifts. The model employs an asymmetric conditioning scheme and a self-forcing training approach, leveraging a teacher model to provide reliable training signals when the student model operates on its own previous outputs. Experiments demonstrate that ACEdit significantly reduces quality degradation over long editing sequences while maintaining editing quality and

News · 2026-10-15

Hugging Face Releases OmniXRef: A Unified Framework for Multimodal Multi-Reference Video Editing

Hugging Face introduces OmniXRef, a unified framework for multimodal multi-reference video editing that allows users to combine text, image, and video references to control diverse attributes of a source video. The model leverages a single-stream diffusion transformer to process the source video and multimodal context from an MLLM in an interleaved sequence. To address data scarcity, OmniXRef-Data is constructed using synthesis pipelines, and OmniXRef-Bench is introduced for systematic evaluatio

Info · 2026-10-15

IBM Releases Granite Switch 4.1: Modular LLM for Multi-Modal Task Processing

IBM has released Granite Switch 4.1, a modular Large Language Model (LLM) designed to handle natural language conversations, decision-making, and speech-to-text tasks. The model leverages an adapter mechanism for dynamic feature activation via control tokens, enabling flexible task handling. Built on a 30B-parameter base model, it integrates speech recognition capabilities and is compatible with HuggingFace Transformers and vLLM backends. The open-source version is licensed under Apache 2.0, pro

News · 2026-10-15

Gen4P Framework Released: Revolutionizing 4D Perception from Monocular Videos

Gen4P is a novel framework for recovering 4D geometry and motion from monocular videos. It repurposes a pretrained video generator into a feed-forward 4D perceiver using a lightweight cross-attention decoder, eliminating the need for iterative denoising. The framework introduces on-demand querying and continuous-time queries for flexible spatial and temporal sampling and enhances geometric knowledge by integrating a pretrained video-depth LoRA as a frozen geometry prior. Gen4P achieves state-of-

News · 2026-10-15

Hugging Face Releases EVA-Cap: Optimizing Audiovisual Video Captioning with Event-Centric Alignment

Hugging Face has released EVA-Cap, a novel framework for audiovisual video captioning that decomposes captions into atomic audiovisual events and aligns them in an event-centric manner. Trained using Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) on the curated EVA-Data dataset, EVA-Cap focuses on global event completeness, intra-event attribution, and inter-event synchronization. Experiments demonstrate significant improvements in event-centric evaluations (e.g., +9.

News · 2026-10-15

Hugging Face Releases Velox: Region-Level Attention Routing Boosts Video Generation Efficiency with Geometric Data

Hugging Face has introduced Velox, an innovative model designed to address the efficiency bottleneck in video generation caused by the growing KV cache. Velox employs a region-level attention routing technique based on geometric data, significantly enhancing the speed and efficiency of video generation. Its core mechanism involves representing historical blocks as camera frustum slabs with depth intervals and obtaining current-block depth through recency-priority multiwarp. This approach efficie

News · 2026-10-15

Hugging Face Releases MaLoW: A Novel Lightweight Memory Framework for Streaming Video Understanding

Hugging Face has introduced MaLoW, a novel streaming memory framework designed to enhance the long-term memory capabilities of streaming vision-language models (VLMs). Unlike traditional mechanisms that compress, retrieve, or cache historical observations, MaLoW internalizes historical context into evolving lightweight LoRA weight updates, allowing it to maintain current inputs while efficiently integrating historical information. Experimental results demonstrate MaLoW's strong performance acros

News · 2026-10-15

Hugging Face Releases Intern-Decision: Revolutionizing Multimodal Decision-Making for Intelligent Agents

Hugging Face has introduced the Intern-Decision family of models (0.8B, 2B, and 4B parameters), designed to enhance decision-making efficiency for intelligent agents in multimodal environments. By converting free-form text responses into structured choices and candidate probabilities through autoregressive masked language modeling, the model reduces latency and simplifies the interaction between the model and its environment. The 4B parameter model achieves a 90.02% average accuracy across seven

News · 2026-10-15

Hugging Face Introduces Speculative Forcing: Revolutionizing Autoregressive Video Generation Efficiency

Hugging Face has introduced Speculative Forcing, a novel framework designed to enhance the efficiency of autoregressive video generation. This method predicts when intermediate clean estimates can be safely accepted as final outputs, avoiding unnecessary computations while maintaining generation quality. In multiple benchmarks, Speculative Forcing demonstrated significant improvements in inference speed, achieving up to 1.5x faster performance across short-video, long-video, and world generation

News · 2026-10-15

Graph-Based Orchestration of Heterogeneous AI Models: A Control Node Approach to Composite Intelligence

This research introduces a graph-based framework for orchestrating composite AI systems that integrate multiple AI techniques, such as planning, machine learning, constraint programming, and large language models. The framework uses specialized control nodes and employs directed acyclic graphs and behavior trees to manage model relationships and execution sequencing. It encompasses data connectors, computational services, and a containerized deployment framework to support hierarchical, sequenti

Info · 2026-10-15

NVIDIA Releases Nemotron-3.5 Super-VL: 120B Parameter Vision-Language Model with NVFP4 and LiLiCorr Decoding

NVIDIA has officially released the Nemotron-3.5 Super-VL model family, featuring a 120 billion parameter scale. The release includes the base version, NVFP4-optimized version, BF16 precision version, and the novel LiLiCorr speculative decoding technique. The LiLiCorr decoding technology addresses latency issues in long-sequence generation, significantly enhancing inference efficiency. Nemotron-3.5 Super-VL supports multimodal inputs (text, images, videos) and is optimized for complex vision-lang

News · 2026-10-15

Meta Releases Muse Spark Open: A Multi-Modal AI Coding Model with 1M Token Context Window

Meta has officially released Muse Spark Open, a multi-modal mixture of experts (MoE) model optimized for coding and agentic tasks. With a total parameter count of 816 billion and 4.2 billion active parameters, the model supports a context window of up to 1 million tokens and can process text, images, videos, and audio. Muse Spark Open excels in improving first-try accuracy, tool-calling capabilities, and large codebase understanding for programming tasks. It is now available on the Together AI p

News · 2026-10-15

Hugging Face Releases Granite Switch 4.1: A Multimodal AI Platform Integrating Chat, Decision-Making, and Speech

Hugging Face has officially released Granite Switch 4.1, a multimodal AI platform that integrates chat, decision-making, and speech functionalities. The platform aims to provide developers with a unified interface to streamline the development of complex AI applications. With enhanced multimodal data processing and improved inference efficiency, Granite Switch 4.1 significantly boosts AI system performance in real-time interaction scenarios. It supports multi-task collaboration and introduces a

News · 2026-10-15

Meta and Apple Collaborate to Enhance Muse Spark AI Model Performance on Macs

Meta has announced the Muse Spark Open AI model, claiming it to be the most capable US open-weight model to date. The company has also revealed a collaboration with Apple to optimize the model for running on two Apple M5 Ultra Mac Studios with the latest MLX framework. This partnership marks a rare alliance between Meta and Apple in the AI domain, showcasing new possibilities for AI model deployment on desktop hardware.

News · 2026-10-15

Meta Releases Muse Spark Open: AI Coding-Optimized Model Now Available on SageMaker JumpStart

Meta has officially released Muse Spark Open, a coding-optimized open-weight model, now available on Amazon SageMaker JumpStart. The model features a Mixture of Experts (MoE) architecture with 816 billion total parameters (42 billion active per token) and supports a 1 million-token context window. It can process multimodal data, including text, images, documents, audio, and audio-video, and is designed for agentic workflows, focusing on first-attempt accuracy, tool calling, and understanding lar

Info · 2026-10-15

Meta Releases Muse Spark Open: The Most Capable US Open-Weight Model to Date

Meta has officially released Muse Spark Open, a large-scale open-weight model designed for coding and agentic tasks. With 816 billion total parameters (42 billion active), a 1-million-token context window, and support for text, image, video, and audio modalities, the model excels in long-horizon tasks, complex software engineering, and multi-step reasoning. It has undergone additional safety post-training under Meta's Superintelligence Scaling Framework. Muse Spark Open is optimized for leading

Info · 2026-10-15

NVIDIA Releases Nemotron 3.5 Bolt 74B-A9B 24GB: Vision-Language Model Optimized for Localized Agentic Tasks

NVIDIA has released the Nemotron 3.5 Bolt 74B-A9B 24GB, a vision-language model (VLM) specifically designed for localized agentic tasks. Utilizing 2-bit quantization, the model fits within a 24GB memory footprint and can run alongside other applications. Its key strength lies in balancing intelligence and speed, outperforming most popular dense models in speed while surpassing most MoE models with a similar memory footprint in intelligence. The model supports multimodal inputs (text, images, vid

News · 2026-10-15

LlamaIndex Releases ExtractBench Pro: Targeting the Long Tail of Document Extraction Challenges

LlamaIndex has released ExtractBench Pro, an extension of ExtractBench designed to address complex extraction challenges in real-world long-tail documents. This new version introduces six new challenges, such as intricate charts, domain-specific jargon, and chaotic layouts, leveraging synthetic data generation techniques to produce high-fidelity training samples. ExtractBench Pro demonstrates cutting-edge capabilities in handling unstructured documents and provides more robust tooling for agenti

News · 2026-10-15

Hugging Face Releases FLUX.3 Image Klein: Revolutionizing the Diffusers Image Generation Framework

Hugging Face has released FLUX.3 Image Klein, the latest version of its Diffusers image generation framework. This update introduces new generative mechanisms and optimization techniques that significantly enhance image quality and efficiency. FLUX.3 Image Klein employs more efficient sampling strategies and improves modeling capabilities for complex scenes, making it highly effective across various image generation tasks.

News · 2026-10-15

arXiv Releases DecisionRobust: A Benchmark for Evaluating Robustness of Multimodal Decision Models

arXiv has released DecisionRobust, a novel benchmark for systematically evaluating the robustness of multimodal decision models under textual and visual inputs. The benchmark employs adversarial attacks such as semantics-preserving text perturbations, image perturbations, and cross-modal prompt injections to assess model reliability in complex decision-making scenarios. Experimental results reveal significant variations in robustness across different tasks, and resistance to input perturbations

News · 2026-10-14

PrismaCoder Launch: A New Breakthrough in Open Multimodal Code Intelligence

PrismaCoder team introduces PrismaForge, a data pipeline centered on synthesized rubrics, and releases PrismaCoder-2M, the largest open dataset for multimodal code intelligence, along with PrismaCoder-9B and PrismaCoder-35B models. These models significantly advance the state-of-the-art in open multimodal code intelligence, rivaling proprietary systems in tasks such as code generation, chart creation, and interactive document generation. The release addresses critical data quality and diversity

News · 2026-10-14

Hugging Face Releases CommerceGen-Studio: Revolutionizing E-commerce Main Image Generation

Hugging Face has released CommerceGen-Studio, an end-to-end framework for generating commercial-grade main images for e-commerce Stock Keeping Units (SKUs). The framework employs a three-stage pipeline: information extraction, semantic rewriting, and multi-condition generation. It utilizes the PERL-Agent for distilling SKU metadata, Layout Thinker Optimization for aligning text and image quality, and SKU-DiT for generating images that maintain subject fidelity and trademark integrity. Operating

News · 2026-10-14

Hugging Face Introduces RepForcing: Enhancing Autoregressive Video Generation Efficiency and Quality

Hugging Face introduces RepForcing, a novel method to enhance the efficiency and quality of autoregressive video generation. This approach leverages representation-space supervision for few-step causal initialization, replacing the traditional intermediate-teacher trajectory regression. Empirical results demonstrate that RepForcing preserves generation quality better at four sampling steps compared to teacher-based initialization before DMD. After DMD, the four-step chunk-wise model shows signif

News · 2026-10-14

Hugging Face Releases Intern Lumina U2: A Multi-Codebook Diffusion LLM for Advanced Visual Understanding and Generation

Hugging Face has unveiled Intern Lumina U2, a multi-codebook diffusion language model designed to address the trade-offs between discrete and continuous representations in multimodal generation. By representing visual tokens in a fully discrete space using multiple complementary codebooks, the model enhances visual representation capacity while maintaining compatibility with discrete text modeling. Intern Lumina U2 supports image, video, and 3D inputs and achieves leading performance in visual u