跳到主内容
智客 ZICQ

技能库 智客分类:媒体内容 gemini-live-api-dev

Gemini Live Api Dev

在与双子座Live API构建实时,双向流线应用程序时使用此技能. 封面基于WebSocket的音频/视频/文本流,语音活动检测(VAD),本地音频功能,函数调用,会话管理,客户端认证的ephemer sords,活翻译,以及所有Live API配置选项. SDK覆盖-google-genai (Python), @google/genai (JavaScript/TypeScript).

8107 安装量

官方网址:skills.sh

技能介绍

先看中文介绍;官方 description 原文单独保留,不改写 SKILL.md。

做什么

在与双子座Live API构建实时,双向流线应用程序时使用此技能. 封面基于WebSocket的音频/视频/文本流,语音活动检测(VAD),本地音频功能,函数调用,会话管理,客户端认证的ephemer sords,活翻译,以及所有Live API配置选项. SDK覆盖-google-genai (Python), @google/genai (JavaScript/TypeScript).

何时用

官方 description 未单独写出 Use when。按规范,代理会在用户任务与这段 description 的关键词匹配时激活本技能。

代理如何加载

按 Agent Skills 渐进披露:启动时只加载 name 与 description(约 100 token);任务匹配后才读入整份 SKILL.md 正文;scripts/、references/、assets/ 仅在需要时再读。 本文件正文结构:Gemini Live API Development Skill、Overview、Models、SDKs、Partner Integrations、Audio Formats。

文件分析

文件分析:这是一份仅含 SKILL.md 的指令型技能,代理激活后整份正文进入上下文。

官方 description(原文)

Use this skill when building real-time, bidirectional streaming applications with the Gemini Live API. Covers WebSocket-based audio/video/text streaming, voice activity detection (VAD), native audio features, function calling, session management, ephemeral tokens for client-side auth, live translation, and all Live API configuration options. SDKs covered - google-genai (Python), @google/genai (JavaScript/TypeScript).

Gemini Live API Development SkillOverviewModelsSDKsPartner IntegrationsAudio FormatsQuick StartAuthenticationConnecting to the Live APISending TextSending AudioSending Video

来源分类:skills.sh agent-skill

SKILL.md 与 Agent 调用

官方规范 ↗
name
gemini-live-api-dev
description
Use this skill when building real-time, bidirectional streaming applications with the Gemini Live API. Covers WebSocket-based audio/video/text streaming, voice activity detection (VAD), native audio features, function calling, session management, ephemeral tokens for client-side auth, live translation, and all Live API configuration options. SDKs covered - google-genai (Python), @google/genai (JavaScript/TypeScript).
  1. 发现技能客户端向 Agent 提供名称与描述目录。
  2. 匹配与调用用户指定或任务匹配后,载入 SKILL.md 指令。
  3. 按需加载按步骤读取参考文档、使用脚本与素材。

具体调用语法与可用工具以目标 Agent 客户端为准。 查看调用机制说明 ↗

安装这个技能

Skills CLI ↗

先选择目标 Agent 和安装范围,保留技能包的附属文件,安装后检查客户端能否发现该技能。

交给 Agent 安装

复制安装指令给支持 Agent Skills 的代理,确认其中的目标目录与客户端匹配。

把 Agent Skill「gemini-live-api-dev」安装到我的项目:SKILL.md 原文与官方 description 见 https://zicq.com/zh/skills/skl-e2868bb759cc920f-%E5%8F%8C%E5%AD%90%E5%BA%A7%E6%B4%BB%E4%BD%93-Api-Dev.html
请存为 .cursor/skills/gemini-live-api-dev/SKILL.md 或 .claude/skills/gemini-live-api-dev/SKILL.md,frontmatter 的 name 与 description 保持原样,不要改写。

GitHub 完整包 ↗

终端安装 · Skills CLI

需要 Node.js 与 npx。先查看仓库技能列表,确认实际名称。

npx skills add 'https://github.com/google-gemini/gemini-skills' --list

npx skills add 'https://github.com/google-gemini/gemini-skills' --skill 'gemini-live-api-dev'

CLI 会交互选择目标 Agent,默认安装到项目;用户级安装使用 -g。先通过查看命令核对仓库内容,再用 npx skills list 检查已安装技能。

阅读排版

name: gemini-live-api-dev description: Use this skill when building real-time, bidirectional streaming applications with the Gemini Live API. Covers WebSocket-based audio/video/text streaming, voice activity detection (VAD), native audio features, function calling, session management, ephemeral tokens for client-side auth, live translation, and all Live API configuration options. SDKs covered - google-genai (Python), @google/genai (JavaScript/TypeScript).

Gemini Live API Development Skill

Overview

The Live API enables low-latency, real-time voice and video interactions with Gemini over WebSockets. It processes continuous streams of audio, video, or text to deliver immediate, human-like spoken responses.

Key capabilities:

  • Bidirectional audio streaming — real-time mic-to-speaker conversations
  • Video streaming — send camera/screen frames alongside audio
  • Text input/output — send and receive text within a live session
  • Audio transcriptions — get text transcripts of both input and output audio
  • Voice Activity Detection (VAD) — automatic interruption handling
  • Native audio — thinking (with configurable thinkingLevel)
  • Function calling — synchronous tool use
  • Google Search grounding — ground responses in real-time search results
  • Session management — context compression, session resumption, GoAway signals
  • Ephemeral tokens — secure client-side authentication

[!NOTE] The Live API currently only supports WebSockets. For WebRTC support or simplified integration, use a partner integration.

Models

  • gemini-3.1-flash-live-preview — Optimized for low-latency, real-time dialogue. Native audio output, thinking (via thinkingLevel). 128k context window. This is the recommended model for all Live API use cases.
  • gemini-3.5-transcribe-live — Real-time streaming speech-to-text with interim hypotheses, finalized transcripts, smart formatting, and Hybrid VAD.
  • gemini-3.5-live-translate-preview — Real-time streaming translation model.

[!WARNING] The following Live API models are deprecated and will be shut down. Migrate to gemini-3.1-flash-live-preview.

  • gemini-2.5-flash-native-audio-preview-12-2025 — Migrate to gemini-3.1-flash-live-preview.
  • gemini-live-2.5-flash-preview — Released June 17, 2025. Shutdown: December 9, 2025.
  • gemini-2.0-flash-live-001 — Released April 9, 2025. Shutdown: December 9, 2025.

SDKs

  • Python: google-genai — pip install google-genai
  • JavaScript/TypeScript: @google/genai — npm install @google/genai

[!WARNING] Legacy SDKs google-generativeai (Python) and @google/generative-ai (JS) are deprecated. Use the new SDKs above.

Partner Integrations

To streamline real-time audio/video app development, use a third-party integration supporting the Gemini Live API over WebRTC or WebSockets:

  • LiveKit — Use the Gemini Live API with LiveKit Agents.
  • Pipecat by Daily — Create a real-time AI chatbot using Gemini Live and Pipecat.
  • Fishjam by Software Mansion — Create live video and audio streaming applications with Fishjam.
  • Vision Agents by Stream — Build real-time voice and video AI applications with Vision Agents.
  • Voximplant — Connect inbound and outbound calls to Live API with Voximplant.
  • Firebase AI SDK — Get started with the Gemini Live API using Firebase AI Logic.

Audio Formats

  • Input: Raw PCM, little-endian, 16-bit, mono. 16kHz native (will resample others). MIME type: audio/pcm;rate=16000
  • Output: Raw PCM, little-endian, 16-bit, mono. 24kHz sample rate.

[!IMPORTANT] Use send_realtime_input / sendRealtimeInput for all real-time user input (audio, video, and text). send_client_content / sendClientContent is only supported for seeding initial context history (requires setting initial_history_in_client_content in history_config). Do not use it to send new user messages during the conversation.

[!WARNING] Do not use media in sendRealtimeInput. Use the specific keys: audio for audio data, video for images/video frames, and text for text input.


Quick Start

Authentication

Python

from google import genai

client = genai.Client(api_key="YOUR_API_KEY")

JavaScript

import { GoogleGenAI } from '@google/genai';

const ai = new GoogleGenAI({ apiKey: 'YOUR_API_KEY' });

Connecting to the Live API

Python

from google.genai import types

config = types.LiveConnectConfig(
    response_modalities=[types.Modality.AUDIO],
    system_instruction=types.Content(
        parts=[types.Part(text="You are a helpful assistant.")]
    )
)

async with client.aio.live.connect(model="gemini-3.1-flash-live-preview", config=config) as session:
    pass  # Session is active

JavaScript

const session = await ai.live.connect({
  model: 'gemini-3.1-flash-live-preview',
  config: {
    responseModalities: ['audio'],
    systemInstruction: { parts: [{ text: 'You are a helpful assistant.' }] }
  },
  callbacks: {
    onopen: () => console.log('Connected'),
    onmessage: (response) => console.log('Message:', response),
    onerror: (error) => console.error('Error:', error),
    onclose: () => console.log('Closed')
  }
});

Sending Text

Python

await session.send_realtime_input(text="Hello, how are you?")

JavaScript

session.sendRealtimeInput({ text: 'Hello, how are you?' });

Sending Audio

Python

await session.send_realtime_input(
    audio=types.Blob(data=chunk, mime_type="audio/pcm;rate=16000")
)

JavaScript

session.sendRealtimeInput({
  audio: { data: chunk.toString('base64'), mimeType: 'audio/pcm;rate=16000' }
});

Sending Video

Python

# frame: raw JPEG-encoded bytes
await session.send_realtime_input(
    video=types.Blob(data=frame, mime_type="image/jpeg")
)

JavaScript

session.sendRealtimeInput({
  video: { data: frame.toString('base64'), mimeType: 'image/jpeg' }
});

Receiving Audio and Text

[!IMPORTANT] A single server event can contain multiple content parts simultaneously (e.g., audio chunks and transcript). Always process all parts in each event to avoid missing content.

Python

async for response in session.receive():
    content = response.server_content
    if content:
        # Audio — process ALL parts in each event
        if content.model_turn:
            for part in content.model_turn.parts:
                if part.inline_data:
                    audio_data = part.inline_data.data
        # Transcription
        if content.input_transcription:
            print(f"User: {content.input_transcription.text}")
        if content.output_transcription:
            print(f"Gemini: {content.output_transcription.text}")
        # Interruption
        if content.interrupted is True:
            pass  # Stop playback, clear audio queue

JavaScript

// Inside the onmessage callback
const content = response.serverContent;
if (content?.modelTurn?.parts) {
  for (const part of content.modelTurn.parts) {
    if (part.inlineData) {
      const audioData = part.inlineData.data; // Base64 encoded
    }
  }
}
if (content?.inputTranscription) console.log('User:', content.inputTranscription.text);
if (content?.outputTranscription) console.log('Gemini:', content.outputTranscription.text);
if (content?.interrupted) { /* Stop playback, clear audio queue */ }

Live Translation (Gemini Live Translate)

The Live API supports real-time, low-latency streaming translation of speech (audio) across 70+ languages. For full details on options and capabilities, see the Live Translate Guide.

Model

  • gemini-3.5-live-translate-preview — The recommended translation model for all Live Translate use cases.

Configuration (TranslationConfig)

To enable translation, specify a TranslationConfig object inside your live session setup:

  • Python SDK: Configure the connection using translation_config on LiveConnectConfig:
    config = types.LiveConnectConfig(
        response_modalities=[types.Modality.AUDIO],
        translation_config=types.TranslationConfig(
            target_language_code="es",  # Target language code (e.g. es, fr, pl)
            echo_target_language=True,
        ),
        input_audio_transcription=types.AudioTranscriptionConfig(),
        output_audio_transcription=types.AudioTranscriptionConfig(),
    )
    
  • Raw WebSockets: Place translationConfig inside generationConfig:
    {
      "setup": {
        "model": "models/gemini-3.5-live-translate-preview",
        "generationConfig": {
          "responseModalities": ["AUDIO"],
          "translationConfig": {
            "targetLanguageCode": "es",
            "echoTargetLanguage": true
          }
        }
      }
    }
    

Live Streaming Transcription (Gemini Live Transcribe)

The Live API supports real-time streaming speech-to-text over WebSockets with low-latency interim hypotheses, finalized transcripts, and Hybrid VAD. For full details, see the Live Transcription Guide and Colab Cookbook.

Model

  • gemini-3.5-transcribe-live

Modes

  • smart: cleans up filler words, resolves inline self-corrections, and structures formatting.
  • verbatim (default): exact word-for-word transcript.

Python

config = types.LiveConnectConfig(
    response_modalities=["TEXT"],
    input_audio_transcription=types.AudioTranscriptionConfig(),
)

async with client.aio.live.connect(model="gemini-3.5-transcribe-live", config=config) as session:
    # Stream audio
    await session.send_realtime_input(audio=types.Blob(data=chunk, mime_type="audio/pcm;rate=16000"))
    # Hybrid VAD: notify turn end on client-detected silence for zero latency
    await session.send_realtime_input(audio_stream_end=True)

JavaScript

const session = await ai.live.connect({
  model: 'gemini-3.5-transcribe-live',
  config: {
    responseModalities: ['text'],
    inputAudioTranscription: { mode: 'smart' }
  },
  callbacks: {
    onmessage: (msg) => {
      if (msg.serverContent?.interimInputTranscription) {
        console.log('Interim:', msg.serverContent.interimInputTranscription.text);
      }
      if (msg.serverContent?.inputTranscription) {
        console.log('Final:', msg.serverContent.inputTranscription.text);
      }
    }
  }
});

session.sendRealtimeInput({ audio: { data: chunkBase64, mimeType: 'audio/pcm;rate=16000' } });
session.sendRealtimeInput({ audioStreamEnd: true }); // Hybrid VAD

Raw WebSockets

{
  "setup": {
    "model": "models/gemini-3.5-transcribe-live",
    "generationConfig": {
      "responseModalities": ["TEXT"],
      "speechConfig": {
        "voiceConfig": {}
      }
    },
    "inputAudioTranscription": {
      "mode": "smart"
    }
  }
}

Limitations

  • Response modality — Only TEXT or AUDIO per session, not both. Native audio models only support audio.
  • Audio-only session — 15 min without compression
  • Audio+video session — 2 min without compression
  • Connection lifetime — ~10 min (use session resumption)
  • Context window — 128k tokens (native audio) / 32k tokens (standard)
  • Async function calling — Not yet supported; function calling is synchronous only. The model will not start responding until you've sent the tool response.
  • Proactive audio — Not yet supported in Gemini 3.1 Flash Live. Remove any configuration for this feature.
  • Affective dialogue — Not yet supported in Gemini 3.1 Flash Live. Remove any configuration for this feature.
  • Code execution — Not supported
  • URL context — Not supported

Migrating from Gemini 2.5 Flash Live

When migrating from gemini-2.5-flash-native-audio-preview-12-2025 to gemini-3.1-flash-live-preview:

  1. Model string — Update from gemini-2.5-flash-native-audio-preview-12-2025 to gemini-3.1-flash-live-preview.
  2. Thinking configuration — Use thinkingLevel (minimal, low, medium, high) instead of thinkingBudget. Default is minimal for lowest latency.
  3. Server events — A single event can contain multiple content parts simultaneously (audio + transcript). Process all parts in each event.
  4. Client content — send_client_content is only for seeding initial context history (set initial_history_in_client_content in history_config). Use send_realtime_input for text during conversation.
  5. Turn coverage — Defaults to TURN_INCLUDES_AUDIO_ACTIVITY_AND_ALL_VIDEO instead of TURN_INCLUDES_ONLY_ACTIVITY. If sending constant video frames, consider sending only during audio activity to reduce costs.
  6. Async function calling — Not yet supported. Function calling is synchronous only.
  7. Proactive audio & affective dialogue — Not yet supported. Remove any configuration for these features.

Best Practices

  1. Use headphones when testing mic audio to prevent echo/self-interruption
  2. Enable context window compression for sessions longer than 15 minutes
  3. Implement session resumption to handle connection resets gracefully
  4. Use ephemeral tokens for client-side deployments — never expose API keys in browsers
  5. Use send_realtime_input for all real-time user input (audio, video, text). Reserve send_client_content only for seeding initial context history
  6. Send audioStreamEnd when the mic is paused to flush cached audio
  7. Clear audio playback queues on interruption signals
  8. Process all parts in each server event — events can contain multiple content parts

Documentation Lookup

When MCP is Installed (Preferred)

If the search_docs tool (from the Google MCP server) is available, use it as your only documentation source:

  1. Call search_docs with your query
  2. Read the returned documentation
  3. Trust MCP results as source of truth for API details — they are always up-to-date.

[!IMPORTANT] When MCP tools are present, never fetch URLs manually. MCP provides up-to-date, indexed documentation that is more accurate and token-efficient than URL fetching.

When MCP is NOT Installed (Fallback Only)

If no MCP documentation tools are available, fetch from the official docs index:

llms.txt URL: https://ai.google.dev/gemini-api/docs/llms.txt

This index contains links to all documentation pages in .md.txt format. Use web fetch tools to:

  1. Fetch llms.txt to discover available documentation pages
  2. Fetch specific pages (e.g., https://ai.google.dev/gemini-api/docs/live-session.md.txt)

Key Documentation Pages

[!IMPORTANT] Those are not all the documentation pages. Use the llms.txt index to discover available documentation pages

Supported Languages

The Live API supports 70 languages including: English, Spanish, French, German, Italian, Portuguese, Chinese, Japanese, Korean, Hindi, Arabic, Russian, and many more. Native audio models automatically detect and switch languages.

相关技能

媒体内容

Nano Banana Pro

以纳米·香蕉Pro(Gemini 3 Pro Image)来生成/编辑图像. 用于创建/修改请求,包括编辑。 支持文本到图像+图像到图像; 1K/2K/4K; 使用 -- input-image.

媒体内容

Youtube Watcher

从YouTube视频中获取并读取文字记录. 需要汇总视频时使用,回答有关视频内容的问题,或者从中提取信息.

媒体内容

Atxp

Access ATXP支付API工具用于网页搜索,AI图像生成,音乐创建,视频生成,X/Twitter搜索,电子邮件,代理账户管理. 当用户需要实时网络搜索时使用,AI生成的媒体(图像,音乐,视频),X/Twitter搜索,发送/接收电子邮件,或创建和资助代理账户. 需要通过“ …

媒体内容

Youtube

YouTube Data API与管理的OAuth的集成. 搜索视频,管理播放列表,访问频道数据,并与评论互动. 用户想与YouTube互动时使用此技能. 对于其他第三方应用,使用api-gateway技能(https://clawhub.ai/byungkyu/api-gate…