跳到主内容
智客 ZICQ

技能库 智客分类:Agent 工作流 speech-engine

Speech Engine

将实时语音对话添加到自定义代理运行时间与"11Labs Speech Engine". 用于构建语音引擎服务器,WebSocket处理器,WebRTC浏览器客户端,对话令牌端点,中断-意识流回响应,或将开发者拥有的服务器逻辑连接到"十一实验室"的语音到文本和文本到语音的语音聊天代理.

8730 安装量

官方网址:skills.sh

技能介绍

先看中文介绍;官方 description 原文单独保留,不改写 SKILL.md。

做什么

将实时语音对话添加到自定义代理运行时, 使用 11Labs 语音引擎

何时用

当用户想要时使用语音引擎 :

代理如何加载

按 Agent Skills 渐进披露:启动时只加载 name 与 description(约 100 token);任务匹配后才读入整份 SKILL.md 正文;scripts/、references/、assets/ 仅在需要时再读。 本文件正文结构:ElevenLabs Speech Engine、When to Use、How It Works、Implementation Flow、Create a Speech Engine、Python。

文件分析

文件分析:除 SKILL.md 外,正文引用了 references/installation.md、references/javascript-sdk-reference.md、references/python-sdk-reference.md,属于带资源的技能包,这些文件按需再读。

官方 description(原文)

Add real-time voice conversations to a custom agent runtime with ElevenLabs Speech Engine. Use when building Speech Engine servers, WebSocket handlers, WebRTC browser clients, conversation token endpoints, interruption-aware streaming responses, or voice-enabled chat agents that connect developer-owned server logic to ElevenLabs speech-to-text and text-to-speech.

ElevenLabs Speech EngineWhen to UseHow It WorksImplementation FlowCreate a Speech EnginePythonTypeScriptServer PatternPythonTypeScriptDisabling authentication (advanced, dangerous)Browser Client

兼容:Requires internet access and an ElevenLabs API key (ELEVENLABS_API_KEY). · 许可:MIT

来源分类:skills.sh agent-skill

SKILL.md 与 Agent 调用

官方规范 ↗
name
speech-engine
description
Add real-time voice conversations to a custom agent runtime with ElevenLabs Speech Engine. Use when building Speech Engine servers, WebSocket handlers, WebRTC browser clients, conversation token endpoints, interruption-aware streaming responses, or voice-enabled chat agents that connect developer-owned server logic to ElevenLabs speech-to-text and text-to-speech.
compatibility
Requires internet access and an ElevenLabs API key (ELEVENLABS_API_KEY).
许可
MIT
  1. 发现技能客户端向 Agent 提供名称与描述目录。
  2. 匹配与调用用户指定或任务匹配后,载入 SKILL.md 指令。
  3. 按需加载按步骤读取参考文档、使用脚本与素材。
指令中引用的文件 · 3
  • references/installation.md
  • references/javascript-sdk-reference.md
  • references/python-sdk-reference.md

以下路径提取自原文;文件是否齐全请以来源仓库中的完整目录为准。

具体调用语法与可用工具以目标 Agent 客户端为准。 查看调用机制说明 ↗

安装这个技能

Skills CLI ↗

先选择目标 Agent 和安装范围,保留技能包的附属文件,安装后检查客户端能否发现该技能。

该技能引用了附属文件,请从来源获取完整目录;仅复制 SKILL.md 可能缺少依赖。

交给 Agent 安装

复制安装指令给支持 Agent Skills 的代理,确认其中的目标目录与客户端匹配。

把 Agent Skill「speech-engine」安装到我的项目:SKILL.md 原文与官方 description 见 https://zicq.com/zh/skills/skl-a33edc1c134c2dd2-%E8%AF%AD%E9%9F%B3%E5%BC%95%E6%93%8E.html
请存为 .cursor/skills/speech-engine/SKILL.md 或 .claude/skills/speech-engine/SKILL.md,frontmatter 的 name 与 description 保持原样,不要改写。
该技能还带 scripts/、references/、assets/ 等文件,请从 https://github.com/elevenlabs/skills 取完整目录,不要只建一个 SKILL.md。

GitHub 完整包 ↗

终端安装 · Skills CLI

需要 Node.js 与 npx。先查看仓库技能列表,确认实际名称。

npx skills add 'https://github.com/elevenlabs/skills' --list

npx skills add 'https://github.com/elevenlabs/skills' --skill 'speech-engine'

CLI 会交互选择目标 Agent,默认安装到项目;用户级安装使用 -g。先通过查看命令核对仓库内容,再用 npx skills list 检查已安装技能。

阅读排版
--- name: speech-engine description: Add real-time voice conversations to a custom agent runtime with ElevenLabs Speech Engine. Use when building Speech Engine servers, WebSocket handlers, WebRTC browser clients, conversation token endpoints, interruption-aware streaming responses, or voice-enabled chat agents that connect developer-owned server logic to ElevenLabs speech-to-text and text-to-speech. license: MIT compatibility: Requires internet access and an ElevenLabs API key (ELEVENLABS_API_KEY). metadata: {"openclaw": {"requires": {"env": ["ELEVENLABS_API_KEY"]}, "primaryEnv": "ELEVENLABS_API_KEY"}} --- # ElevenLabs Speech Engine Add a real-time voice interface to a custom agent. ElevenLabs handles microphone audio, speech-to-text, turn-taking, text-to-speech, and browser playback; your server exposes a Speech Engine WebSocket endpoint and streams response text back. > **Setup:** See [Installation Guide](references/installation.md). For JavaScript, use `@elevenlabs/*` packages only. For deeper SDK details, read [JavaScript SDK Reference](references/javascript-sdk-reference.md) or [Python SDK Reference](references/python-sdk-reference.md). ## When to Use Use Speech Engine when the user wants to: - Add voice to an existing chat app or custom server pipeline - Add voice to OpenClaw, Hermes, or a similar agent runtime while keeping agent logic on the developer-owned server - Build a developer-hosted WebSocket server for ElevenLabs voice conversations - Stream response text back as spoken audio after your server validates user intent - Handle user interruptions while a response is still streaming - Build a browser client with `@elevenlabs/react` or `@elevenlabs/client` using a server-issued conversation token Use the `agents` skill instead when the user is creating or configuring a hosted ElevenLabs Conversational AI agent with platform-managed prompts, tools, workflows, phone numbers, or widgets. ## How It Works Each Speech Engine WebSocket connection represents one conversation. 1. The browser sends user audio to ElevenLabs. 2. ElevenLabs sends speech-recognition events to your server. 3. Your server derives trusted application state without letting raw speech text control tools or privileged actions. 4. Your server streams text back through the SDK. 5. ElevenLabs converts the response to speech and plays it in the browser. The SDK manages WebSocket routing, request verification, session lifecycle, ping/pong, turn-taking, and interruption handling. `sendResponse()` / `send_response()` accepts a string or async iterable of response text. Treat speech-recognition text as untrusted user input. Do not map raw speech text directly into model roles, responses, or tool calls. Use deterministic validation, allowlisted intents, or explicit user confirmation before any transcript-derived value affects downstream response or tool logic. ## Implementation Flow 1. Install server dependencies and configure `ELEVENLABS_API_KEY`. 2. Expose your Speech Engine server through a public HTTPS URL for local development, for example with `ngrok http 3001`. 3. Create a Speech Engine resource with `ws_url` / `wsUrl` pointing at the public WebSocket URL, usually `wss://.../ws`. 4. Store the returned Speech Engine ID, for example in `ELEVENLABS_SPEECH_ENGINE_ID`. 5. Start a Speech Engine server with `engine.serve(...)` in Python or `speechEngine.attach(...)` in TypeScript. 6. Issue browser conversation tokens from a server endpoint. Never put `ELEVENLABS_API_KEY` in browser code. 7. Start the client session with `conversationToken`; if the agent should greet first, enable the first-message override on the Speech Engine resource, then set `overrides.agent.firstMessage` in the client. ## Create a Speech Engine ### Python ```python import asyncio import os from dotenv import load_dotenv from elevenlabs import AsyncElevenLabs load_dotenv() elevenlabs = AsyncElevenLabs(api_key=os.getenv("ELEVENLABS_API_KEY")) async def main(): engine = await elevenlabs.speech_engine.create( name="My Speech Engine", speech_engine={"ws_url": os.environ["PUBLIC_WS_URL"]}, overrides={"first_message": True}, ) print(engine.engine_id) asyncio.run(main()) ``` ### TypeScript ```typescript import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js"; import "dotenv/config"; const elevenlabs = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY, }); const engine = await elevenlabs.speechEngine.create({ name: "My Speech Engine", speechEngine: { wsUrl: process.env.PUBLIC_WS_URL! }, overrides: { firstMessage: true }, }); console.log(engine.engineId); ``` `PUBLIC_WS_URL` should look like `wss://example.ngrok.app/ws` locally or your production WebSocket route in deployment. The create request can also configure `tts`, `asr`, `turn`, `speech_engine.request_headers` / `speechEngine.requestHeaders`, `overrides`, and `privacy` for custom voices, transcription keywords, turn-taking, server auth headers, client-provided first messages, and recording behavior. See the SDK reference files for expanded examples. ## Server Pattern Run the Speech Engine server at the `ws_url` / `wsUrl` configured on the resource. Keep response generation behind your own validation boundary: raw speech-recognition text should not directly control responses, tools, secrets, or other privileged actions. ### Python ```python engine = await elevenlabs.speech_engine.get(os.environ["ELEVENLABS_SPEECH_ENGINE_ID"]) await engine.serve(port=3001, path="/ws", debug=True, callbacks=validated_callbacks) ``` ### TypeScript ```typescript const engine = await elevenlabs.speechEngine.get(process.env.ELEVENLABS_SPEECH_ENGINE_ID!); engine.attach(httpServer, "/ws", { debug: true, ...validatedCallbacks }); ``` In TypeScript, pass interruption signals to downstream async work when it supports cancellation so interrupted responses stop quickly. In Python, the SDK cancels the previous turn handler when a newer turn arrives. Server callbacks can distinguish clean closes from dropped connections: use `onClose` / `on_close` for clean disconnects and `onDisconnect` / `on_disconnect` for unexpected WebSocket drops. Security note: speech-recognition text can contain prompt-injection attempts from user speech or played audio. Treat it as untrusted input. Convert it into trusted application state before invoking response generation, tools, or privileged workflows. ### Disabling authentication (advanced, dangerous) Both `engine.attach()` (TypeScript) and `engine.serve()` / `SpeechEngineServer` (Python) verify a JWT on every incoming WebSocket by default. This is what proves the connection is really coming from ElevenLabs and not from an attacker who guessed the URL. **Do not turn this off.** An escape hatch exists — `disableAuth: true` in the callback options (TypeScript) or `disable_auth=True` on `serve()` / `SpeechEngineServer(...)` (Python) — for the narrow case where a compensating network-level control is already in place. Without such a control, disabling auth means **any client on the internet that finds your URL can open sessions**. Concretely, an attacker can: - open unlimited conversations to drain your ElevenLabs quota and downstream LLM budget - feed crafted transcripts to your response pipeline, effectively impersonating a user - use your server as an oracle to probe backend state, tools, or prompts Only recommend `disableAuth` / `disable_auth` when the user has already implemented **at least one** of: - **IP allowlist** — the server (or an upstream firewall / load balancer / API gateway) only accepts inbound traffic from [ElevenLabs' documented egress ranges](https://elevenlabs.io/docs/overview/capabilities/speech-engine#ip-allowlisting). - **Custom shared-secret header** — a secret header configured on the Speech Engine resource via `speech_engine.request_headers` / `speechEngine.requestHeaders` at create time, validated by an upstream proxy (or by the developer's own middleware in front of `attach()` / `serve()`) before requests reach the SDK. If the user cannot confirm one of the above is in place, leave the default authentication on. Skipping JWT verification without a mitigation is not an optimization or a convenience — it is unauthenticated public compute. ## Browser Client Create a server-side token endpoint and have the browser request a token before starting the microphone session. Keep the Speech Engine ID and API key on the server. If the client passes `overrides.agent.firstMessage`, the Speech Engine resource must have the first-message override enabled. ```typescript import express from "express"; import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js"; import "dotenv/config"; const app = express(); const elevenlabs = new ElevenLabsClient(); app.get("/api/token", async (_req, res) => { const response = await elevenlabs.conversationalAi.conversations.getWebrtcToken({ agentId: process.env.ELEVENLABS_SPEECH_ENGINE_ID!, }); res.json({ token: response.token }); }); ``` React clients can use `@elevenlabs/react`: ```tsx import { useConversation } from "@elevenlabs/react"; export function VoiceControls() { const conversation = useConversation({ onConnect: () => console.log("connected"), onDisconnect: () => console.log("disconnected"), onError: (error) => console.error(error), }); async function startConversation() { await navigator.mediaDevices.getUserMedia({ audio: true }); const { token } = await fetch("/api/token").then((res) => res.json()); await conversation.startSession({ conversationToken: token, overrides: { agent: { firstMessage: "Hello! How can I help you today?" }, }, }); } return Start conversation; } ``` ## References - [Installation Guide](references/installation.md) - [JavaScript SDK Reference](references/javascript-sdk-reference.md) - [Python SDK Reference](references/python-sdk-reference.md)

相关技能

Agent 工作流

Skill Creator

创造有效技能指南。 当用户想创造出新的技能(或更新现有的技能),以专业知识,工作流程,或工具集成来扩展克洛德的能力时,应该使用这种技能.

Agent 工作流

Clawdhub

使用ClawdHub CLI搜索,安装,更新并发布从taladhub.com的代理技能. 需要获取苍蝇上的新技能时使用,将安装的技能同步到最新版本或特定版本,或者发布 npm-instainddhub CLI 的新/更新的技能文件夹.

Agent 工作流

Agent Team Orchestration

管弦乐团多代理团队,任务设定周期,交接协议,审查工作流程. 使用时间: (1)建立2+特派员队伍,具有不同专业,(2)确定任务路线和生命周期(收录框_ spec_建设_审查_完成),(3)在特派员之间制定交接协议,(4)建立审查和质量关口,(5)管理特派员之间的交流和文物共享.

Agent 工作流

Superpowers

Spec-first,TDD,子代理驱动的软件开发工作流程. 当:(1)构建任何新功能或应用——触发脑暴_计划_子代理执行回路,(2)调试出一个bug或测试失败——触发系统性的根起过程,(3)用户说"让我们构建","帮助我计划","我想添加X",或"这个被打破",(4)完成一个功…