跳到主内容
智客 ZICQ

技能库 智客分类:Agent 工作流 jetson-speculative-decoding

Jetson Speculative Decoding

当TPOT是瓶颈时,在Jetson vLLM服务器上添加EAGLE-3或草稿-模型投机解码.

1148 安装量

官方网址:skills.sh

技能介绍

先看中文介绍;官方 description 原文单独保留,不改写 SKILL.md。

做什么

当TPOT是瓶颈时,在Jetson vLLM服务器上添加EAGLE-3或草稿-模型投机解码.

何时用

- TPOT/ITL是瓶颈(TTFT很好,输出很慢)

代理如何加载

按 Agent Skills 渐进披露:启动时只加载 name 与 description(约 100 token);任务匹配后才读入整份 SKILL.md 正文;scripts/、references/、assets/ 仅在需要时再读。 本文件正文结构:Jetson Speculative Decoding (vLLM)、Purpose、When to use、When NOT to use、Prerequisites、Instructions。 其中含规范建议的小节:边界情况。

文件分析

文件分析:这是一份仅含 SKILL.md 的指令型技能,代理激活后整份正文进入上下文。

官方 description(原文)

Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck.

Jetson Speculative Decoding (vLLM)PurposeWhen to useWhen NOT to usePrerequisitesInstructionsJetson-specific tuning rulesHow to verify it actually helpedLimitationsError handlingHand off toSource

· 许可:Apache-2.0

来源分类:skills.sh agent-skill

SKILL.md 与 Agent 调用

官方规范 ↗
name
jetson-speculative-decoding
description
Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck.
许可
Apache-2.0
  1. 发现技能客户端向 Agent 提供名称与描述目录。
  2. 匹配与调用用户指定或任务匹配后,载入 SKILL.md 指令。
  3. 按需加载按步骤读取参考文档、使用脚本与素材。

具体调用语法与可用工具以目标 Agent 客户端为准。 查看调用机制说明 ↗

安装这个技能

Skills CLI ↗

先选择目标 Agent 和安装范围,保留技能包的附属文件,安装后检查客户端能否发现该技能。

交给 Agent 安装

复制安装指令给支持 Agent Skills 的代理,确认其中的目标目录与客户端匹配。

把 Agent Skill「jetson-speculative-decoding」安装到我的项目:SKILL.md 原文与官方 description 见 https://zicq.com/zh/skills/skl-a264d15b4f5f7aa8-%E6%9D%B0%E7%89%B9%E6%A3%AE%E5%88%86%E5%85%89%E5%89%82-%E8%A7%A3%E7%A0%81.html
请存为 .cursor/skills/jetson-speculative-decoding/SKILL.md 或 .claude/skills/jetson-speculative-decoding/SKILL.md,frontmatter 的 name 与 description 保持原样,不要改写。

GitHub 完整包 ↗

终端安装 · Skills CLI

需要 Node.js 与 npx。先查看仓库技能列表,确认实际名称。

npx skills add 'https://github.com/nvidia/skills' --list

npx skills add 'https://github.com/nvidia/skills' --skill 'jetson-speculative-decoding'

CLI 会交互选择目标 Agent,默认安装到项目;用户级安装使用 -g。先通过查看命令核对仓库内容,再用 npx skills list 检查已安装技能。

阅读排版
--- name: jetson-speculative-decoding description: Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck. version: 0.0.1 license: "Apache-2.0" metadata: author: "Jetson Team" tags: [jetson, llm, speculative-decoding] languages: [markdown] data-classification: public --- # Jetson Speculative Decoding (vLLM) Speculative decoding lets a small "draft" model propose tokens that the target model verifies in a single forward pass, reducing per-token latency. On Jetson, the win/loss is **dominated by VRAM headroom**, not by the draft quality. This skill encodes the parts an LLM won't already know. ## Purpose Tune an existing Jetson vLLM deployment for faster token generation by appending the right `--speculative-config` and validating whether it improves single-stream decode speed. ## When to use - TPOT/ITL is the bottleneck (TTFT is fine, output is just slow). - Workload is single-stream or low-concurrency (≤2). Speculation usually loses at high concurrency. - Jetson family is **Thor or AGX Orin**. Do **not** suggest EAGLE-3 on Orin Nano/NX — there is rarely enough VRAM headroom to host both target and draft, and you'll OOM at startup. ## When NOT to use - High-concurrency serving (≥8): batched decode usually beats speculation; the draft model just steals VRAM. - Models without a published EAGLE-3 head — do not train one ad-hoc as a "fix". - After applying `jetson-inference-mem-tune` flags that already pushed `--gpu-memory-utilization` near the ceiling. Free at least ~2 GB first. ## Prerequisites - A working vLLM server recipe from `jetson-llm-serve`. - Enough memory headroom for the draft model or EAGLE-3 head in addition to the target model. - A benchmark baseline from `jetson-llm-benchmark` before enabling speculation. - A target model with a compatible EAGLE-3 head, or a small same-family draft model for the fallback path. ## Instructions Append `--speculative-config` to the `vllm serve` command shown in `jetson-llm-serve`. EAGLE-3 (preferred when a head is published for the target model): ```bash --speculative-config '{ "method": "eagle3", "model": "", "num_speculative_tokens": 5, "draft_tensor_parallel_size": 1 }' ``` Draft-model (fallback — pair a small same-family model): ```bash --speculative-config '{ "method": "draft_model", "model": "", "num_speculative_tokens": 4, "draft_tensor_parallel_size": 1 }' ``` ### Jetson-specific tuning rules - `num_speculative_tokens`: start at **5** on Thor, **3** on AGX Orin. Higher values pay off only if the draft acceptance rate is >0.6. - Always pair with the same vLLM runtime path used by `jetson-llm-serve`: upstream vLLM 0.20+ (`vllm/vllm-openai:latest`) or validated native vLLM 0.20+ on Thor, upstream vLLM 0.20+ on Orin JetPack 7.2 / L4T r39+, or the NVIDIA-AI-IOT vLLM image on older Orin. Do not use an Orin NVIDIA-AI-IOT vLLM image on Thor. Older runtimes may lack EAGLE-3 or the current `--speculative-config` shape. - Drop `--gpu-memory-utilization` by ~0.05 vs the non-speculative baseline to give the draft model headroom. ## How to verify it actually helped 1. Run `jetson-llm-benchmark` (vLLM path) at `--concurrency 1` **before and after** enabling speculation. 2. Acceptance: target ≥30% improvement in `throughput_tok_s` and ≥20% drop in `tpot_ms_p50` at concurrency 1. 3. If improvement is <10%, or `throughput_tok_s` regresses at concurrency 8, **disable** speculation. The draft model is costing more than it returns. ## Limitations - Speculative decoding improves decode-heavy workloads; it does not reduce TTFT-dominated latency. - High concurrency can erase the benefit because continuous batching already keeps the GPU busy. - Orin Nano/NX usually lack enough memory headroom for both target and draft models. - Acceptance rate and draft overhead are model-specific, so benchmark before and after instead of assuming a speedup. ## Error handling - If vLLM rejects `--speculative-config`, verify that Thor and Orin JetPack 7.2 / L4T r39+ are using vLLM 0.20+ and that older Orin is using a JetPack-matched NVIDIA-AI-IOT vLLM image; then switch back to the non-speculative serving command if the runtime still rejects it. - If startup OOMs, lower `--gpu-memory-utilization`, use a smaller draft, or disable speculation and hand off to `jetson-inference-mem-tune`. - If benchmark throughput regresses, remove `--speculative-config`; a bad draft path is worse than no speculation. ## Hand off to - `jetson-llm-benchmark` to quantify the change. - `jetson-inference-mem-tune` if startup OOMs after enabling speculation. ## Source vLLM speculative decoding docs and the [Jetson AI Lab GenAI tutorial](https://www.jetson-ai-lab.com/tutorials/genai-on-jetson-llms-vlms/).

相关技能

Agent 工作流

Skill Creator

创造有效技能指南。 当用户想创造出新的技能(或更新现有的技能),以专业知识,工作流程,或工具集成来扩展克洛德的能力时,应该使用这种技能.

Agent 工作流

Clawdhub

使用ClawdHub CLI搜索,安装,更新并发布从taladhub.com的代理技能. 需要获取苍蝇上的新技能时使用,将安装的技能同步到最新版本或特定版本,或者发布 npm-instainddhub CLI 的新/更新的技能文件夹.

Agent 工作流

Agent Team Orchestration

管弦乐团多代理团队,任务设定周期,交接协议,审查工作流程. 使用时间: (1)建立2+特派员队伍,具有不同专业,(2)确定任务路线和生命周期(收录框_ spec_建设_审查_完成),(3)在特派员之间制定交接协议,(4)建立审查和质量关口,(5)管理特派员之间的交流和文物共享.

Agent 工作流

Superpowers

Spec-first,TDD,子代理驱动的软件开发工作流程. 当:(1)构建任何新功能或应用——触发脑暴_计划_子代理执行回路,(2)调试出一个bug或测试失败——触发系统性的根起过程,(3)用户说"让我们构建","帮助我计划","我想添加X",或"这个被打破",(4)完成一个功…