跳到主内容
智客 ZICQ

技能库 智客分类:Agent 工作流 arize-prompt-optimization

Arize Prompt Optimization

优化、改进和调试LLM,使用生产跟踪数据、评价和说明。 从跨度取出提示,收集性能信号,并使用轴 CLI 运行一个由数据驱动的优化循环. 当用户提到优化即时,改进即时,使AI反应更好,提高输出质量,即时工程,即时调试,或系统即时改进.

2928 安装量

官方网址:skills.sh

技能介绍

先看中文介绍;官方 description 原文单独保留,不改写 SKILL.md。

做什么

优化,改进并调试 LLM 使用生产痕量数据,评价和说明来提示. 从跨度取出提示,收集性能信号,并使用轴 CLI 运行数据驱动优化循环

何时用

官方 description 未单独写出 Use when。按规范,代理会在用户任务与这段 description 的关键词匹配时激活本技能。

代理如何加载

按 Agent Skills 渐进披露:启动时只加载 name 与 description(约 100 token);任务匹配后才读入整份 SKILL.md 正文;scripts/、references/、assets/ 仅在需要时再读。 本文件正文结构:Arize Prompt Optimization Skill、Related skills、Concepts、Where Prompts Live in Trace Data、Finding Prompts by Span Kind、Performance Signal Columns。

文件分析

文件分析:除 SKILL.md 外,正文引用了 references/ax-setup.md、references/ax-profiles.md、references/optimization-meta-prompt.md,属于带资源的技能包,这些文件按需再读。

官方 description(原文)

Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement.

Arize Prompt Optimization SkillRelated skillsConceptsWhere Prompts Live in Trace DataFinding Prompts by Span KindPerformance Signal ColumnsPrerequisitesWhen you must ask the user firstPhase 1: Extract the Current PromptFind LLM spans containing promptsSample LLM spans (where prompts live)Filter by model

兼容:Requires the ax CLI and a configured Arize profile.

来源分类:skills.sh agent-skill

SKILL.md 与 Agent 调用

官方规范 ↗
name
arize-prompt-optimization
description
Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement.
compatibility
Requires the ax CLI and a configured Arize profile.
  1. 发现技能客户端向 Agent 提供名称与描述目录。
  2. 匹配与调用用户指定或任务匹配后,载入 SKILL.md 指令。
  3. 按需加载按步骤读取参考文档、使用脚本与素材。
指令中引用的文件 · 3
  • references/ax-setup.md
  • references/ax-profiles.md
  • references/optimization-meta-prompt.md

以下路径提取自原文;文件是否齐全请以来源仓库中的完整目录为准。

具体调用语法与可用工具以目标 Agent 客户端为准。 查看调用机制说明 ↗

安装这个技能

Skills CLI ↗

先选择目标 Agent 和安装范围,保留技能包的附属文件,安装后检查客户端能否发现该技能。

该技能引用了附属文件,请从来源获取完整目录;仅复制 SKILL.md 可能缺少依赖。

交给 Agent 安装

复制安装指令给支持 Agent Skills 的代理,确认其中的目标目录与客户端匹配。

把 Agent Skill「arize-prompt-optimization」安装到我的项目:SKILL.md 原文与官方 description 见 https://zicq.com/zh/skills/skl-b6b48a4724346465-%E8%B0%83%E6%95%B4%E5%BF%AB%E9%80%9F%E4%BC%98%E5%8C%96.html
请存为 .cursor/skills/arize-prompt-optimization/SKILL.md 或 .claude/skills/arize-prompt-optimization/SKILL.md,frontmatter 的 name 与 description 保持原样,不要改写。
该技能还带 scripts/、references/、assets/ 等文件,请从 https://github.com/arize-ai/arize-skills 取完整目录,不要只建一个 SKILL.md。

GitHub 完整包 ↗

终端安装 · Skills CLI

需要 Node.js 与 npx。先查看仓库技能列表,确认实际名称。

npx skills add 'https://github.com/arize-ai/arize-skills' --list

npx skills add 'https://github.com/arize-ai/arize-skills' --skill 'arize-prompt-optimization'

CLI 会交互选择目标 Agent,默认安装到项目;用户级安装使用 -g。先通过查看命令核对仓库内容,再用 npx skills list 检查已安装技能。

阅读排版
--- name: arize-prompt-optimization description: Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement. metadata: author: arize version: "1.0" compatibility: Requires the ax CLI and a configured Arize profile. --- # Arize Prompt Optimization Skill > **`SPACE`** — `--space` flags accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`. ## Related skills - **arize-prompts**: Create, version, and label prompts in **Prompt Hub** with `ax prompts` (JSON messages, providers, labels such as `production`). Use that skill when the artifact should live in Arize; use **arize-prompt-optimization** below to improve prompt text from traces, datasets, and experiments. ## Concepts ### Where Prompts Live in Trace Data LLM applications emit spans following OpenInference semantic conventions. Prompts are stored in different span attributes depending on the span kind and instrumentation: | Column | What it contains | When to use | |--------|-----------------|-------------| | `attributes.llm.input_messages` | Structured chat messages (system, user, assistant, tool) in role-based format | **Primary source** for chat-based LLM prompts | | `attributes.llm.input_messages.roles` | Array of roles: `system`, `user`, `assistant`, `tool` | Extract individual message roles | | `attributes.llm.input_messages.contents` | Array of message content strings | Extract message text | | `attributes.input.value` | Serialized prompt or user question (generic, all span kinds) | Fallback when structured messages are not available | | `attributes.llm.prompt_template.template` | Template with `{variable}` placeholders (e.g., `"Answer {question} using {context}"`) | When the app uses prompt templates | | `attributes.llm.prompt_template.variables` | Template variable values (JSON object) | See what values were substituted into the template | | `attributes.output.value` | Model response text | See what the LLM produced | | `attributes.llm.output_messages` | Structured model output (including tool calls) | Inspect tool-calling responses | ### Finding Prompts by Span Kind - **LLM span** (`attributes.openinference.span.kind = 'LLM'`): Check `attributes.llm.input_messages` for structured chat messages, OR `attributes.input.value` for a serialized prompt. Check `attributes.llm.prompt_template.template` for the template. - **Chain/Agent span**: `attributes.input.value` contains the user's question. The actual LLM prompt lives on **child LLM spans** -- navigate down the trace tree. - **Tool span**: `attributes.input.value` has tool input, `attributes.output.value` has tool result. Not typically where prompts live. ### Performance Signal Columns These columns carry the feedback data used for optimization: | Column pattern | Source | What it tells you | |---------------|--------|-------------------| | `annotation..label` | Human reviewers | Categorical grade (e.g., `correct`, `incorrect`, `partial`) | | `annotation..score` | Human reviewers | Numeric quality score (e.g., 0.0 - 1.0) | | `annotation..text` | Human reviewers | Freeform explanation of the grade | | `eval..label` | LLM-as-judge evals | Automated categorical assessment | | `eval..score` | LLM-as-judge evals | Automated numeric score | | `eval..explanation` | LLM-as-judge evals | Why the eval gave that score -- **most valuable for optimization** | | `attributes.input.value` | Trace data | What went into the LLM | | `attributes.output.value` | Trace data | What the LLM produced | | `{experiment_name}.output` | Experiment runs | Output from a specific experiment | ## Prerequisites Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront. If an `ax` command fails, troubleshoot based on the error: - `command not found` or version error → see [references/ax-setup.md](references/ax-setup.md) - `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow [references/ax-profiles.md](references/ax-profiles.md) to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys - Space unknown → run `ax spaces list` to pick by name, or ask the user - Project unclear → ask the user, or run `ax projects list -o json --limit 100` and present as selectable options - LLM provider call fails (missing provider credentials) → run `ax ai-integrations list --space SPACE` to check for platform-managed credentials. If none exist, use the **arize-ai-provider-integration** skill — never ask the user to paste a provider key into chat. - **Security:** Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. Never ask the user to paste secrets into chat. For missing credentials, see [references/ax-profiles.md](references/ax-profiles.md). ### When you must ask the user first Still prefer `ax spaces list`, `ax projects list`, `ax datasets list`, `ax experiments list`, and exports over open-ended questions. If you still cannot proceed (e.g. multiple projects match the name the user gave, unclear trace vs experiment path, or destructive scope), **do not** jump straight into questions — use the same explicit framing as **arize-instrumentation** when it stops for scope or confirmation: 1. Acknowledge the skill, e.g.: **I found the arize-prompt-optimization skill in this repo** (you may add `skills/arize-prompt-optimization/SKILL.md` if helpful). 2. Then a clear pause line, e.g.: **A few clarifying questions before I invoke it:** 3. Ask **minimal** numbered or short bullet questions — only what blocks the next `ax` step in this skill. ## Phase 1: Extract the Current Prompt ### Find LLM spans containing prompts ```bash # Sample LLM spans (where prompts live) ax spans export PROJECT --filter "attributes.openinference.span.kind = 'LLM'" -l 10 --stdout # Filter by model ax spans export PROJECT --filter "attributes.llm.model_name = 'gpt-4o'" -l 10 --stdout # Filter by span name (e.g., a specific LLM call) ax spans export PROJECT --filter "name = 'ChatCompletion'" -l 10 --stdout ``` ### Export a trace to inspect prompt structure ```bash # Export all spans in a trace ax spans export PROJECT --trace-id TRACE_ID # Export a single span ax spans export PROJECT --span-id SPAN_ID ``` ### Extract prompts from exported JSON ```bash # Extract structured chat messages (system + user + assistant) jq '.[0] | { messages: .attributes.llm.input_messages, model: .attributes.llm.model_name }' trace_*/spans.json # Extract the system prompt specifically jq '[.[] | select(.attributes.llm.input_messages.roles[]? == "system")] | .[0].attributes.llm.input_messages' trace_*/spans.json # Extract prompt template and variables jq '.[0].attributes.llm.prompt_template' trace_*/spans.json # Extract from input.value (fallback for non-structured prompts) jq '.[0].attributes.input.value' trace_*/spans.json ``` ### Reconstruct the prompt as messages Once you have the span data, reconstruct the prompt as a messages array: ```json [ {"role": "SYSTEM", "content": "You are a helpful assistant that..."}, {"role": "USER", "content": "Given {input}, answer the question: {question}"} ] ``` If the span has `attributes.llm.prompt_template.template`, the prompt uses variables. Preserve these placeholders (`{variable}` or `{{variable}}`) -- they are substituted at runtime. ## Phase 2: Gather Performance Data ### From traces (production feedback) ```bash # Find error spans -- these indicate prompt failures ax spans export PROJECT \ --filter "status_code = 'ERROR' AND attributes.openinference.span.kind = 'LLM'" \ -l 20 --stdout # Find spans with low eval scores ax spans export PROJECT \ --filter "annotation.correctness.label = 'incorrect'" \ -l 20 --stdout # Find spans with high latency (may indicate overly complex prompts) ax spans export PROJECT \ --filter "attributes.openinference.span.kind = 'LLM' AND latency_ms > 10000" \ -l 20 --stdout # Export error traces for detailed inspection ax spans export PROJECT --trace-id TRACE_ID ``` ### From datasets and experiments ```bash # Export a dataset (ground truth examples) ax datasets export DATASET_NAME --space SPACE # -> dataset_*/examples.json # Export experiment results (what the LLM produced) ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE # -> experiment_*/runs.json ``` ### Merge dataset + experiment for analysis Join the two files by `example_id` to see inputs alongside outputs and evaluations: ```bash # Count examples and runs jq 'length' dataset_*/examples.json jq 'length' experiment_*/runs.json # View a single joined record jq -s ' .[0] as $dataset | .[1][0] as $run | ($dataset[] | select(.id == $run.example_id)) as $example | { input: $example, output: $run.output, evaluations: $run.evaluations } ' dataset_*/examples.json experiment_*/runs.json # Find failed examples (where eval score < threshold) jq '[.[] | select(.evaluations.correctness.score < 0.5)]' experiment_*/runs.json ``` ### Identify what to optimize Look for patterns across failures: 1. **Compare outputs to ground truth**: Where does the LLM output differ from expected? 2. **Read eval explanations**: `eval.*.explanation` tells you WHY something failed 3. **Check annotation text**: Human feedback describes specific issues 4. **Look for verbosity mismatches**: If outputs are too long/short vs ground truth 5. **Check format compliance**: Are outputs in the expected format? ## Phase 3: Optimize the Prompt ### The Optimization Meta-Prompt Fill in the three placeholders in [references/optimization-meta-prompt.md](references/optimization-meta-prompt.md) and send it to your LLM (GPT-4o, Claude, etc.) to generate an improved version of the prompt. ### Preparing the performance data Format the records as a JSON array before pasting into the template: ```bash # From dataset + experiment: join and select relevant columns jq -s ' .[0] as $ds | [.[1][] | . as $run | ($ds[] | select(.id == $run.example_id)) as $ex | { input: $ex.input, expected: $ex.expected_output, actual_output: $run.output, eval_score: $run.evaluations.correctness.score, eval_label: $run.evaluations.correctness.label, eval_explanation: $run.evaluations.correctness.explanation } ] ' dataset_*/examples.json experiment_*/runs.json # From exported spans: extract input/output pairs with annotations jq '[.[] | select(.attributes.openinference.span.kind == "LLM") | { input: .attributes.input.value, output: .attributes.output.value, status: .status_code, model: .attributes.llm.model_name }]' trace_*/spans.json ``` ### Applying the revised prompt After the LLM returns the revised messages array: 1. Compare the original and revised prompts side by side 2. Verify all template variables are preserved 3. Check that format instructions are intact 4. Test on a few examples before full deployment ## Phase 4: Iterate ### The optimization loop ``` 1. Extract prompt -> Phase 1 (once) 2. Run experiment -> ax experiments create ... 3. Export results -> ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE 4. Analyze failures -> jq to find low scores 5. Run meta-prompt -> Phase 3 with new failure data 6. Apply revised prompt 7. Repeat from step 2 ``` ### Measure improvement ```bash # Compare scores across experiments # Experiment A (baseline) jq '[.[] | .evaluations.correctness.score] | add / length' experiment_a/runs.json # Experiment B (optimized) jq '[.[] | .evaluations.correctness.score] | add / length' experiment_b/runs.json # Find examples that flipped from fail to pass jq -s ' [.[0][] | select(.evaluations.correctness.label == "incorrect")] as $fails | [.[1][] | select(.evaluations.correctness.label == "correct") | select(.example_id as $id | $fails | any(.example_id == $id)) ] | length ' experiment_a/runs.json experiment_b/runs.json ``` ### A/B compare two prompts 1. Create two experiments against the same dataset, each using a different prompt version 2. Export both: `ax experiments export EXP_A` and `ax experiments export EXP_B` 3. Compare average scores, failure rates, and specific example flips 4. Check for regressions -- examples that passed with prompt A but fail with prompt B ## Prompt Engineering Best Practices Apply these when writing or revising prompts: | Technique | When to apply | Example | |-----------|--------------|---------| | Clear, detailed instructions | Output is vague or off-topic | "Classify the sentiment as exactly one of: positive, negative, neutral" | | Instructions at the beginning | Model ignores later instructions | Put the task description before examples | | Step-by-step breakdowns | Complex multi-step processes | "First extract entities, then classify each, then summarize" | | Specific personas | Need consistent style/tone | "You are a senior financial analyst writing for institutional investors" | | Delimiter tokens | Sections blend together | Use `---`, `###`, or XML tags to separate input from instructions | | Few-shot examples | Output format needs clarification | Show 2-3 synthetic input/output pairs | | Output length specifications | Responses are too long or short | "Respond in exactly 2-3 sentences" | | Reasoning instructions | Accuracy is critical | "Think step by step before answering" | | "I don't know" guidelines | Hallucination is a risk | "If the answer is not in the provided context, say 'I don't have enough information'" | ### Variable preservation When optimizing prompts that use template variables: - **Single braces** (`{variable}`): Python f-string / Jinja style. Most common in Arize. - **Double braces** (`{{variable}}`): Mustache style. Used when the framework requires it. - Never add or remove variable placeholders during optimization - Never rename variables -- the runtime substitution depends on exact names - If adding few-shot examples, use literal values, not variable placeholders ## Workflows ### Optimize a prompt from a failing trace 1. Find a failing trace ID directly from the known filter — skip `ax traces list`'s human-browsing view and go straight to the scriptable path: ```bash ax spans export PROJECT --filter "status_code = 'ERROR'" -l 5 --stdout | jq -r '.[0].context.trace_id' ``` 2. Export the full trace: ```bash ax spans export PROJECT --trace-id TRACE_ID ``` 3. Extract the prompt from the LLM span: ```bash jq '[.[] | select(.attributes.openinference.span.kind == "LLM")][0] | { messages: .attributes.llm.input_messages, template: .attributes.llm.prompt_template, output: .attributes.output.value, error: .attributes.exception.message }' trace_*/spans.json ``` 4. Identify what failed from the error message or output 5. Fill in the optimization meta-prompt (Phase 3) with the prompt and error context 6. Apply the revised prompt ### Optimize using a dataset and experiment 1. Find the dataset and experiment: ```bash ax datasets list --space SPACE ax experiments list --dataset DATASET_NAME --space SPACE ``` 2. Export both: ```bash ax datasets export DATASET_NAME --space SPACE ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE ``` 3. Prepare the joined data for the meta-prompt 4. Run the optimization meta-prompt 5. Create a new experiment with the revised prompt to measure improvement ### Debug a prompt that produces wrong format 1. Export spans where the output format is wrong: ```bash ax spans export PROJECT \ --filter "attributes.openinference.span.kind = 'LLM' AND annotation.format.label = 'incorrect'" \ -l 10 --stdout > bad_format.json ``` 2. Look at what the LLM is producing vs what was expected 3. Add explicit format instructions to the prompt (JSON schema, examples, delimiters) 4. Common fix: add a few-shot example showing the exact desired output format ### Reduce hallucination in a RAG prompt 1. Find traces where the model hallucinated: ```bash ax spans export PROJECT \ --filter "annotation.faithfulness.label = 'unfaithful'" \ -l 20 --stdout ``` 2. Export and inspect the retriever + LLM spans together: ```bash ax spans export PROJECT --trace-id TRACE_ID jq '[.[] | {kind: .attributes.openinference.span.kind, name, input: .attributes.input.value, output: .attributes.output.value}]' trace_*/spans.json ``` 3. Check if the retrieved context actually contained the answer 4. Add grounding instructions to the system prompt: "Only use information from the provided context. If the answer is not in the context, say so." ## Troubleshooting | Problem | Solution | |---------|----------| | `ax: command not found` | See [references/ax-setup.md](references/ax-setup.md) | | `No profile found` | No profile is configured. See [references/ax-profiles.md](references/ax-profiles.md) to create one. | | No `input_messages` on span | Check span kind -- Chain/Agent spans store prompts on child LLM spans, not on themselves | | Prompt template is `null` | Not all instrumentations emit `prompt_template`. Use `input_messages` or `input.value` instead | | Variables lost after optimization | Verify the revised prompt preserves all `{var}` placeholders from the original | | Optimization makes things worse | Check for overfitting -- the meta-prompt may have memorized test data. Ensure few-shot examples are synthetic | | No eval/annotation columns | Run evaluations first (via Arize UI or SDK), then re-export | | Experiment output column not found | The column name is `{experiment_name}.output` -- check exact experiment name via `ax experiments get` | | `jq` errors on span JSON | Ensure you're targeting the correct file path (e.g., `trace_*/spans.json`) |

相关技能

Agent 工作流

Skill Creator

创造有效技能指南。 当用户想创造出新的技能(或更新现有的技能),以专业知识,工作流程,或工具集成来扩展克洛德的能力时,应该使用这种技能.

Agent 工作流

Clawdhub

使用ClawdHub CLI搜索,安装,更新并发布从taladhub.com的代理技能. 需要获取苍蝇上的新技能时使用,将安装的技能同步到最新版本或特定版本,或者发布 npm-instainddhub CLI 的新/更新的技能文件夹.

Agent 工作流

Agent Team Orchestration

管弦乐团多代理团队,任务设定周期,交接协议,审查工作流程. 使用时间: (1)建立2+特派员队伍,具有不同专业,(2)确定任务路线和生命周期(收录框_ spec_建设_审查_完成),(3)在特派员之间制定交接协议,(4)建立审查和质量关口,(5)管理特派员之间的交流和文物共享.

Agent 工作流

Superpowers

Spec-first,TDD,子代理驱动的软件开发工作流程. 当:(1)构建任何新功能或应用——触发脑暴_计划_子代理执行回路,(2)调试出一个bug或测试失败——触发系统性的根起过程,(3)用户说"让我们构建","帮助我计划","我想添加X",或"这个被打破",(4)完成一个功…