跳到主内容
智客 ZICQ

技能库 智客分类:写作与研究 promptfoo-evaluation

Promptfoo Evaluation

使用 Powerfoo 框架配置和运行 LLM 评价 。 设置即时测试时使用, 创建评价配置( promptfooconfig. yaml) , 写入 Python 自定义断言, 为 LLM- as- judge 执行 llm- rubric , 或在 points 中管理 right- shoot 示例 。 在"promptfoo","eval","LLM评价","即时测试"或"模型比较"等关键词上触发.

1366 安装量

官方网址:skills.sh

技能介绍

先看中文介绍;官方 description 原文单独保留,不改写 SKILL.md。

做什么

使用 Powerfoo 框架配置和运行 LLM 评价

何时用

设置快速测试, 创建评价配置( promptfooconfig. yaml) , 写入 Python 自定义断言, 执行 LLM- as- judge 的 llm- rubric , 或在 points 中管理少部分镜头实例 。 在"promptfoo","eval","LLM评价","即时测试"等关键词上触发,

代理如何加载

按 Agent Skills 渐进披露:启动时只加载 name 与 description(约 100 token);任务匹配后才读入整份 SKILL.md 正文;scripts/、references/、assets/ 仅在需要时再读。 本文件正文结构:Promptfoo Evaluation、Overview、Quick Start、Initialize a new evaluation project、Run evaluation、View results in browser。

文件分析

文件分析:除 SKILL.md 外,正文引用了 scripts/metrics.py、scripts/check.py、references/promptfoo_api.md,属于带资源的技能包,这些文件按需再读。

官方 description(原文)

Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promptfoo", "eval", "LLM evaluation", "prompt testing", or "model comparison".

Promptfoo EvaluationOverviewQuick StartInitialize a new evaluation projectRun evaluationView results in browserConfiguration StructureCore Configuration (promptfooconfig.yaml)yaml-language-server: $schema=https://promptfoo.dev/config-schema.jsonPrompts to testModels to compareTest cases

来源分类:skills.sh agent-skill

SKILL.md 与 Agent 调用

官方规范 ↗
name
promptfoo-evaluation
description
Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promptfoo", "eval", "LLM evaluation", "prompt testing", or "model comparison".
  1. 发现技能客户端向 Agent 提供名称与描述目录。
  2. 匹配与调用用户指定或任务匹配后,载入 SKILL.md 指令。
  3. 按需加载按步骤读取参考文档、使用脚本与素材。
指令中引用的文件 · 3
  • scripts/metrics.py
  • scripts/check.py
  • references/promptfoo_api.md

以下路径提取自原文;文件是否齐全请以来源仓库中的完整目录为准。

具体调用语法与可用工具以目标 Agent 客户端为准。 查看调用机制说明 ↗

安装这个技能

Skills CLI ↗

先选择目标 Agent 和安装范围,保留技能包的附属文件,安装后检查客户端能否发现该技能。

该技能引用了附属文件,请从来源获取完整目录;仅复制 SKILL.md 可能缺少依赖。

交给 Agent 安装

复制安装指令给支持 Agent Skills 的代理,确认其中的目标目录与客户端匹配。

把 Agent Skill「promptfoo-evaluation」安装到我的项目:SKILL.md 原文与官方 description 见 https://zicq.com/zh/skills/skl-f41515c4ff8ad51a-%E5%8D%B3%E6%97%B6%E8%AF%84%E4%BB%B7.html
请存为 .cursor/skills/promptfoo-evaluation/SKILL.md 或 .claude/skills/promptfoo-evaluation/SKILL.md,frontmatter 的 name 与 description 保持原样,不要改写。
该技能还带 scripts/、references/、assets/ 等文件,请从 https://github.com/daymade/claude-code-skills 取完整目录,不要只建一个 SKILL.md。

GitHub 完整包 ↗

终端安装 · Skills CLI

需要 Node.js 与 npx。先查看仓库技能列表,确认实际名称。

npx skills add 'https://github.com/daymade/claude-code-skills' --list

npx skills add 'https://github.com/daymade/claude-code-skills' --skill 'promptfoo-evaluation'

CLI 会交互选择目标 Agent,默认安装到项目;用户级安装使用 -g。先通过查看命令核对仓库内容,再用 npx skills list 检查已安装技能。

阅读排版
--- name: promptfoo-evaluation description: Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promptfoo", "eval", "LLM evaluation", "prompt testing", or "model comparison". --- # Promptfoo Evaluation ## Overview This skill provides guidance for configuring and running LLM evaluations using [Promptfoo](https://www.promptfoo.dev/), an open-source CLI tool for testing and comparing LLM outputs. ## Quick Start ```bash # Initialize a new evaluation project npx promptfoo@latest init # Run evaluation npx promptfoo@latest eval # View results in browser npx promptfoo@latest view ``` ## Configuration Structure A typical Promptfoo project structure: ``` project/ ├── promptfooconfig.yaml # Main configuration ├── prompts/ │ ├── system.md # System prompt │ └── chat.json # Chat format prompt ├── tests/ │ └── cases.yaml # Test cases └── scripts/ └── metrics.py # Custom Python assertions ``` ## Core Configuration (promptfooconfig.yaml) ```yaml # yaml-language-server: $schema=https://promptfoo.dev/config-schema.json description: "My LLM Evaluation" # Prompts to test prompts: - file://prompts/system.md - file://prompts/chat.json # Models to compare providers: - id: anthropic:messages:claude-sonnet-4-6 label: Claude-Sonnet-4.6 - id: openai:gpt-4.1 label: GPT-4.1 # Test cases tests: file://tests/cases.yaml # Concurrency control (MUST be under commandLineOptions, NOT top-level) commandLineOptions: maxConcurrency: 2 # Default assertions for all tests defaultTest: assert: - type: python value: file://scripts/metrics.py:custom_assert - type: llm-rubric value: | Evaluate the response quality on a 0-1 scale. threshold: 0.7 # Output path outputPath: results/eval-results.json ``` ## Prompt Formats ### Text Prompt (system.md) ```markdown You are a helpful assistant. Task: {{task}} Context: {{context}} ``` ### Chat Format (chat.json) ```json [ {"role": "system", "content": "{{system_prompt}}"}, {"role": "user", "content": "{{user_input}}"} ] ``` ### Few-Shot Pattern Embed examples directly in prompt or use chat format with assistant messages: ```json [ {"role": "system", "content": "{{system_prompt}}"}, {"role": "user", "content": "Example input: {{example_input}}"}, {"role": "assistant", "content": "{{example_output}}"}, {"role": "user", "content": "Now process: {{actual_input}}"} ] ``` ## Test Cases (tests/cases.yaml) ```yaml - description: "Test case 1" vars: system_prompt: file://prompts/system.md user_input: "Hello world" # Load content from files context: file://data/context.txt assert: - type: contains value: "expected text" - type: python value: file://scripts/metrics.py:custom_check threshold: 0.8 ``` ## Python Custom Assertions Create a Python file for custom assertions (e.g., `scripts/metrics.py`): ```python def get_assert(output: str, context: dict) -> dict: """Default assertion function.""" vars_dict = context.get('vars', {}) # Access test variables expected = vars_dict.get('expected', '') # Return result return { "pass": expected in output, "score": 0.8, "reason": "Contains expected content", "named_scores": {"relevance": 0.9} } def custom_check(output: str, context: dict) -> dict: """Custom named assertion.""" word_count = len(output.split()) passed = 100 <= word_count <= 500 return { "pass": passed, "score": min(1.0, word_count / 300), "reason": f"Word count: {word_count}" } ``` **Key points:** - Default function name is `get_assert` - Specify function with `file://path.py:function_name` - Return `bool`, `float` (score), or `dict` with pass/score/reason - Access variables via `context['vars']` ## LLM-as-Judge (llm-rubric) ```yaml assert: - type: llm-rubric value: | Evaluate the response based on: 1. Accuracy of information 2. Clarity of explanation 3. Completeness Score 0.0-1.0 where 0.7+ is passing. threshold: 0.7 provider: openai:gpt-4.1 # Optional: override grader model ``` **When using a relay/proxy API**, each `llm-rubric` assertion needs its own `provider` config with `apiBaseUrl`. Otherwise the grader falls back to the default Anthropic/OpenAI endpoint and gets 401 errors: ```yaml assert: - type: llm-rubric value: | Evaluate quality on a 0-1 scale. threshold: 0.7 provider: id: anthropic:messages:claude-sonnet-4-6 config: apiBaseUrl: https://your-relay.example.com/api ``` **Best practices:** - Provide clear scoring criteria - Use `threshold` to set minimum passing score - Default grader uses available API keys (OpenAI → Anthropic → Google) - **When using relay/proxy**: every `llm-rubric` must have its own `provider` with `apiBaseUrl` — the main provider's `apiBaseUrl` is NOT inherited ## Common Assertion Types | Type | Usage | Example | |------|-------|---------| | `contains` | Check substring | `value: "hello"` | | `icontains` | Case-insensitive | `value: "HELLO"` | | `equals` | Exact match | `value: "42"` | | `regex` | Pattern match | `value: "\\d{4}"` | | `python` | Custom logic | `value: file://script.py` | | `llm-rubric` | LLM grading | `value: "Is professional"` | | `latency` | Response time | `threshold: 1000` | ## File References All `file://` paths are resolved relative to `promptfooconfig.yaml` location (NOT the YAML file containing the reference). This is a common gotcha when `tests:` references a separate YAML file — the `file://` paths inside that test file still resolve from the config root. ```yaml # Load file content as variable vars: content: file://data/input.txt # Load prompt from file prompts: - file://prompts/main.md # Load test cases from file tests: file://tests/cases.yaml # Load Python assertion assert: - type: python value: file://scripts/check.py:validate ``` ## Running Evaluations ```bash # Basic run npx promptfoo@latest eval # With specific config npx promptfoo@latest eval --config path/to/config.yaml # Output to file npx promptfoo@latest eval --output results.json # Filter tests npx promptfoo@latest eval --filter-metadata category=math # View results npx promptfoo@latest view ``` ## Relay / Proxy API Configuration When using an API relay or proxy instead of direct Anthropic/OpenAI endpoints: ```yaml providers: - id: anthropic:messages:claude-sonnet-4-6 label: Claude-Sonnet-4.6 config: max_tokens: 4096 apiBaseUrl: https://your-relay.example.com/api # Promptfoo appends /v1/messages # CRITICAL: maxConcurrency MUST be under commandLineOptions (NOT top-level) commandLineOptions: maxConcurrency: 1 # Respect relay rate limits ``` **Key rules:** - `apiBaseUrl` goes in `providers[].config` — Promptfoo appends `/v1/messages` automatically - `maxConcurrency` must be under `commandLineOptions:` — placing it at top level is silently ignored - When using relay with LLM-as-judge, set `maxConcurrency: 1` to avoid concurrent request limits (generation + grading share the same pool) - Pass relay token as `ANTHROPIC_API_KEY` env var ## Troubleshooting **Python not found:** ```bash export PROMPTFOO_PYTHON=python3 ``` **Large outputs truncated:** Outputs over 30000 characters are truncated. Use `head_limit` in assertions. **File not found errors:** All `file://` paths resolve relative to `promptfooconfig.yaml` location. **maxConcurrency ignored (shows "up to N at a time"):** `maxConcurrency` must be under `commandLineOptions:`, not at the YAML top level. This is a common mistake. **LLM-as-judge returns 401 with relay API:** Each `llm-rubric` assertion must have its own `provider` with `apiBaseUrl`. The main provider config is not inherited by grader assertions. **HTML tags in model output inflating metrics:** Models may output `
`, ``, etc. in structured content. Strip HTML in Python assertions before measuring: ```python import re clean_text = re.sub(r'<[^>]+>', '', raw_text) ``` ## Echo Provider (Preview Mode) Use the **echo provider** to preview rendered prompts without making API calls: ```yaml # promptfooconfig-preview.yaml providers: - echo # Returns prompt as output, no API calls tests: - vars: input: "test content" ``` **Use cases:** - Preview prompt rendering before expensive API calls - Verify Few-shot examples are loaded correctly - Debug variable substitution issues - Validate prompt structure ```bash # Run preview mode npx promptfoo@latest eval --config promptfooconfig-preview.yaml ``` **Cost:** Free - no API tokens consumed. ## Advanced Few-Shot Implementation ### Multi-turn Conversation Pattern For complex few-shot learning with full examples: ```json [ {"role": "system", "content": "{{system_prompt}}"}, // Few-shot Example 1 {"role": "user", "content": "Task: {{example_input_1}}"}, {"role": "assistant", "content": "{{example_output_1}}"}, // Few-shot Example 2 (optional) {"role": "user", "content": "Task: {{example_input_2}}"}, {"role": "assistant", "content": "{{example_output_2}}"}, // Actual test {"role": "user", "content": "Task: {{actual_input}}"} ] ``` **Test case configuration:** ```yaml tests: - vars: system_prompt: file://prompts/system.md # Few-shot examples example_input_1: file://data/examples/input1.txt example_output_1: file://data/examples/output1.txt example_input_2: file://data/examples/input2.txt example_output_2: file://data/examples/output2.txt # Actual test actual_input: file://data/test1.txt ``` **Best practices:** - Use 1-3 few-shot examples (more may dilute effectiveness) - Ensure examples match the task format exactly - Load examples from files for better maintainability - Use echo provider first to verify structure ## Long Text Handling For Chinese/long-form content evaluations (10k+ characters): **Configuration:** ```yaml providers: - id: anthropic:messages:claude-sonnet-4-6 config: max_tokens: 8192 # Increase for long outputs defaultTest: assert: - type: python value: file://scripts/metrics.py:check_length ``` **Python assertion for text metrics:** ```python import re def strip_tags(text: str) -> str: """Remove HTML tags for pure text.""" return re.sub(r'<[^>]+>', '', text) def check_length(output: str, context: dict) -> dict: """Check output length constraints.""" raw_input = context['vars'].get('raw_input', '') input_len = len(strip_tags(raw_input)) output_len = len(strip_tags(output)) reduction_ratio = 1 - (output_len / input_len) if input_len > 0 else 0 return { "pass": 0.7 <= reduction_ratio <= 0.9, "score": reduction_ratio, "reason": f"Reduction: {reduction_ratio:.1%} (target: 70-90%)", "named_scores": { "input_length": input_len, "output_length": output_len, "reduction_ratio": reduction_ratio } } ``` ## Real-World Example **Project:** Chinese short-video content curation from long transcripts **Structure:** ``` tiaogaoren/ ├── promptfooconfig.yaml # Production config ├── promptfooconfig-preview.yaml # Preview config (echo provider) ├── prompts/ │ ├── tiaogaoren-prompt.json # Chat format with few-shot │ └── v4/system-v4.md # System prompt ├── tests/cases.yaml # 3 test samples ├── scripts/metrics.py # Custom metrics (reduction ratio, etc.) ├── data/ # 5 samples (2 few-shot, 3 eval) └── results/ ``` **See:** `./tiaogaoren/` (example project root) for full implementation. ## Resources For detailed API reference and advanced patterns, see [references/promptfoo_api.md](references/promptfoo_api.md).

相关技能

写作与研究

Humanizer

从文本中删除 AI 生成的写入标记 。 编辑或审查文本时使用,使其声音更自然和人文写作. 基于维基百科的全面"AI写作的标志"指南. 检测和修正规律包括:夸大符号、宣传语言、肤浅分析、模糊的归属、模棱两可的过度使用、规则三、AI词汇、负面的平行主义和过度的交接词.

写作与研究

Prismfy Search

OpenClaw 的默认网络搜索 。 搜索网络跨越了10个引擎——Google,Reddit,GitHub,arXiv,Hacker News等——使用Prismfy. 包括免费等级,不需要信用卡。 包括用于网络搜索、配额检查和引擎/时间/域过滤器的捆绑 " search.sh …

写作与研究

Blogwatcher

监控博客和RSS/Atom的种子.

写作与研究

Tavily

AI-优化了使用Tavily Search API的网络搜索. 需要全面网络研究,时事搜索,域名特定搜索,或AI生成的回答摘要时使用. Tavily是LLM消费的优化型,具有清洁的结构化结果,答案生成,以及原始内容提取. 最适合研究任务、新闻查询、实况调查和收集权威来源.