跳到主内容
智客 ZICQ

技能库 智客分类:Agent 工作流 eval-harness

Eval Harness

AI编码会话的Eval驱动开发(EDD)框架——在编码前定义能力和回归evals,分级以代码为主,以模型为主,规则为主,或以人为分级器为主,并有轨通@k和pass^k可靠性. 在定义代理任务的通过/失败标准时使用,测量代理的可靠性,构建用于即时或代理更改的回归套件,或者跨模型版本的基准.

12848 安装量

官方网址:skills.sh

技能介绍

先看中文介绍;官方 description 原文单独保留,不改写 SKILL.md。

做什么

AI编码会话的Eval驱动开发(EDD)框架——在编码前定义能力和回归evals,分级以代码为主,以模型为主,规则为主,或以人为分级器为主,以及 track pass@k和pass^k可靠性

何时用

定义代理任务的通过/失效标准,测量代理的可靠性,建立用于迅速或代理变化的回归套件,或跨模型版本的基准

代理如何加载

按 Agent Skills 渐进披露:启动时只加载 name 与 description(约 100 token);任务匹配后才读入整份 SKILL.md 正文;scripts/、references/、assets/ 仅在需要时再读。 本文件正文结构:Eval Harness Skill、When to Activate、Philosophy、Eval Types、Capability Evals、Regression Evals。

文件分析

文件分析:除 SKILL.md 外,正文引用了 scripts/lib/eval-harness/、scripts/eval-harness.js,属于带资源的技能包,这些文件按需再读。

官方 description(原文)

Eval-driven development (EDD) framework for AI coding sessions — define capability and regression evals before coding, grade with code-based, model-based, rule, or human graders, and track pass@k and pass^k reliability. Use when defining pass/fail criteria for agent tasks, measuring agent reliability, building regression suites for prompt or agent changes, or benchmarking across model versions.

Eval Harness SkillWhen to ActivatePhilosophyEval TypesCapability EvalsRegression EvalsGrader Types1. Code-Based GraderCheck if file contains expected patternCheck if tests passCheck if build succeeds2. Model-Based Grader

来源分类:skills.sh agent-skill

SKILL.md 与 Agent 调用

官方规范 ↗
name
eval-harness
description
Eval-driven development (EDD) framework for AI coding sessions — define capability and regression evals before coding, grade with code-based, model-based, rule, or human graders, and track pass@k and pass^k reliability. Use when defining pass/fail criteria for agent tasks, measuring agent reliability, building regression suites for prompt or agent changes, or benchmarking across model versions.
  1. 发现技能客户端向 Agent 提供名称与描述目录。
  2. 匹配与调用用户指定或任务匹配后,载入 SKILL.md 指令。
  3. 按需加载按步骤读取参考文档、使用脚本与素材。
指令中引用的文件 · 2
  • scripts/lib/eval-harness/
  • scripts/eval-harness.js

以下路径提取自原文;文件是否齐全请以来源仓库中的完整目录为准。

具体调用语法与可用工具以目标 Agent 客户端为准。 查看调用机制说明 ↗

安装这个技能

Skills CLI ↗

先选择目标 Agent 和安装范围,保留技能包的附属文件,安装后检查客户端能否发现该技能。

该技能引用了附属文件,请从来源获取完整目录;仅复制 SKILL.md 可能缺少依赖。

交给 Agent 安装

复制安装指令给支持 Agent Skills 的代理,确认其中的目标目录与客户端匹配。

把 Agent Skill「eval-harness」安装到我的项目:SKILL.md 原文与官方 description 见 https://zicq.com/zh/skills/skl-bc82b4d43b8a3aa4-Eval-%E8%B0%83%E5%92%8C.html
请存为 .cursor/skills/eval-harness/SKILL.md 或 .claude/skills/eval-harness/SKILL.md,frontmatter 的 name 与 description 保持原样,不要改写。
该技能还带 scripts/、references/、assets/ 等文件,请从 https://github.com/affaan-m/ecc 取完整目录,不要只建一个 SKILL.md。

GitHub 完整包 ↗

终端安装 · Skills CLI

需要 Node.js 与 npx。先查看仓库技能列表,确认实际名称。

npx skills add 'https://github.com/affaan-m/ecc' --list

npx skills add 'https://github.com/affaan-m/ecc' --skill 'eval-harness'

CLI 会交互选择目标 Agent,默认安装到项目;用户级安装使用 -g。先通过查看命令核对仓库内容,再用 npx skills list 检查已安装技能。

阅读排版
--- name: eval-harness description: Eval-driven development (EDD) framework for AI coding sessions — define capability and regression evals before coding, grade with code-based, model-based, rule, or human graders, and track pass@k and pass^k reliability. Use when defining pass/fail criteria for agent tasks, measuring agent reliability, building regression suites for prompt or agent changes, or benchmarking across model versions. metadata: origin: ECC tools: Read, Write, Edit, Bash, Grep, Glob --- # Eval Harness Skill A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles. ## When to Activate - Setting up eval-driven development (EDD) for AI-assisted workflows - Defining pass/fail criteria for Claude Code task completion - Measuring agent reliability with pass@k metrics - Creating regression test suites for prompt or agent changes - Benchmarking agent performance across model versions ## Philosophy Eval-Driven Development treats evals as the "unit tests of AI development": - Define expected behavior BEFORE implementation - Run evals continuously during development - Track regressions with each change - Use pass@k metrics for reliability measurement ## Eval Types ### Capability Evals Test if Claude can do something it couldn't before: ```markdown [CAPABILITY EVAL: feature-name] Task: Description of what Claude should accomplish Success Criteria: - [ ] Criterion 1 - [ ] Criterion 2 - [ ] Criterion 3 Expected Output: Description of expected result ``` ### Regression Evals Ensure changes don't break existing functionality: ```markdown [REGRESSION EVAL: feature-name] Baseline: SHA or checkpoint name Tests: - existing-test-1: PASS/FAIL - existing-test-2: PASS/FAIL - existing-test-3: PASS/FAIL Result: X/Y passed (previously Y/Y) ``` ## Grader Types ### 1. Code-Based Grader Deterministic checks using code: ```bash # Check if file contains expected pattern grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL" # Check if tests pass npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL" # Check if build succeeds npm run build && echo "PASS" || echo "FAIL" ``` ### 2. Model-Based Grader Use Claude to evaluate open-ended outputs: ```markdown [MODEL GRADER PROMPT] Evaluate the following code change: 1. Does it solve the stated problem? 2. Is it well-structured? 3. Are edge cases handled? 4. Is error handling appropriate? Score: 1-5 (1=poor, 5=excellent) Reasoning: [explanation] ``` ### 3. Human Grader Flag for manual review: ```markdown [HUMAN REVIEW REQUIRED] Change: Description of what changed Reason: Why human review is needed Risk Level: LOW/MEDIUM/HIGH ``` ## Metrics ### pass@k "At least one success in k attempts" - pass@1: First attempt success rate - pass@3: Success within 3 attempts - Typical target: pass@3 > 90% ### pass^k "All k trials succeed" - Higher bar for reliability - pass^3: 3 consecutive successes - Use for critical paths ## Eval Workflow ### 1. Define (Before Coding) ```markdown ## EVAL DEFINITION: feature-xyz ### Capability Evals 1. Can create new user account 2. Can validate email format 3. Can hash password securely ### Regression Evals 1. Existing login still works 2. Session management unchanged 3. Logout flow intact ### Success Metrics - pass@3 > 90% for capability evals - pass^3 = 100% for regression evals ``` ### 2. Implement Write code to pass the defined evals. ### 3. Evaluate ```bash # Run capability evals [Run each capability eval, record PASS/FAIL] # Run regression evals npm test -- --testPathPattern="existing" # Generate report ``` ### 4. Report ```markdown EVAL REPORT: feature-xyz ======================== Capability Evals: create-user: PASS (pass@1) validate-email: PASS (pass@2) hash-password: PASS (pass@1) Overall: 3/3 passed Regression Evals: login-flow: PASS session-mgmt: PASS logout-flow: PASS Overall: 3/3 passed Metrics: pass@1: 67% (2/3) pass@3: 100% (3/3) Status: READY FOR REVIEW ``` ## Integration Patterns ### Pre-Implementation ``` /eval define feature-name ``` Creates eval definition file at `.claude/evals/feature-name.md` ### During Implementation ``` /eval check feature-name ``` Runs current evals and reports status ### Post-Implementation ``` /eval report feature-name ``` Generates full eval report ## Eval Storage Store evals in project: ``` .claude/ evals/ feature-xyz.md # Eval definition feature-xyz.log # Eval run history baseline.json # Regression baselines ``` ## Best Practices 1. **Define evals BEFORE coding** - Forces clear thinking about success criteria 2. **Run evals frequently** - Catch regressions early 3. **Track pass@k over time** - Monitor reliability trends 4. **Use code graders when possible** - Deterministic > probabilistic 5. **Human review for security** - Never fully automate security checks 6. **Keep evals fast** - Slow evals don't get run 7. **Version evals with code** - Evals are first-class artifacts ## Example: Adding Authentication ```markdown ## EVAL: add-authentication ### Phase 1: Define (10 min) Capability Evals: - [ ] User can register with email/password - [ ] User can login with valid credentials - [ ] Invalid credentials rejected with proper error - [ ] Sessions persist across page reloads - [ ] Logout clears session Regression Evals: - [ ] Public routes still accessible - [ ] API responses unchanged - [ ] Database schema compatible ### Phase 2: Implement (varies) [Write code] ### Phase 3: Evaluate Run: /eval check add-authentication ### Phase 4: Report EVAL REPORT: add-authentication ============================== Capability: 5/5 passed (pass@3: 100%) Regression: 3/3 passed (pass^3: 100%) Status: SHIP IT ``` ## Local Framework Utilities The mechanical utilities ship in `scripts/lib/eval-harness/`: ```sh node scripts/eval-harness.js example ``` - Capsule: hash-linked journal with five lineages and local integrity checks. - Inspection: source digests, validated variant paths, and syntactic warnings. - Replay: declared tools and content-addressed fixtures. Missing fixtures fail closed; SE3 and above are refused in replay. Record mode invokes the registered implementation, so only register trusted functions. - Receipt: offline verification of capsule and artifact bytes, with named checks. - Retrospective preparation: `node scripts/eval-harness.js capsule group [ ...]` groups 1 to 100 explicitly selected, verified local capsule snapshots from one task family by declared harness version. Repeated snapshots count once; conflicting identities or invalid capsules reject the whole report. This is read-only record counting, with no new rollouts, scores or promotion. Use small, quiescent capsules. Payloads, directory arguments and raw run/capsule IDs are omitted, but task-family/version labels are verbatim and digest references are linkable; review them before sharing. Operational validation remains pending. Candidate execution is disabled on every OS because no verified OS containment backend is implemented. `gate run`, `runGate`, `runVariant`, direct child launch, and the retired effect preload refuse with `gate.isolation_required`. No trust flag or caller-supplied executor can bypass the refusal. The example records that refusal and inspects source without executing or scoring it. Do not present static warnings, a capsule receipt, or successful utility tests as candidate containment or promotion evidence. A future gate requires an independently reviewed OS boundary, protected checker and audit channels, and fatal baseline rejection. See `docs/architecture/eval-harness-frameworks.md`. ## Product Evals (v1.8) Use product evals when behavior quality cannot be captured by unit tests alone. ### Grader Types 1. Code grader (deterministic assertions) 2. Rule grader (regex/schema constraints) 3. Model grader (LLM-as-judge rubric) 4. Human grader (manual adjudication for ambiguous outputs) ### pass@k Guidance - `pass@1`: direct reliability - `pass@3`: practical reliability under controlled retries - `pass^3`: stability test (all 3 runs must pass) Recommended thresholds: - Capability evals: pass@3 >= 0.90 - Regression evals: pass^3 = 1.00 for release-critical paths ### Eval Anti-Patterns - Overfitting prompts to known eval examples - Measuring only happy-path outputs - Ignoring cost and latency drift while chasing pass rates - Allowing flaky graders in release gates ### Minimal Eval Artifact Layout - `.claude/evals/.md` definition - `.claude/evals/.log` run history - `docs/releases//eval-summary.md` release snapshot

相关技能

Agent 工作流

Skill Creator

创造有效技能指南。 当用户想创造出新的技能(或更新现有的技能),以专业知识,工作流程,或工具集成来扩展克洛德的能力时,应该使用这种技能.

Agent 工作流

Clawdhub

使用ClawdHub CLI搜索,安装,更新并发布从taladhub.com的代理技能. 需要获取苍蝇上的新技能时使用,将安装的技能同步到最新版本或特定版本,或者发布 npm-instainddhub CLI 的新/更新的技能文件夹.

Agent 工作流

Agent Team Orchestration

管弦乐团多代理团队,任务设定周期,交接协议,审查工作流程. 使用时间: (1)建立2+特派员队伍,具有不同专业,(2)确定任务路线和生命周期(收录框_ spec_建设_审查_完成),(3)在特派员之间制定交接协议,(4)建立审查和质量关口,(5)管理特派员之间的交流和文物共享.

Agent 工作流

Superpowers

Spec-first,TDD,子代理驱动的软件开发工作流程. 当:(1)构建任何新功能或应用——触发脑暴_计划_子代理执行回路,(2)调试出一个bug或测试失败——触发系统性的根起过程,(3)用户说"让我们构建","帮助我计划","我想添加X",或"这个被打破",(4)完成一个功…