跳到主内容
智客 ZICQ

技能库 智客分类:设计创意 agentic-eval

Agentic Eval

评估和改进人工智能代理产出的模式和技术。 在以下情况下使用这一技能: - 实施自我约束和反射循环 - 为质量关键一代构建评价-优化管道 - 创建测试驱动的代码完善工作流程 - 设计基于Rumric或LLM-as-judge评价系统 - 在代理输出(代码、报告、分析)中添加迭接改进 - 测量和提高代理响应质量

10210 安装量

官方网址:skills.sh

技能介绍

先看中文介绍;官方 description 原文单独保留,不改写 SKILL.md。

做什么

评估和改进人工智能代理产出的模式和技术。 在以下情况下使用这一技能: - 实施自我约束和反射循环 - 为质量关键一代构建评价-优化管道 - 创建测试驱动的代码完善工作流程 - 设计基于Rumric或LLM-as-judge评价系统 - 在代理输出(代码、报告、分析)中添加迭接改进 - 测量和提高代理响应质量

何时用

- 质量关键一代**: 代码、报告、分析要求高度准确

代理如何加载

按 Agent Skills 渐进披露:启动时只加载 name 与 description(约 100 token);任务匹配后才读入整份 SKILL.md 正文;scripts/、references/、assets/ 仅在需要时再读。 本文件正文结构:Agentic Evaluation Patterns、Overview、When to Use、Pattern 1: Basic Reflection、Pattern 2: Evaluator-Optimizer、Pattern 3: Code-Specific Reflection。

文件分析

文件分析:这是一份仅含 SKILL.md 的指令型技能,代理激活后整份正文进入上下文。

官方 description(原文)

Patterns and techniques for evaluating and improving AI agent outputs. Use this skill when: - Implementing self-critique and reflection loops - Building evaluator-optimizer pipelines for quality-critical generation - Creating test-driven code refinement workflows - Designing rubric-based or LLM-as-judge evaluation systems - Adding iterative improvement to agent outputs (code, reports, analysis) - Measuring and improving agent response quality

Agentic Evaluation PatternsOverviewWhen to UsePattern 1: Basic ReflectionPattern 2: Evaluator-OptimizerPattern 3: Code-Specific ReflectionEvaluation StrategiesOutcome-BasedLLM-as-JudgeRubric-BasedBest PracticesQuick Start Checklist

来源分类:skills.sh agent-skill

SKILL.md 与 Agent 调用

官方规范 ↗
name
agentic-eval
description
Patterns and techniques for evaluating and improving AI agent outputs. Use this skill when: - Implementing self-critique and reflection loops - Building evaluator-optimizer pipelines for quality-critical generation - Creating test-driven code refinement workflows - Designing rubric-based or LLM-as-judge evaluation systems - Adding iterative improvement to agent outputs (code, reports, analysis) - Measuring and improving agent response quality
  1. 发现技能客户端向 Agent 提供名称与描述目录。
  2. 匹配与调用用户指定或任务匹配后,载入 SKILL.md 指令。
  3. 按需加载按步骤读取参考文档、使用脚本与素材。

具体调用语法与可用工具以目标 Agent 客户端为准。 查看调用机制说明 ↗

安装这个技能

Skills CLI ↗

先选择目标 Agent 和安装范围,保留技能包的附属文件,安装后检查客户端能否发现该技能。

交给 Agent 安装

复制安装指令给支持 Agent Skills 的代理,确认其中的目标目录与客户端匹配。

把 Agent Skill「agentic-eval」安装到我的项目:SKILL.md 原文与官方 description 见 https://zicq.com/zh/skills/skl-8c434c8b00e40462-%E4%BB%A3%E7%90%86%E5%9F%83%E7%93%A6%E5%B0%94.html
请存为 .cursor/skills/agentic-eval/SKILL.md 或 .claude/skills/agentic-eval/SKILL.md,frontmatter 的 name 与 description 保持原样,不要改写。

GitHub 完整包 ↗

终端安装 · Skills CLI

需要 Node.js 与 npx。先查看仓库技能列表,确认实际名称。

npx skills add 'https://github.com/github/awesome-copilot' --list

npx skills add 'https://github.com/github/awesome-copilot' --skill 'agentic-eval'

CLI 会交互选择目标 Agent,默认安装到项目;用户级安装使用 -g。先通过查看命令核对仓库内容,再用 npx skills list 检查已安装技能。

阅读排版

name: agentic-eval description: | Patterns and techniques for evaluating and improving AI agent outputs. Use this skill when:

  • Implementing self-critique and reflection loops
  • Building evaluator-optimizer pipelines for quality-critical generation
  • Creating test-driven code refinement workflows
  • Designing rubric-based or LLM-as-judge evaluation systems
  • Adding iterative improvement to agent outputs (code, reports, analysis)
  • Measuring and improving agent response quality

Agentic Evaluation Patterns

Patterns for self-improvement through iterative evaluation and refinement.

Overview

Evaluation patterns enable agents to assess and improve their own outputs, moving beyond single-shot generation to iterative refinement loops.

Generate → Evaluate → Critique → Refine → Output
    ↑                              │
    └──────────────────────────────┘

When to Use

  • Quality-critical generation: Code, reports, analysis requiring high accuracy
  • Tasks with clear evaluation criteria: Defined success metrics exist
  • Content requiring specific standards: Style guides, compliance, formatting

Pattern 1: Basic Reflection

Agent evaluates and improves its own output through self-critique.

def reflect_and_refine(task: str, criteria: list[str], max_iterations: int = 3) -> str:
    """Generate with reflection loop."""
    output = llm(f"Complete this task:\n{task}")
    
    for i in range(max_iterations):
        # Self-critique
        critique = llm(f"""
        Evaluate this output against criteria: {criteria}
        Output: {output}
        Rate each: PASS/FAIL with feedback as JSON.
        """)
        
        critique_data = json.loads(critique)
        all_pass = all(c["status"] == "PASS" for c in critique_data.values())
        if all_pass:
            return output
        
        # Refine based on critique
        failed = {k: v["feedback"] for k, v in critique_data.items() if v["status"] == "FAIL"}
        output = llm(f"Improve to address: {failed}\nOriginal: {output}")
    
    return output

Key insight: Use structured JSON output for reliable parsing of critique results.


Pattern 2: Evaluator-Optimizer

Separate generation and evaluation into distinct components for clearer responsibilities.

class EvaluatorOptimizer:
    def __init__(self, score_threshold: float = 0.8):
        self.score_threshold = score_threshold
    
    def generate(self, task: str) -> str:
        return llm(f"Complete: {task}")
    
    def evaluate(self, output: str, task: str) -> dict:
        return json.loads(llm(f"""
        Evaluate output for task: {task}
        Output: {output}
        Return JSON: {{"overall_score": 0-1, "dimensions": {{"accuracy": ..., "clarity": ...}}}}
        """))
    
    def optimize(self, output: str, feedback: dict) -> str:
        return llm(f"Improve based on feedback: {feedback}\nOutput: {output}")
    
    def run(self, task: str, max_iterations: int = 3) -> str:
        output = self.generate(task)
        for _ in range(max_iterations):
            evaluation = self.evaluate(output, task)
            if evaluation["overall_score"] >= self.score_threshold:
                break
            output = self.optimize(output, evaluation)
        return output

Pattern 3: Code-Specific Reflection

Test-driven refinement loop for code generation.

class CodeReflector:
    def reflect_and_fix(self, spec: str, max_iterations: int = 3) -> str:
        code = llm(f"Write Python code for: {spec}")
        tests = llm(f"Generate pytest tests for: {spec}\nCode: {code}")
        
        for _ in range(max_iterations):
            result = run_tests(code, tests)
            if result["success"]:
                return code
            code = llm(f"Fix error: {result['error']}\nCode: {code}")
        return code

Evaluation Strategies

Outcome-Based

Evaluate whether output achieves the expected result.

def evaluate_outcome(task: str, output: str, expected: str) -> str:
    return llm(f"Does output achieve expected outcome? Task: {task}, Expected: {expected}, Output: {output}")

LLM-as-Judge

Use LLM to compare and rank outputs.

def llm_judge(output_a: str, output_b: str, criteria: str) -> str:
    return llm(f"Compare outputs A and B for {criteria}. Which is better and why?")

Rubric-Based

Score outputs against weighted dimensions.

RUBRIC = {
    "accuracy": {"weight": 0.4},
    "clarity": {"weight": 0.3},
    "completeness": {"weight": 0.3}
}

def evaluate_with_rubric(output: str, rubric: dict) -> float:
    scores = json.loads(llm(f"Rate 1-5 for each dimension: {list(rubric.keys())}\nOutput: {output}"))
    return sum(scores[d] * rubric[d]["weight"] for d in rubric) / 5

Best Practices

| Practice | Rationale | |----------|-----------| | Clear criteria | Define specific, measurable evaluation criteria upfront | | Iteration limits | Set max iterations (3-5) to prevent infinite loops | | Convergence check | Stop if output score isn't improving between iterations | | Log history | Keep full trajectory for debugging and analysis | | Structured output | Use JSON for reliable parsing of evaluation results |


Quick Start Checklist

## Evaluation Implementation Checklist

### Setup
- [ ] Define evaluation criteria/rubric
- [ ] Set score threshold for "good enough"
- [ ] Configure max iterations (default: 3)

### Implementation
- [ ] Implement generate() function
- [ ] Implement evaluate() function with structured output
- [ ] Implement optimize() function
- [ ] Wire up the refinement loop

### Safety
- [ ] Add convergence detection
- [ ] Log all iterations for debugging
- [ ] Handle evaluation parse failures gracefully

相关技能

设计创意

Automation Workflows

设计和实施自动化工作流程,以节省时间和规模化操作作为独家. 用于识别重复任务实现自动化,构建跨工具的工作流程,设置触发器和行动,或优化现有自动化. 包括自动化机会识别,工作流程设计,工具选择(Zapier,Make,n8n),测试,和维护. 触发"自动","自动","工作流程自动…

设计创意

Ui Ux Pro Max

UI/UX设计智能及建筑抛光接口实施指导. 当用户要求UI设计,UX流量,信息架构,视觉风格方向,设计系统/托盘,组件规格,副本/显微镜,可访问性,或生成/critique/refine前端UI(HTML/CSS/JS,React,Next.js,Vue,Svelte,Tailw…

设计创意

Frontend Design

创建美丽现代UI的专家前端设计指南. 在构建起落架页面,仪表板,或任何用户界面时使用.

设计创意

N8n Workflow Automation

设计和输出 n8n 工作流程 JSON 有强力触发器, idempotency, 错误处理, 记录, 重试, 以及 人入"一站"审查队列. 当您需要可审计的自动化时使用.