arXiv Introduces Seq@K: Evaluating Language Models Under Sequential Feedback

Summary:arXiv introduces Seq@K, a novel evaluation protocol designed to assess language models' ability to iteratively refine their responses based on feedback. Unlike traditional metrics like Pass@K, Seq@K evaluates a model's performance over K sequential attempts, incorporating feedback after each failure. The study demonstrates that Seq@K provides a more nuanced understanding of model behavior in feedback-driven tasks, revealing different model rankings and Pareto frontiers compared to Pass@K. This n

---

Background and Motivation

In modern agentic workflows, language models are expected to iteratively refine their outputs based on feedback. However, traditional evaluation metrics like Pass@K fail to capture the nuances of model performance in such feedback-driven tasks. Seq@K addresses this gap by evaluating a model's ability to solve a problem within K sequential attempts, incorporating feedback after each failure, thus providing a more accurate assessment of model performance.

Technical Mechanism Analysis

The core mechanisms of Seq@K include:

1. Feedback-driven iterative attempts: The model receives feedback after each failure and uses it to improve its responses.
2. K-attempt limitation: The model is evaluated based on its ability to solve the problem within K attempts, assessing both efficiency and accuracy.
3. Multi-benchmark validation: Seq@K is validated across 7 benchmarks and 16 frontier models, ensuring its general applicability and effectiveness.

Engineering Trade-offs and Performance

• Advantages: Seq@K provides a more accurate reflection of model performance in feedback-driven tasks, revealing differences in model behavior during iterative refinement.
• Challenges: Implementing Seq@K requires more complex experimental design and data collection processes, increasing the complexity of evaluation.
• Performance: In multiple benchmarks, Seq@K revealed different model rankings and Pareto frontiers compared to Pass@K, demonstrating its superiority in evaluating models' iterative improvement capabilities.

Developer Deployment and Implementation Recommendations

1. Model Selection Optimization: Use Seq@K to evaluate model performance in feedback-driven tasks, enabling better model configuration selection.
2. Feedback Design Improvement: Leverage Seq@K evaluation results to refine feedback design, enhancing the efficiency of model iterative improvement.
3. Resource Allocation Adjustment: In resource-constrained environments, Seq@K can help developers better allocate computational resources, optimizing model training and inference processes.

Conclusion

Seq@K offers a more accurate evaluation method for assessing language models' performance in feedback-driven tasks. Its application not only aids in optimizing model selection and feedback design but also provides new insights for future AI system development.

---

Source: ArXiv AI (cs.AI) (2026-10-30)

— END —

Tags: #arXiv Follow #Language Model Follow #Evaluation Protocol Follow #Feedback Iteration Follow #Seq@K Follow