ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #ArXiv #Multi-Expert Aggregation #Conformal Risk Control #LLM-as-a-Judge #Open-Ended Dialogue

ArXiv Proposes Multi-Expert Conformal Risk Control for Pairwise LLM-as-a-Judge Evaluation in Open-Ended Dialogue

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 6 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:The ArXiv team introduces a novel algorithm called Multi-Expert Conformal Risk Control (CRC) for pairwise LLM-as-a-Judge evaluation in open-ended dialogue. This method enhances CRC by aggregating multiple experts, with two strategies—Score Averaging and Decision Voting—designed for score and decision levels, respectively. These strategies outperform single-expert methods on homogeneous expert panels. Additionally, the team proposes Marginal-Calibrated Conformal Consensus (MC3) to address the cha


Background and Motivation

In the context of LLM-as-a-Judge evaluation for open-ended dialogue, controlling risk and improving assessment accuracy are critical challenges. Traditional methods relying on single-expert judgments struggle to handle the uncertainties of complex scenarios.

Methodology and Innovation

The ArXiv team proposes a Multi-Expert Conformal Risk Control (CRC) algorithm with the following innovations:

  1. Multi-Expert Aggregation Mechanisms:

    • Score Averaging: Aggregates scores at the scoring level by averaging multiple expert judgments to enhance accuracy.
    • Decision Voting: Aggregates at the decision level through a voting mechanism to select the optimal decision.
  2. Marginal-Calibrated Conformal Consensus (MC3):

    • Introduces initial threshold ratios to capture the distinct scoring scales of different experts.
    • Jointly tunes a unified decision function $C_t(x)$ to maintain consistency between calibration and testing, addressing the challenge of heterogeneous expert scoring differences.

Experiments and Results

The team constructed a benchmark dataset called Panel, consisting of 1,800 human pairwise preferences across three dialogue domains (ESConv, MSC, DREAM), and evaluated responses from four open-weight LLMs. The results demonstrate that:

  • Score Averaging and Decision Voting significantly improve accuracy and acceptance rates on homogeneous expert panels.
  • MC3 further extends these gains to heterogeneous panels, proving its effectiveness in handling different scoring scales.

Industry Impact and Future Directions

This research provides a new technical path for LLM-as-a-Judge evaluation in open-ended dialogue, particularly excelling in handling complex scenarios and heterogeneous experts. In the future, this method could be applied to more fields such as intelligent customer service, virtual assistants, and automated decision-making systems, further enhancing the reliability and user experience of AI systems.

Developer Recommendations

  • Multi-Expert Integration: It is recommended to integrate multi-expert mechanisms into LLM-as-a-Judge evaluation to enhance robustness and accuracy.
  • Model Optimization: Developers can try to integrate the MC3 method into existing models to address the scoring differences of heterogeneous experts.
  • Data Diversity: When building evaluation datasets, consider dialogue contexts from different domains to enhance the model's generalization capabilities.

Source: ArXiv cs.CL (2026-08-27)

— END —

Tags: #ArXiv #Multi-Expert Aggregation #Conformal Risk Control #LLM-as-a-Judge #Open-Ended Dialogue

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…