Hugging Face Releases HICO-Bench: Revolutionizing Social Agent Interaction Benchmarking

Summary:Hugging Face introduces HICO-Bench, a benchmark comprising 180 everyday coordination scenarios such as resource allocation, scheduling, and authorization. HICO-Bench evaluates social agents by verifying the feasibility of the agreed-upon plan and the integrity of the conversation supporting it. The study reveals that 36% of conversations rated highly for goal completion fail the verification across thirteen LLMs. Additionally, the proposed Paired-Credit Policy Optimization (PaCPO) method enhance

---

Background and Challenges

In the field of social agent interactions, traditional evaluation methods rely heavily on Large Language Models (LLMs) to rate conversations, without considering the feasibility and integrity of the resulting plans. This approach fails to fully reflect the performance of agents in real-world scenarios, especially when dealing with complex coordination tasks such as resource allocation, scheduling, and authorization.

Innovations of HICO-Bench

HICO-Bench revolutionizes the evaluation of social agent interactions through the following:

1. Multi-scenario Coverage: HICO-Bench includes 180 everyday coordination scenarios, ranging from simple resource allocation to complex authorization tasks.
2. Dual Verification Mechanism: Each scenario has two versions that share the same public context but differ in one private condition. Agents need to record the plan they agree upon. A program checks the feasibility of the plan (Feasibility), ensuring the two records match and satisfy both participants' private conditions. An LLM judge checks the integrity of the conversation (Integrity), verifying that the conversation supports the recorded plan. Only when both checks are passed is it considered a Verified Success.
3. Experimental Results: Across thirteen LLMs, 36% of conversations rated highly for goal completion fail the verification, highlighting the limitations of traditional evaluation methods.

Paired-Credit Policy Optimization (PaCPO)

To address the challenge of generating effective plans during conversations, researchers propose the PaCPO method. After each turn, PaCPO asks the agent for a tentative plan and checks it against both versions of the scenario. If the tentative plan fits its respective version, the corresponding conversation turn is rewarded. Experimental results show that PaCPO boosts Qwen3-8B's Verified Success rate from 20.4% to 28.3%.

Engineering Trade-offs and Practical Applications

• Advantages: HICO-Bench provides a more realistic evaluation framework that accurately reflects the performance of agents in complex tasks. The PaCPO method significantly improves the Verified Success rate by optimizing the plan generation capability during conversations.
• Challenges: The evaluation process of HICO-Bench is complex and requires substantial computational resources. Additionally, the PaCPO method may require further optimization and adjustments when dealing with more complex scenarios.

Recommendations for Developers

1. Use HICO-Bench for Evaluation: Developers are encouraged to use HICO-Bench to evaluate their agents for more comprehensive performance feedback.
2. Combine with PaCPO for Training: When training agents, combining with the PaCPO method can improve their performance in complex tasks.
3. Focus on Multi-scenario Adaptability: When designing agents, consider their adaptability to different scenarios to ensure they perform well in various coordination tasks.

---

Source: Hugging Face Daily Papers (2026-10-14)

— END —

Tags: #Hugging Face Follow #Intelligent Agents Follow #Benchmarking Follow #PaCPO Follow #HICO-Bench Follow