Hugging Face Releases LeanGuard: A New Framework for Enhancing Tool-Calling Safety in AI Agents

Summary:Hugging Face has released LeanGuard, a runtime framework designed to enhance the safety of tool-calling in AI agents. LeanGuard checks each proposed tool call against a policy written in Lean4 before execution, ensuring compliance with domain policies. It supports two correction methods: feedback, which provides the violated rule and a hint, and proof-guided repair, which uses Lean metaprogramming to find an admissible call that preserves the user's intent. Experiments on Tau-Bench and DTap benc

---

Technical Mechanism Analysis

LeanGuard's core mechanisms include the following key components:

1. Policy Definition and Execution: LeanGuard uses Lean4 to define policies that combine state predicates with scoped past-time temporal operators to describe the legal conditions for tool calls.

2. Real-time Checking and Decision-making: Before executing a tool call, LeanGuard performs a real-time check. The decision process is proven to be equivalent to LeanLTL semantics, ensuring the strictness and correctness of policy execution.

3. Correction Mechanisms:

   • Feedback Mechanism: When a call is denied, the agent receives the violated rule and a hint for correction.
   • Proof-guided Repair: Lean metaprogramming searches for an admissible call that preserves the user's intent and complies with the policy, without relying on the model.

Engineering Trade-offs and Performance

• Enhanced Safety: On the Tau-Bench benchmark, agents using LeanGuard executed no prohibited calls, while unprotected agents executed at least 33 such calls.

• Task Success Rate: Although LeanGuard significantly reduced the number of prohibited calls, the task success rate did not significantly decrease, indicating that it maintains high performance while enhancing safety.

• Protection Against Prompt Injection Attacks: On the DTap benchmark, LeanGuard reduced the direct prompt injection success rate from 57.9% to 5.9%, demonstrating its effectiveness in defending against malicious attacks.

• Resource Overhead: LeanGuard's runtime overhead is low, making it suitable for resource-constrained environments.

Developer Implementation and Deployment Recommendations

• Policy Design: Developers need to design detailed policies based on specific application scenarios to ensure the effectiveness of LeanGuard.

• Feedback Mechanism Optimization: It is recommended to continuously optimize policies based on user feedback to improve the adaptability and robustness of the agent.

• Application of Metaprogramming Techniques: Utilizing Lean metaprogramming for proof-guided repair can further enhance the safety and reliability of the agent.

Conclusion

LeanGuard provides an effective tool-calling safety mechanism for AI agents. Through real-time policy checking and flexible correction methods, it significantly enhances the safety and reliability of agents. Its excellent performance in multiple benchmarks demonstrates its great potential in practical applications.

---

Source: Hugging Face Daily Papers (2026-11-02)

— END —

Tags: #Hugging Face Follow #LeanGuard Follow #AI Safety Follow #Agentic Follow #Tool-Calling Follow