Hugging Face Releases Last Secure Code Benchmark: Evaluating AI Coding Agents' Code Security

Summary:Hugging Face has released the Last Secure Code Benchmark, a comprehensive evaluation framework for assessing the security of code generated by AI coding agents. Built on 150 real vulnerabilities, the benchmark includes 450 tasks across seven widely used programming languages (Java, C, C++, JavaScript, Python, PHP, and Go) and covers 24 Common Weakness Enumerations (CWEs), including 21 of the 2025 CWE Top 25 and 8 of the Top 10 Known Exploited Vulnerabilities (KEV). The results show that while AI

---

Background and Motivation

As AI coding agents become increasingly prevalent in software development, the quality and security of the generated code have become critical issues. While AI agents excel at functional implementation, their ability to generate secure code has not been thoroughly evaluated. The Last Secure Code Benchmark, introduced by Hugging Face, aims to address this gap by providing a systematic framework for assessing the security of code generated by AI agents.

Technical Mechanism Analysis

The benchmark is built on the following core design principles:

1. Vulnerability Sources and Task Design:

   • The benchmark includes 150 real vulnerabilities and covers seven mainstream programming languages (Java, C, C++, JavaScript, Python, PHP, and Go).
   • Tasks are designed across three scenarios: function completion, file writing, and building a repository's source tree from scratch.
   • Each task only specifies functionality without mentioning security, and hidden functional and security tests evaluate the results.

2. Weakness Coverage:

   • The benchmark covers 24 Common Weakness Enumerations (CWEs), including 21 of the 2025 CWE Top 25 and 8 of the Top 10 Known Exploited Vulnerabilities (KEV).
   • This broad coverage ensures the comprehensiveness and representativeness of the tests.

3. Evaluation Metrics:

   • Functional pass rate: 73.6% to 91.3% of tasks pass.
   • Security pass rate: Even the strongest agent, Claude Fable 5, only achieves a 25.1% success rate in generating both functional and secure code.

Engineering Trade-offs and Performance

• Strengths:

  • Provides a systematic evaluation framework covering multiple programming languages and weakness types.
  • Reveals the shortcomings of AI agents in code security, providing direction for future improvements.

• Challenges:

  • AI agents perform poorly when dealing with complex vulnerabilities, indicating a significant need for improvement in secure coding capabilities.
  • The complexity of the benchmark may impose a heavy computational burden on some agents.

Developer Implementation and Deployment Recommendations

1. Secure Coding Practices:

   • Developers should combine traditional secure coding practices with AI coding agents for code review.

2. Agent Improvement Directions:

   • AI model developers should incorporate security as a training objective, introducing more secure coding knowledge into the training process.

3. Continuous Evaluation and Feedback:

   • Regularly use similar benchmarks to evaluate agents and iteratively improve based on the results.

Conclusion

The release of the Last Secure Code Benchmark provides an important tool for assessing the security of AI coding agents, revealing current shortcomings and guiding future improvements.

---

Source: Hugging Face Daily Papers (2026-10-10)

— END —

Tags: #Hugging Face Follow #AI Coding Agents Follow #Code Security Follow #Benchmark Follow