Hugging Face Releases Circuit Oracle: A New Tool for Automating Attribution Graph Analysis

Summary:Hugging Face has released Circuit Oracle, an LLM-based tool designed to automate attribution graph analysis through natural language queries. Circuit Oracle leverages a multi-agent framework and task-specific skills to answer questions about a target model by reading attribution graphs via tool calls. Evaluations on three safety-related proxy tasks show that while it achieves 78.2% accuracy in detecting spurious features—outperforming single-layer baselines—its performance in secret elicitation

---

Overview

Hugging Face has recently released a new tool named Circuit Oracle, designed to automate attribution graph analysis through natural language queries. Attribution graphs are an emerging tool in mechanistic interpretability that decompose language model computations into sparse interpretable features connected by causal edges. However, transforming these graphs into safety-related insights typically requires hours of manual analysis by experts.

Technical Mechanism Analysis

Circuit Oracle integrates a large language model (LLM), a multi-agent framework, and task-specific skills to read attribution graphs and answer natural language questions about a target model via tool calls. Its core mechanisms include:

1. Multi-Agent Framework: Multiple agents work collaboratively, each responsible for specific analysis tasks such as feature detection and secret extraction.
2. Tool Invocation Mechanism: By invoking predefined tool interfaces, agents can access detailed data within the attribution graph and perform in-depth analysis.
3. Task-Specific Skills: Circuit Oracle is equipped with specialized skill modules tailored to different tasks (e.g., detecting spurious features, extracting hidden knowledge), enhancing analysis accuracy.

Engineering Trade-offs and Performance

Evaluations on three safety-related tasks show:

• Spurious Feature Detection: Circuit Oracle achieves 78.2% accuracy, outperforming single-layer feature-ranking baselines.
• Secret Elicitation: While Circuit Oracle performs well in some cases, it lags behind trained self-explainers like Activation Oracles.
• Refusal Suppression: Its single committed intervention is slightly below a diff-in-means baseline, but improves when allowed five attempts.

Additionally, the pruned subgraphs and context-specific feature explanations generated by Circuit Oracle provide valuable insights for analysis.

Developer Deployment and Implementation Tips

For developers looking to deploy Circuit Oracle, the following tips may be helpful:

1. Attribution Graph Quality: Ensure the input attribution graph has sufficient high-quality data to improve the reliability of analysis results.
2. Task Customization: Customize the task-specific skill modules of Circuit Oracle according to specific needs to enhance its performance in particular domains.
3. Multi-Agent Collaboration: Leverage the advantages of the multi-agent framework to improve overall analysis efficiency through collaborative work.

Conclusion

Circuit Oracle demonstrates the potential of automating attribution graph analysis based on large language models. Although there is room for improvement in certain tasks, its innovative approach provides new directions for future research.

---

Source: ArXiv NLP/LLM (cs.CL) (2026-11-03)

— END —

Tags: #Hugging Face Follow #Large Language Model Follow #Attribution Graph Analysis Follow