arXiv Releases BILAL-BENCH: Benchmarking Islamic-Source Fabrication in Bengali for Large Language Models
Summary:arXiv has released BILAL-BENCH, a new benchmark designed to evaluate large language models (LLMs) on fabricating Islamic-source content in Bengali, such as Qur'anic verses or hadiths. The benchmark consists of 240 expert-verified questions covering Qur'anic attribution, hadith, fiqh, contemporary issues, and adversarial false-premise prompts, using a two-axis rubric to separate source errors from assertiveness. The evaluation of seven open-weight models (ranging from 1B to 120B parameters) revea
Core Breakthrough
The BILAL-BENCH benchmark, released by arXiv, addresses a critical gap in the evaluation of Islamic-source content generation, particularly in the Bengali language. While existing benchmarks are predominantly in Arabic and often closed-ended, BILAL-BENCH introduces 240 expert-verified open-ended questions, covering Qur'anic attribution, hadith, fiqh, contemporary issues, and adversarial false-premise prompts.
Technical Mechanism Analysis
BILAL-BENCH employs a two-axis rubric:
- Source Errors: Assessing the deviation of the generated content from the original Islamic sources.
- Assertiveness Errors: Evaluating the model's behavior when it cannot support its assertions, such as being overconfident or misattributing sources.
The evaluation of seven open-weight models (ranging from 1B to 120B parameters) revealed that six of them confidently fabricated content in 79.3% to 95.3% of source-generation questions, with no monotonic improvement observed as model size increased. Hadith provenance emerged as the weakest category, with only two out of 315 responses being both correct and properly sourced.
Engineering Trade-offs and Performance
- Model Scale vs. Performance: Despite the increase in model size, no performance improvement was observed, indicating fundamental limitations in current models when handling complex religious texts.
- Error Type Analysis: Hadith provenance errors were the highest, highlighting significant deficiencies in models' handling of specific domain knowledge.
- Assertiveness Behavior: Models varied widely in whether they withheld unsupported claims, indicating different strategies in handling uncertainty.
Developer Deployment and Implementation
- Domain-Adaptive Training: It is recommended to perform domain-adaptive training, particularly in specific areas like religious texts, to improve model performance in handling complex cultural contexts.
- Multilingual Support: The release of BILAL-BENCH provides a new tool for evaluating multilingual models in religious text generation, allowing developers to optimize models for multilingual capabilities.
- Ethics and Safety: Special attention should be paid to ethical considerations when deploying such models to avoid generating misleading or offensive content.
Conclusion
The release of BILAL-BENCH provides an important tool for evaluating the capabilities of large language models in generating Islamic-source content in Bengali, revealing the limitations of current models in handling complex religious texts and pointing the way for future research.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-11-02)
Tags: #arXiv #Large Language Models #Islamic-Source Content #Bengali #Benchmark
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments