arXiv Research: Dissociation of Fairness and Explanation Faithfulness in Legal Document Classification
Summary:This study investigates the relationship between fairness and model explanation faithfulness in legal document classification tasks. Using the ECtHR dataset from LexGLUE, the researchers compared a standard LegalBERT model with a fairness-regularized variant that penalizes stereotypical warmth and competence representations during fine-tuning. The results indicate that while the fairness intervention did not significantly reduce demographic disparities, it consistently degraded the sufficiency o
Background and Motivation
In recent years, Transformer-based models like LegalBERT have been widely adopted in legal decision support systems. However, these models face challenges in terms of fairness and explainability. Fairness concerns the model's performance across different demographic groups, while explainability focuses on the transparency of the decision-making process. These two properties are usually evaluated separately, leading to an incomplete understanding of the effects of debiasing interventions.
Methodology
The study uses the ECtHR dataset from LexGLUE to compare a standard LegalBERT model with a fairness-regularized variant that penalizes stereotypical warmth and competence representations during fine-tuning. Evaluation metrics include predictive performance, demographic fairness, and SHAP explanation faithfulness.
Key Findings
- Limited Fairness Impact: At the performance-optimal regularization strength, the fairness intervention did not significantly reduce demographic disparities. This result holds across two fairness definitions and a conventional word-pair control on the gender axis.
- Explainability Degradation: While classification performance remained largely unchanged, the intervention consistently degraded explanation sufficiency across all five seeds and three thresholds.
- Dissociation Revealed: The shuffled-pair control experiment showed that the degradation in explainability was due to contrastive representational regularization rather than specifically to the warmth and competence structure. This indicates a dissociation between fairness and explainability.
Industry Impact and Recommendations
- Independent Fairness Evaluation: The study emphasizes the need for fairness to be evaluated independently of explainability to ensure fair performance across different demographic groups.
- Optimizing Explainability Mechanisms: While pursuing fairness, attention should be paid to the design of explainability mechanisms to avoid degradation in explanation quality.
- Multi-Dimensional Model Evaluation: It is recommended to introduce more dimensions, such as explanation sufficiency and fairness, into model evaluation to comprehensively assess model performance.
Technical Highlights
- Fairness Regularization Method: Penalizing stereotypical representations to optimize model fairness.
- SHAP Explainability Evaluation: Using SHAP values to assess the sufficiency of model explanations, providing a deeper understanding of the decision-making process.
- Dissociation Revelation: Systematically revealing the dissociation between fairness and explainability for the first time, offering a new direction for future research.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-10-02)
Tags: #Fairness #Explainability #Legal AI #Transformer Models #LexGLUE
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments