arXiv Study Reveals the Impact of Mastery Thresholds on Learning Outcomes in Knowledge Tracing Models

Summary:A new arXiv study examines the impact of mastery thresholds in Knowledge Tracing (KT) models across various instructional settings and datasets. The research reveals that while Bayesian Knowledge Tracing (BKT) is relatively insensitive to threshold changes, neural models become increasingly selective as thresholds rise. The study emphasizes the importance of calibrating mastery thresholds dynamically based on the KT model and instructional priorities to balance performance, practice burden, and

---

Background and Motivation

In intelligent tutoring systems, mastery thresholds are commonly used to determine when students can stop practicing and advance to the next stage. However, the same numerical threshold can lead to vastly different decisions depending on the underlying Knowledge Tracing (KT) model, resulting in significant disparities in learning outcomes and resource allocation.

Methodology and Findings

The research team evaluated six KT models across four public educational datasets and analyzed 12 thresholds ranging from 0.50 to 0.99. Key findings include:

• Model Sensitivity Differences: Bayesian Knowledge Tracing (BKT) is relatively insensitive to threshold changes, whereas neural models become increasingly selective as thresholds increase. This is partly due to the different outputs of the models: BKT estimates the latent mastery probability, while neural models estimate the probability of a correct next response.

• Varying Optimal Thresholds: The best-balanced threshold varies substantially across models and settings. In half of the tested settings, neural models and BKT differed by more than 0.10 in their selected thresholds.

• Impact of Stricter Thresholds: Stricter thresholds do not reliably reduce performance gaps and can disproportionately restrict advancement, with stronger-prior students advancing up to 3.26 times as often as weaker-prior students.

Conclusions and Recommendations

The study concludes that mastery thresholds should be recalibrated dynamically based on the KT model and instructional priorities, with evaluations focusing on:

• Learning Outcomes: Ensuring that threshold adjustments enhance overall learning effectiveness.

• Practice Burden: Avoiding excessive burden on students due to overly high thresholds.

• Advancement Opportunities: Providing fair advancement opportunities for students at different levels.

Industry Impact and Developer Recommendations

• EdTech Companies: Reassess the mastery threshold settings in their KT models and adjust them dynamically based on specific instructional goals and model characteristics.

• Developers: When developing new KT models, consider introducing adaptive threshold mechanisms that automatically adjust thresholds based on student performance and instructional objectives.

• Educational Policy Makers: Pay attention to the impact of threshold settings on student equity and ensure that the application of educational technology does not exacerbate educational inequalities.

---

Source: ArXiv Machine Learning (cs.LG) (2026-10-02)

— END —

Tags: #Knowledge Tracing #EdTech #Machine Learning #Intelligent Tutoring Systems #Bayesian Knowledge Tracing