Explainable Multimodal Artificial Intelligence Framework for Real-Time Cognitive Engagement Assessment in Smart Learning Environments

Abstract

An explainable, multimodal AI framework for assessing cognitive engagement in smart learning environments is presented in this paper. This framework integrates five modalities of sensing: facial expression, gaze, speech, interaction, and physiological sensing. The framework uses a preprocessing pipeline consisting of denoising, normalization, and time alignment of the input streams to the framework to perform feature extraction and fusion. The framework supports learning state assessments and generation of actionable instructional guidance using learning engagement state modeling, attention-based multimodal fusion, and explainable decision support. The framework was implemented, and the Open University Learning Analytics Dataset (OULAD) was used for evaluation. OULAD contains anonymized learner activity, assessments, demographic data, and logs of interactions with the virtual learning environment. To support the real-time assessment of cognitive engagement, additional multimodal engagement features were generated following a preprocessing, time alignment, feature extraction, and normalization. The framework was evaluated against several state-of-the-art models, including a diverse set of deep learning and multimodal models focused on engagement and learning. The framework surpassed its peers, achieving cognitive engagement assessment with 98.81% accuracy, 98.45% precision, and 98.12% recall with a 48.15 ms response time. Compared to the best competing model, the framework achieved superior explainability and transparency of its recommendations. The results show that the framework can offer learner-centered, trustworthy, and timely assessments of cognitive engagement in intelligent and adaptive learning environments.