Prediction of Relapse in Adolescent Depression using Fusion of Video and Speech Data

This article presents an innovative approach to predicting depression relapse in adolescents. Adolescentsíntensive use of video and voice-based smartphone apps presents a rich, multimodal dataset that can be utilized for this purpose. This work uses a dataset from the Depression Early Warning study conducted at the Center for Addiction and Mental Health. After using a pre-trained Inception ResNet to generate embeddings of video frames, the proposed framework integrates this with synchronized speech data. These embeddings are fused with audio features, resulting in a multimodal dataset. The combined features are processed through a Long Short-Term Memory model and a fully connected network to predict relapse of depression. An average accuracy of 0.80 highlights the effectiveness of the proposed multimodal approach and underscores its potential to effectively predict depression relapse in adolescents.