This article proposes a multimodal sentiment analysis system for recognizing a person’s aggressiveness in pain. The implementation has been divided into five components. The first three steps are related to a text-based sentiment analysis system to perform classification tasks such as predicting the classes into non-aggressive, covertly aggressive, and overtly aggressive classes. The remaining two components are related to an image-based sentiment analysis system. A deep learning-based approach has been employed to do feature learning and predict the three types of pain classes. An aggression dataset for the text-based system and the UNBC-McMaster database for an image-based system has been employed, respectively. Experimental results have been compared with the state-of-the-art methods, showing the superiority of the proposed approach. Finally, the scores due to text-based and image-based sentiment analysis systems are fused to obtain the performance for the proposed multimodal sentiment analysis system.

This study introduces a multimodal sentiment analysis system to assess and recognize human pain sentiments within an Internet of Things (IoT)-enabled healthcare framework. This system integrates facial expressions and speech-audio recordings to evaluate human pain intensity levels. This integration aims to enhance the recognition system’s performance and enable a more accurate assessment of pain intensity. Such a multimodal approach supports improved decision making in real-time patient care, addressing limitations inherent in unimodal systems for measuring pain sentiment. So, the primary contribution of this work lies in developing a multimodal pain sentiment analysis system that integrates the outcomes of image-based and audio-based pain sentiment analysis models. The system implementation contains five key phases. The first phase focuses on detecting the facial region from a video sequence, a crucial step for extracting facial patterns indicative of pain. In the second phase, the system extracts discriminant and divergent features from the facial region using deep learning techniques, utilizing some convolutional neural network (CNN) architectures, which are further refined through transfer learning and fine-tuning of parameters, alongside fusion techniques aimed at optimizing the model’s performance. The third phase performs the speech-audio recording preprocessing; the extraction of significant features is then performed through conventional methods followed by using the deep learning model to generate divergent features to recognize audio-based pain sentiments in the fourth phase. The final phase combines the outcomes from both image-based and audio-based pain sentiment analysis systems, improving the overall performance of the multimodal system. This fusion enables the system to accurately predict pain levels, including ‘high pain’, ‘mild pain’, and ‘no pain’. The performance of the proposed system is tested with the three image-based databases such as a 2D Face Set Database with Pain Expression, the UNBC-McMaster database (based on shoulder pain), and the BioVid database (based on heat pain), along with the VIVAE database for the audio-based dataset. Extensive experiments were performed using these datasets. Finally, the proposed system achieved accuracies of 76.23%, 84.27%, and 38.04% for two, three, and five pain classes, respectively, on the 2D Face Set Database with Pain Expression, UNBC, and BioVid datasets. The VIVAE audio-based system recorded a peak performance of 97.56% and 98.32% accuracy for varying training–testing protocols. These performances were compared with some state-of-the-art methods that show the superiority of the proposed system. By combining the outputs of both deep learning frameworks on image and audio datasets, the proposed multimodal pain sentiment analysis system achieves accuracies of 99.31% for the two-class, 99.54% for the three-class, and 87.41% for the five-class pain problems.

This paper presents a multimodal sentiment analysis system for analyzing and recognizing pain sentiments for the Internet of Things (IoT) Healthcare framework. This multimodal sentiment analysis system will analyze both facial patterns and speech-audio recordings to analyze human pain intensities. The implementation of the proposed system is divided into four components. The first component is to detect the facial region from the given image sequences of a video. Detecting facial regions is crucial based on the identified textual facial patterns. The second phase extracts discriminant and divergent features from the detected facial region using deep-learning techniques based on building convolutional neural network architectures, followed by improving fine-tuning parameters with transfer learning of some optimized models and fusion techniques. In the third phase, the speech-audio recordings are analyzed by extracting some divergent and valuable features, which undergo deep learning-based approaches to generate more divergent features. The performances reported due to image-based and audio-based pain sentiment analysis systems are fused at the fourth component to enhance the performance of the proposed multimodal sentiment analysis system, which is eligible to predict the different pain intensities, such as `High-Pain’, `Mild-Pain’, and `No-Pain’, that appeared on the facial pattern or the speech-audio recording of a person when they feel pain. Three standard image-based databases, such as 2D Face-pain-set, UNBC-McMaster shoulder pain, and BioVid Heat Pain databases, and one audio-based Variably Intense Vocalizations of Affect and Emotion Corpus (VIVAE) database, have been employed to test the proposed system. The performance of the image-based pain sentiment analysis system using a 2D Face-pain-set (2-class problem) is 76.23% accuracy, using the UNBC-McMaster shoulder pain database (3-class problem), 84.27% accuracy, and for BioVid Heat Pain database (5-class problem) 38.04% is obtained. Similarly, for the proposed audio-based pain sentiment analysis system, the highest performance is 97.56% for (50-50%) and 98.32% for (75-25%) Training-testing samples using the VIVAE database. The performance of the proposed system has been compared with some state-of-the-art methods that show its superiority. Finally, by fusing the decisions obtained due to image-based and audio-based deep learning models, the achieved performance is 99.31% for 2-class, 99.54% for 3-class, and 87.41% for 5-class obtained for the proposed multimodal pain sentiment analysis system.