We propose a multi-task ensemble framework that jointly learns multiple related problems. The ensemble model aims to leverage the learned representations of three deep learning models (i.e., CNN, LSTM and GRU) and a hand-crafted feature representation for the predictions. Through multi-task framework, we address four problems of emotion and sentiment analysis, i.e., "emotion classification & intensity", "valence, arousal & dominance for emotion", "valence & arousal for sentiment", and "3-class categorical & 5-class ordinal classification for sentiment". The underlying problems cover two granularity (i.e., coarse-grained and fine-grained) and a diverse range of domains (i.e., tweets, Facebook posts, news headlines, blogs, letters etc.). Experimental results suggest that the proposed multi-task framework outperforms the single-task frameworks in all experiments.
11 August 2020 in IEEE Transactions on Affective Computing
An important skill for effective communication is the ability to express specific sentiment and emotion in a conversation. It is desirable for any robust dialogue system to handle the combined effect of both sentiment and emotion while generating responses to provide a better experience and concurrently increase user satisfaction. Previously, research on either emotion or sentiment controlled dialogue generation has shown impressive performance, but the simultaneous effect of both is still unexplored. The existing dialogue systems are majorly based on unimodal sources, predominantly the text, and thereby cannot utilize the information present in the other sources such as video, audio, image etc. In this work, we present a first large scale benchmark Sentiment Emotion aware MultimodalDialogue (SEMD) dataset for the task of sentiment and emotion controlled dialogue generation. The SEMD dataset consists of55k conversations from 10 TV shows having text, audio and video information. To utilize multimodal information we propose multimodal attention based conditional variational autoencoder (M-CVAE) that outperforms several sophisticated multimodal baselines. Quantitative and qualitative analysis show that multimodality along with contextual information plays an essential role in generating coherent and diverse responses for any given emotion and sentiment.
People suffering from stress and various mental health problems find it easier to express and share their feelings on online platforms, such as Twitter. However, the imposed character limit (280 characters) by Twitter and infrequent online activities of a section of users poses a serious setback in using computational methods for mental health analysis or emotion research. Twitter provides rich metadata information about its users (such as user’s description, geolocation, and profile image URL), which can provide valuable information regarding the mental state of the users. We hypothesize that Twitter’s rich metadata information about their users can provide some valuable depression cues, which may help in an early low-profile evaluation. In this article, we investigate this hypothesis by developing an end-to-end multimodal multitask (MT) system for depression detection (primary task) and emotion recognition (auxiliary task), where the variation of emotion information based on different user descriptions assists the learning of the primary task. The proposed system attains 70% accuracy on the depression detection task outperforming several single-task (ST) baselines built on the various combination of input features. Our findings indicate that Twitters’s rich metadata information can be leveraged to detect depression among users with significant confidence.