Multimodal emotion recognition is an emerging interdisciplinary field of research in the area of affective computing and sentiment analysis. It aims at exploiting the information carried by signals of different nature to make emotion recognition systems more accurate. This is achieved by employing a powerful multimodal fusion method. In this study, a hybrid multimodal data fusion method is proposed in which the audio and visual modalities are fused using a latent space linear map and then, their projected features into the cross-modal space are fused with the textual modality using a Dempster-Shafer (DS) theory-based evidential fusion method. The evaluation of the proposed method on the videos of the DEAP dataset shows its superiority over both decision-level and non-latent space fusion methods. Furthermore, the results reveal that employing Marginal Fisher Analysis (MFA) for feature-level audio-visual fusion results in higher improvement in comparison to cross-modal factor analysis (CFA) and canonical correlation analysis (CCA). Also, the implementation results show that exploiting textual users’ comments with the audiovisual content of movies improves the performance of the system.

Affective video retrieval systems aim at finding video contents matching the desires and needs of users. Existing systems typically use the information contained in the video itself to specify its affect category. These systems either extract low-level features or build up higher-level attributes to train classification algorithms. However, using low-level features ignores global relations in data and constructing high-level features is time consuming and problem dependent. To overcome these drawbacks, an external source of information may be helpful. With the explosive growth and availability of social media, users’ comments could be such a valuable source of information. In this study, a new method for incorporating social media comments with the audio-visual contents of videos is proposed. Furthermore, for the combination stage a decision-level fusion method based on the Dempster–Shafer theory of evidence is presented. Experiments are carried out on the video clips of the DEAP (Database for Emotion Analysis using Physiological signals) dataset and their associated users’ comments on YouTube. Results show that the proposed system significantly outperforms the baseline method of using only the audio-visual contents for affective video retrieval.

In an opinionated long review, there may be several targets described by different potential terms. Traditional review-level techniques for Persian sentiment analysis addressed the problem using a one-method-fits-all solution in which the overall polarity of a review is calculated using all its opinionated words without considering their target. In this article, a new method is proposed, which first decomposes a long review into its constituent sentences and then detects the main target of each sentence. In the next step, five policies, including most occurring first (MOF), most general first (MGF), most specific first (MSF), first occurring first (FOF), and last occurring first (LOF), are proposed to come up with the main target of the review. Finally, using the part-of-speech (POS) tags, potential terms in the sentences are specified and a comprehensive sentiment lexicon is employed to compute the polarity of the sentences. In order to evaluate the proposed method, three data sets of user reviews about different topics, including digital equipment, hotels, and movies, are created as no previous study addressed the problem of target identification in the Persian language. The results of comparing the proposed method with a state-of-the-art lexicon-based method show that specifying the main targets of reviews can improve the performance of the systems about 17% and 12% in terms of accuracy and F1-measure. Moreover, the proposed method using the MGF policy achieves the best performance in finding the main target of reviews, while for finding the ultimate polarity of reviews, the MOF outperforms other policies.

Affective video retrieval systems seek to retrieve video contents concerning their impact on viewers' emotions. These systems typically apply a multimodal approach that fuses information from different modalities to specify the affect category. The m

Developing techniques to retrieve video contents with regard to their impact on viewers' emotions is the main goal of affective video retrieval systems. Existing systems mainly apply a multimodal approach that fuses information from different modalities to specify the affect category. In this paper, the effect of exploiting two types of textual information to enrich the audio-visual content of music video is evaluated; subtitles or songs' lyrics and texts obtained from viewers' comments in video sharing websites. In order to specify the emotional content of texts, an unsupervised lexicon-based method is applied. This method does not need any human-coded corpus for training and is much faster than supervised approach. In order to integrate these modalities, a new information fusion method is proposed based on the Dempster-Shafer theory of evidence. Experiments are conducted on the video clips of DEAP dataset and their associated viewers' comments on YouTube. Results show that incorporating songs' lyrics with the audio-visual content has no positive effect on the retrieval performance, whereas exploiting viewers' comments significantly improves the affective retrieval system. This could be justified by the fact that viewers' affective responses depend not only on the video itself but also on its context.

Multimodal emotion recognition systems aim at classifying emotion data, usually from different natures, into discrete affective categories. These systems fuse different modalities as each modality classifies the data from its own viewpoint and can compensate the limitations of others when combining with them. Existing approaches for multimodal data fusion either use feature-level or decision-level fusion. The former needs different modalities to be synchronized while the latter has not this limitation. In the current study, audio, visual, and users' comments are used as modalities for video emotion recognition. Although the first two modalities are synchronized, users' comments are not synchronized with them and this makes the use of pure feature-level data fusion impossible. In order to exploit the benefits of the feature-level approach, a hybrid method is proposed in the current study which first applies feature-level canonical correlation analysis (CCA) to audio and visual modalities and then combines the outputs with users' comments using a decision-level fusion. The results of applying this proposed method to DEAP dataset shows that using feature-level CCA not only outperforms the baseline feature-level method but also achieves a higher performance than the decision-level fusion.