2 October 2023 in IEEE Transactions on Circuits and Systems for Video Technology
Temporal action localization (TAL) is a prevailing task due to its great application potential. Existing works in this field mainly suffer from two weaknesses: (1) They often neglect the multi-label case and only focus on temporal modeling. (2) They ignore the semantic information in class labels and only use the visual information. To solve these problems, we propose a novel Co-Occurrence Relation Module (CORM) that explicitly models the co-occurrence relationship between actions. Besides the visual information, it further utilizes the semantic embeddings of class labels to model the co-occurrence relationship. The CORM works in a plug-and-play manner and can be easily incorporated with the existing sequence models. By considering both visual and semantic co-occurrence, our method achieves high multi-label relationship modeling capacity. Meanwhile, existing datasets in TAL always focus on low-semantic atomic actions. Thus we construct a challenging multi-label dataset UCF-Crime-TAL that focuses on high-semantic actions by annotating the UCF-Crime dataset at frame level and considering the semantic overlap of different events. Extensive experiments on two commonly used TAL datasets, i.e., MultiTHUMOS and TSU, and our newly proposed UCF-Crime-TAL demenstrate the effectiveness of the proposed CORM, which achieves state-of-the-art performance on these datasets.
23 October 2023 in Visual Intelligence
Gesture recognition has attracted considerable attention and made encouraging progress in recent years due to its great potential in applications. However, the spatial and temporal modeling in gesture recognition is still a problem to be solved. Specifically, existing works lack efficient temporal modeling and effective spatial attention capacity. To efficiently model temporal information, we first propose a long- and short-term temporal shift module (LS-TSM) that models the long-term and short-term temporal information simultaneously. Then, we propose a spatial attention module (SAM) that focuses on where the change primarily occurs to obtain effective spatial attention capacity. In addition, the semantic relationship among gestures is helpful in gesture recognition. However, this is usually neglected by previous works. Therefore, we propose a label relation module (LRM) that takes full advantage of the relationship among classes based on their labels’ semantic information. To explore the best form of LRM, we design four different semantic reconstruction methods to incorporate the semantic relationship information into the class label’s semantic space. We perform extensive ablation studies to analyze the best settings of each module. The best form of LRM is utilized to build our visual-semantic network (VS Network), which achieves the state-of-the-art performance on two gesture datasets, i.e., EgoGesture and NVGesture.
23 October 2023 in Applied and Computational Engineering
This study presents a sophisticated personalized movie rating prediction model that incorporates user-generated content from Douyin, a prominent video-based social media platform. The proposed model effectively leverages user demographic data, movie metadata, historical rating records, and sentiment analysis of user comments to precisely predict a user's rating score for films. This study employs lexicon-integrated two-channel CNN-LSTM family models for conducting sentiment analysis. Furthermore, the study explores a variety of regression models, such as linear regression, ridge regression, LASSO regression, and Elastic Net regression, to determine the most suitable model for the dataset being analyzed. This analysis aims to enhance the accuracy and effectiveness of the sentiment analysis performed on the given dataset. Furthermore, we contemplate the application of collaborative filtering algorithms, such as Alternating Least Squares (ALS), in developing our personalized movie rating prediction model. By incorporating user-generated content from Douyin, our methodology enhances the accuracy of movie rating predictions and underscores the significance of personalized recommendations in the era of social media.