The objective of multimodal information fusion is to mathematically analyze information carried in different sources and create a new representation which will be more effectively utilized in pattern recognition and other multimedia information processing tasks. In this paper, we introduce a new method for multimodal information fusion and representation based on the Labeled Multiple Canonical Correlation Analysis (LMCCA). By incorporating class label information of the training samples, the proposed LMCCA ensures that the fused features carry discriminative characteristics of the multimodal information representations, and are capable of providing superior recognition performance. We implement a prototype of LMCCA to demonstrate its effectiveness on handwritten digit recognition, face recognition and object recognition utilizing multiple features, bimodal human emotion recognition involving information from both audio and visual domains. The generic nature of LMCCA allows it to take as input features extracted by any means, including those by deep learning (DL) methods. Experimental results show that the proposed method enhanced the performance of both statistical machine learning (SML) methods, and methods based on DL.

At present, in the research of multimodal human action recognition, the weighted fusion method with fixed weight is widely applied in the decision level fusion of most models. In this way, the weight is usually obtained from the original experience or traversal search, which is inaccurate or has a large amount of calculation, and ignores the different representation ability of various modal data for various classes of action information. With the help of the powerful decision-making ability of deep reinforcement learning, we propose a multimodal decision-making fusion weight allocation network based on deep reinforcement learning. This letter mainly discusses the design of the model, which involves the modeling of reinforcement learning problem in action recognition, the design of neural network and the selection of problem-solving scheme. Experimental results on NTU RGB + D and HMDB51 datasets show the effectiveness of the proposed method.

The method based on the two-stream networks has achieved great success in video action recognition. However, most existing methods employ the same structure for both spatial and temporal networks, leading to unsatisfied performance. In this paper, we propose a spatiotemporal heterogeneous two-stream network, which employs two different network structures for spatial and temporal information, respectively. Specifically, the Residual network (ResNet) and BN-Inception are utilized as the base networks to present the spatiotemporal characteristics of different human actions. In addition, a segmental architecture is employed to model long-range temporal structure over video sequences to better distinguish the similar actions owning sub-action sharing phenomenon. Moreover, combined with the strategy of data augment, a modified cross-modal pre-training strategy is proposed and applied to the spatiotemporal heterogeneous network to improve the final performance of human actions recognition. The experiments on UCF101 and HMDB51 datasets demonstrate the proposed spatiotemporal heterogeneous two-stream network outperforms the spatiotemporal isomorphic networks and other related methods.

Human action recognition (HAR) by skeleton data is considered a potential research aspect in computer vision. Three-dimensional HAR with skeleton data has been used commonly because of its effective and efficient results. Several models have been developed for learning spatiotemporal parameters from skeleton sequences. However, two critical problems exist: (1) previous skeleton sequences were created by connecting different joints with a static order; (2) earlier methods were not efficient enough to focus on valuable joints. Specifically, this study aimed to (1) demonstrate the ability of convolutional neural networks to learn spatiotemporal parameters of skeleton sequences from different frames of human action, and (2) to combine the process of all frames created by different human actions and fit in the spatial structure information necessary for action recognition, using multi-task learning networks (MTLNs). The results were significantly improved compared with existing models by executing the proposed model on an NTU RGB+D dataset, an SYSU dataset, and an SBU Kinetic Interaction dataset. We further implemented our model on noisy expected poses from subgroups of the Kinetics dataset and the UCF101 dataset. The experimental results also showed significant improvement using our proposed model.

The detection and recognition of moving objects in image sequence images involve many aspects, such as pattern recognition, image processing, and computer vision. The main difficulties of target detection and recognition are complex background interference, local occlusion, real-time recognition, illumination changes, target size type changes, etc. However, it is very difficult to solve these problems in practical applications. This article introduces image pre-processing for the pre-processing of image sequences. Selectively we highlight the visually obvious features that are helpful for target detection in the image, weaken the image background and features that are not related to the target, and improve the quality of the image sequence. A multi-information integrated probability density estimation kernel integrating gray scale, spatial relationship and local standard deviation information is designed, and the multiinformation integrated kernel is used to extract the feature of the moving target. In terms of moving target recognition, Naive Bayes is used as a weak learner. In order to avoid the over-fitting of the classifier caused by high-noise moving image sequence features, the regularized Adaboost recognition model is introduced as a moving target recognition classifier. In order to completely separate the target and the background, we propose a moving target extraction method based on multi-information kernel density estimation, and input relevant target feature description vectors into the regularized Adaboost-based moving target recognition framework. Robust target recognition performance is obtained, and the reliability of target recognition under high noise data is improved.