The interest in action and gesture recognition has grown considerably in the last years. In this paper, we present a survey on current deep learning methodologies for action and gesture recognition in image sequences. We introduce a taxonomy that summarizes important aspects of deep learning for approaching both tasks. We review the details of the proposed architectures, fusion strategies, main datasets, and competitions. We summarize and discuss the main works proposed so far with particular interest on how they treat the temporal dimension of data, discussing their main features and identify opportunities and challenges for future research.

Time series of displacement are now routinely available from satellite InSAR and are used for flagging anomalous ground motion, but not yet forecasting. We test conventional time series forecasting methods such as SARIMA and supervised machine learning approaches such as long short‐term memory (LSTM) compared to simple function extrapolation. We focus initially on forecasting seasonal signals and begin by characterizing the time‐series using sinusoid fitting, seasonal decomposition, and autocorrelation functions. We find that the three measures are broadly comparable but identify different types of seasonal characteristic. We use this to select a set of 310 points with highly seasonal characteristics and test the three chosen forecasting methods over prediction windows of 1–9 months. The lowest overall median RMSE values are obtained for SARIMA when considering short term predictions (6 months). Machine learning methods (LSTM) perform less well. We then test the prediction methods on 2,000 randomly selected points with a range of seasonalities and find that simple extrapolation of a constant function performed better overall than any of the more sophisticated time series prediction methods. Comparisons between seasonality and RMSE show a small improvement in performance with increasing seasonality. This proof‐of‐concept study demonstrates the potential of time‐series prediction for InSAR data but also highlights the limitations of applying these techniques to nonperiodic signals or individual measurement points. We anticipate future developments, especially to shorter timescales, will have a broad range of potential applications, from infrastructure stability to volcanic eruptions.

This paper provides an overview of the Joint Contest on Multimedia Challenges Beyond Visual Analysis. We organized an academic competition that focused on four problems that require effective processing of multimodal information in order to be solved. Two tracks were devoted to gesture spotting and recognition from RGB-D video, two fundamental problems for human computer interaction. Another track was devoted to a second round of the first impressions challenge of which the goal was to develop methods to recognize personality traits from short video clips. For this second round we adopted a novel collaborative-competitive (i.e., coopetition) setting. The fourth track was dedicated to the problem of video recommendation for improving user experience. The challenge was open for about 45 days, and received outstanding participation: almost 200 participants registered to the contest, and 20 teams sent predictions in the final stage. The main goals of the challenge were fulfilled: the state of the art was advanced considerably in the four tracks, with novel solutions to the proposed problems (mostly relying on deep learning). However, further research is still required. The data of the four tracks will be available to allow researchers to keep making progress in the four tracks.

We present a methodology to address the problem of human gesture segmentation and recognition in video and depth image sequences. A Bag-of-Visual-and-Depth-Words (BoVDW) model is introduced as an extension of the Bag-of-Visual-Words (BoVW) model. State-of-the-art RGB and depth features, including a newly proposed depth descriptor, are analysed and combined in a late fusion form. The method is integrated in a Human Gesture Recognition pipeline, together with a novel probability-based Dynamic\ud Time Warping (PDTW) algorithm which is used to perform prior segmentation of idle gestures. The proposed DTW variant uses samples of the same gesture category to build a Gaussian Mixture Model driven probabilistic model of that gesture class. Results of the whole Human Gesture Recognition pipeline in a public data set show better performance in comparison to both standard BoVW model and DTW approach.Postprint (published version