In this paper we introduce a Kinect based posture recognition approach that can classify the user’s pose and gesture and match them to a set of predefined musical instruments. The efficiency of the approach is then demonstrated using two applications. The Virtual Orchestra system uses pose- and gesture-recognition along with Augmented Reality technology to add a virtual musical instrument into the scene, both visually and audibly: the visual representation of the instrument is placed into the user’s hands and the sound of the corresponding instrument is played. An additional functionality is that the user can control the intensity and the pitch of the sound by changing the speed of his hand or finger movements. The Magic Mirror game is the second application developed for the Berlin Concert Hall that uses the posture recognition approach to introduce the visitors to some classical music pieces and familiarize them with various classical musical instruments.

We investigate an edge-computing scenario for robot control, where two similar neural networks are running on one computational node. We test the feasibility of using a single object-detection model (YOLOv5) with the benefit of reduced computational resources against the potentially more accurate independent and specialized models. Our results show that using one single convolutional neural network (for object detection and hand-gesture classification) instead of two separate ones can reduce resource usage by almost $50\%$. For many classes, we observed an increase in accuracy when using the model trained with more labels. For small datasets (a few hundred instances per label), we found that it is advisable to add labels with many instances from another dataset to increase detection accuracy.

In this paper we propose a system for real-time hand gesture recognition for use as a human-computer interface. Firstly, hand detection is performed using a Viola-Jones algorithm. We use the Continuously Adaptive Mean Shift Algorithm (CAM Shift) to track the position of each detected hand in each video frame. The hand contour is then extracted using a Border Following algorithm, which is preceded by skin-colour thresholding, performed in HSV colour space. Finally, a semantic-probabilistic network is introduced, which uses an ontological gesture model and Bayesian network for gesture recognition. We then demonstrate the effectiveness of our technique on a training data set, as well as on a real-life video sequence.

Accurate and reliable gesture recognition is a central problem in human-computer interaction (HCI). Many applications that make use of gesture recognition call for mobile devices with reduced power consumption, weight and form factors. Recent advances in computer vision were particularly brought by deep neural networks and come at the cost of high computational complexity that hinders the employment on mobile devices. In this study, we evaluate the usability of a low-cost Raspberry Pi 4B amended by a Coral USB Accelerator, or a Neural Compute Stick 2, respectively, for low power real-time gesture recognition. To this end we evaluate the accuracy, inference time and power consumption for two different deep neural network-based recognition models and compare the results to other computer systems available as standard. Our experiments show that a combination of a Raspberry Pi 4B and Coral USB Accelerator allows for hand gesture recognition at frame rates of up to 30 frames per second at a power consumption of less than 5 Watts.

In this paper a semantic-probabilistic network for event recognition is proposed. The approach uses pre-defined domain ontology to describe the events and scenarios in the scene as a hierarchical decomposition of simple concepts and variables and then perform an automated conversion of the ontology into a Bayesian network. A novel approach to Bayesian network nodes weights calculation is used based on the weighted relation between concepts of the ontology. We then test the performance of our approach to recognize gestures in a human gesture recognition system.