23 July 2019 in IEEE Robotics and Automation Letters
In this paper we address the problem of multi-cue affect recognition in challenging scenarios such as child-robot interaction. Towards this goal we propose a method for automatic recognition of affect that leverages body expressions alongside facial ones, as opposed to traditional methods that typically focus only on the latter. Our deep-learning based method uses hierarchical multi-label annotations and multi-stage losses, can be trained both jointly and separately, and offers us computational models for both individual modalities, as well as for the whole body emotion. We evaluate our method on a challenging child-robot interaction database of emotional expressions collected by us, as well as on the GEMEP public database of acted emotions by adults, and show that the proposed method achieves significantly better results than facial-only expression baselines.
1 April 2022 in Robotics and Autonomous Systems
23 July 2023 in Universal Access in the Information Society
Recent advances in sign language (SL) technologies, along with demand for SL education, have led to increased interest in developing tools that enable automatic assessment of learners’ SL video productions, helping both students and their instructors. At the very least, such tools should perform automatic SL recognition (SLR) of non-studio quality videos in a signer-independent (SI) fashion, thus providing simple binary feedback on learners’ signing under realistic usage scenarios. Motivated by the above and the lack of any such tools for the Greek SL (GSL), we have been developing the “SL-ReDu” education platform for both receptive and productive GSL learning and student assessment. In this paper, we present our SLR module for GSL, developed for and integrated to the “SL-ReDu” system. The module incorporates state-of-the-art deep-learning based visual detection, feature extraction, and classification, operates in an SI mode on web-cam videos, and accommodates a small-size vocabulary of isolated signs and continuously fingerspelled letter sequences. We train the module on collected GSL data and demonstrate its superiority over a number of alternative SLR algorithms. We then conduct its objective evaluation within the “SL-ReDu” system and carry out a subjective evaluation of the overall platform, obtaining very satisfactory results in both.
18 June 2021 in IEEE Access
Understanding human-object interaction is a fundamental challenge in computer vision and robotics. Crucial to it is the ability to infer “object affordances” from visual data, namely the types of interaction supported by an object of interest and the object parts involved. Such inference can be approached as an “affordance reasoning” task, where object affordances are recognized and localized as image heatmaps, and as an “affordance segmentation” task, where affordance labels are obtained at a more detailed, image pixel level. To tackle the two tasks, existing methods typically: (i) treat them independently; (ii) adopt static image-based models, ignoring the temporal aspect of human-object interaction; and / or (iii) require additional strong supervision concerning object class and location. In this paper, we focus on both tasks, while addressing all three aforementioned shortcomings. For this purpose, we propose a deep-learning based dual encoder-decoder model for joint affordance reasoning and segmentation, which learns from our recently introduced SOR3D-AFF corpus of RGB-D human-object interaction videos, without relying on object localization and classification. The basic components of the model comprise: (i) two parallel encoders that capture spatio-temporal interaction information; (ii) a reasoning decoder that predicts affordance heatmaps, assisted by an affordance classifier and an attention mechanism; and (iii) a segmentation decoder that exploits the predicted heatmap to yield pixel-level affordance segmentation. All modules are jointly trained, while the system can operate on both static images and videos. The approach is evaluated on four datasets, surpassing the current state-of-the-art in both affordance reasoning and segmentation.