This study proposes a monitoring system to prevent falls from height (FFH) accidents in construction sites. In our previous work, a method was proposed for recognizing the fastening state of the safety hook based on the motion similarity between two Inertial Measurement Unit (IMU) sensors, but its performance was limited to 90.64% Youden's Index (YI). This study introduces a safety hook monitoring system that achieves better performances by utilizing a deep siamese neural network based-model to develop valid feature representations, which enhance performances. Our proposed approach achieves 97.69% YI, surpassing previous works in recognition performances.

Fall from height (FFH) is an accident that leads fatalities in construction workers, and a major cause of FFH is due to the improper fastening of a safety hook of safety harness to a temporary structure. In this study, we propose a new approach for monitoring the proper use of a safety harness in workers by recognizing fastening state of the safety hook. The safety hook fastening state is recognized based on the similarity of the motion of the hook and the body. This is based on the assumption that if a safety hook is fastened to a part of a body, the motion of the hook and the body are similar as compared to when the hook is fastened to a temporary structure. In this study, we propose a method for measuring the similarity of motion by representing the magnitude of the acceleration as an ordinal variable to alleviate the effects from the outliers, and the magnitude of the acceleration is represented with rotation in a spherical coordinate system. The effectiveness of our approach is verified by our newly collected task-related human activity dataset comprising the motion data of the hook and the body from inertial measurement unit (IMU) embedded mobile devices, each of which is attached to the neck of the hook and a chest strap of the safety harness, respectively. Our proposed method confirmed that representing the magnitude of acceleration as an ordinal variable shows an improved performance of 87.18% in terms of Youden’s index. Our proposed method further verified that jointly representing the magnitude of the acceleration and the rotation in the spherical coordinate system shows an improved performance of 90.00% in terms of Youden’s index. The dataset will be available upon the publication of the journal article.

With the growth of the social robot market, much research has been undertaken on facial expression recognition, which is an important function of a social robot. Facial expression recognition models have shown good performance in a facial expression image dataset that expresses emotion without considering speaking effect. However, in reality, humans often express emotions by speaking and moving the muscles around the mouth. Therefore, the lack of consideration of speech leads to unsatisfactory emotion recognition results. In this paper, we investigated two points to be considered in learning a facial expression recognition model. First, we confirmed whether the information around a mouth induces the recognition model in speech act to misrecognition like the case of a facial expression recognition in non-speech acts or it has valid information for facial expression recognition. Second, Generative Adversarial Network (GAN)-based data augmentation has been performed to cover the problem in which the accuracy of the recognition model in speech acts is low because of the relatively small variance about the subject in RML dataset. The results showed that the information around the mouth made facial expression recognition in speech acts exhibit higher performance, unlike the case of facial expression recognition in non-speech acts. In addition, the GAN-based data augmentation alleviated the accuracy degradation in facial expression recognition because of the low variance of the dataset.

Face spoofing detection could be used in conjunction with face recognition or facial expression recognition in social robots for better interaction with the users. Recent works have employed thermal image-based spoofing detection due to its competitive spoofing detection performance. However, existing methods utilized indoor thermal images which put a limit on using thermal image-based spoofing detection in outdoor environments. Moreover, several thermal image-based spoofing detection datasets consist of facial images at a frontal viewpoint, which would be put a limit on the usage in social robots with short heights. In this paper, we present our novel thermal image dataset for spoofing detection in outdoor environments. Unlike most existing datasets, our dataset is collected in outdoor environments with the camera facing the upper view to simulate the viewpoint of a social robot. To cover diverse outdoor environments, the dataset was collected at varying locations, times, and weather. Furthermore, we trained an event recognition-based spoofing detection on our collected dataset and also have provided benchmarking results for several baselines. Our spoofing detection method based on MobileNetV2 yielded a spoofing detection accuracy of 96.04% and GFLOPs of 0.88. Our dataset will be open-sourced for users pursuing academic research and will be available upon request to the authors.