Sign language recognition (SLR) is one of the crucial applications of the hand gesture recognition and computer vision research domain. There are many researchers who have been working to develop a hand gesture-based SLR application for English, Turkey, Arabic, and other sign languages. However, few studies have been conducted on Korean sign language classification because few KSL datasets are publicly available. In addition, the existing Korean sign language recognition work still faces challenges in being conducted efficiently because light illumination and background complexity are the major problems in this field. In the last decade, researchers successfully applied a vision-based transformer for recognizing sign language by extracting long-range dependency within the image. Moreover, there is a significant gap between the CNN and transformer in terms of the performance and efficiency of the model. In addition, we have not found a combination of CNN and transformer-based Korean sign language recognition models yet. To overcome the challenges, we proposed a convolution and transformer-based multi-branch network aiming to take advantage of the long-range dependencies computation of the transformer and local feature calculation of the CNN for sign language recognition. We extracted initial features with the grained model and then parallelly extracted features from the transformer and CNN. After concatenating the local and long-range dependencies features, a new classification module was applied for the classification. We evaluated the proposed model with a KSL benchmark dataset and our lab dataset, where our model achieved 89.00% accuracy for 77 label KSL dataset and 98.30% accuracy for the lab dataset. The higher performance proves that the proposed model can achieve a generalized property with considerably less computational cost.

Sign language recognition is crucial for improving communication accessibility for the deaf community and reducing dependence on human interpreters. Notably, while significant research efforts have been devoted to many prevalent languages, Korean Sign Language (KSL) remains relatively underexplored, particularly concerning dynamic signs and generalizability. The scarcity of KSL datasets has exacerbated this limitation, hindering progress. Furthermore, most KSL research predominantly relies on static image-based datasets for recognition, leading to diminished accuracy and the inability to detect dynamic sign words. Furthermore, most KSL research predominantly relies on static image-based datasets for recognition, leading to diminished accuracy and the inability to detect dynamic sign words. Additionally, existing KSL recognition systems grapple with suboptimal performance accuracy and heightened computational complexity, further emphasizing the existing research gap. To address these formidable challenges, we propose a robust dynamic KSL recognition system that combines a skeleton-based Graph Convolution network with an attention-based neural network, effectively bridging the gap. Our solution employs a two-stream deep learning network to navigate the intricacies of dynamic signs, enhancing accuracy by effectively handling non-connected joint skeleton features. In this system, the first stream meticulously processes 47 pose landmarks using the Graph Convolutional Network (GCN) to extract graph-based features. These features are meticulously refined through a channel attention module and a general CNN, enhancing their temporal context. Concurrently, the second stream focuses on joint motion-based features, employing a similar approach. Subsequently, these distinct features from both streams are harmoniously integrated and channelled through a classification module to achieve precise sign-word recognition. A significant contribution of our work lies in the creation of a novel KSL video dataset, addressing the scarcity of data in this domain. This dataset comprises comprehensive information, including skeletal data from 47 joint skeleton points and details from both hands, body, and facial expressions. Our dataset aims to fill a critical gap in KSL research and provides a solid foundation for more extensive and inclusive studies in the field. Through this innovative approach, we aim to contribute significantly to the field of KSL recognition, filling the gaps in dynamic sign recognition and bolstering the accessibility of sign language communication within the Korean deaf community and beyond. Our evaluations on a benchmark KSL-77 dataset and our proprietary lab dataset resulted in recognition accuracies of 99.87% and 100%, respectively. These results highlight the superiority of our model in the KSL recognition domain, outperforming existing models in terms of accuracy and computational efficiency.

The prevention of falls has become crucial in the modern healthcare domain and in society for improving ageing and supporting the daily activities of older people. Falling is mainly related to age and health problems such as muscle, cardiovascular, and locomotive syndrome weakness, etc. Among elderly people, the number of falls is increasing every year, and they can become life-threatening if detected too late. Most of the time, ageing people consume prescription medication after a fall and, in the Japanese community, the prevention of suicide attempts due to taking an overdose is urgent. Many researchers have been working to develop fall detection systems to observe and notify about falls in real-time using handcrafted features and machine learning approaches. Existing methods may face difficulties in achieving a satisfactory performance, such as limited robustness and generality, high computational complexity, light illuminations, data orientation, and camera view issues. We proposed a graph-based spatial-temporal convolutional and attention neural network (GSTCAN) with an attention model to overcome the current challenges and develop an advanced medical technology system. The spatial-temporal convolutional system has recently proven the power of its efficiency and effectiveness in various fields such as human activity recognition and text recognition tasks. In the procedure, we first calculated the motion along the consecutive frame, then constructed a graph and applied a graph-based spatial and temporal convolutional neural network to extract spatial and temporal contextual relationships among the joints. Then, an attention module selected channel-wise effective features. In the same procedure, we repeat it six times as a GSTCAN and then fed the spatial-temporal features to the network. Finally, we applied a softmax function as a classifier and achieved high accuracies of 99.93%, 99.74%, and 99.12% for ImViA, UR-Fall, and FDD datasets, respectively. The high-performance accuracy with three datasets proved the proposed system’s superiority, efficiency, and generality.

Recognizing sign language plays a crucial role in improving communication accessibility for the Deaf and hard-of-hearing communities. In Korea, many individuals facing hearing and speech challenges depend on Korean Sign Language (KSL) as their primary means of communication. Many researchers have been working to develop a sign language recognition system for other sign languages. Still, little research has been done for KSL alphabet recognition due to the lack of dataset availability. Moreover, existing KSL recognition systems have faced significant performance limitations due to the ineffectiveness of the features. To overcome the challenges, we newly created a KSL alphabet dataset and introduced an innovative KSL recognition system employing a strategic fusion approach. In the proposed study, we combined joint skeleton-based handcrafted features and pixel-based resnet101 transfer learning features to overcome the limitations of feature effectiveness in traditional systems. Our system consists of two distinct feature extraction streams: the first stream extracts essential handcrafted features, emphasizing capturing hand orientation information within KSL gestures. In the second stream, concurrently, we employed a deep learning-based resnet101 module stream to capture hierarchical representations of the KSL alphabet sign. By combining essential features from the first stream with the hierarchical features from the second stream, we generate multiple levels of fused features with the goal of forming a comprehensive representation of KSL gestures. Finally, we fed the concatenated feature into the deep learning-based classification module for the classification. We conducted extensive experiments with the newly created KSL alphabet dataset, the existing KSL digit and the existing ArSL and ASL benchmark datasets. Our proposed model undeniably shows that our fusion approach substantially improves high-performance accuracy in both cases, which proves the system’s superiority.

Attention deficit hyperactivity disorder (ADHD) for children is one of the most common neurodevelopmental disorders and its prevalence has increased globally. Children with ADHD are faced with various difficulties, including inattention, impulsivity, and hyperactivity. Therefore, it is important to use an early detection system that is simple, non-invasive, and automated. Children with ADHD also suffer from other coexisting one or more disorders, including major depressive disorder (MDD), autism spectrum disorder (ASD), etc and it creates more challenges to detect ADHD children. Very few researchers considered such kinds of these comorbidities in their studies to detect ADHD for children. In this work, we proposed a deep learning (DL)-based algorithm to identify ADHD children with coexisting ASD. Functional near-infrared spectroscopy (fNIRs) signals from thirteen ADHD children who have coexisting ASD and fifteen typically developing (TD) children were recorded during the drawing of handwriting patterns. We asked each child to draw periodic lines (PL) and zigzag lines (ZL) under the predict and trace condition and repeated them three times. Finally, a hybrid approach was designed by combining convolutional neural networks (CNN) and bidirectional long short-time memory (Bi-LSTM) to determine children with ADHD who have ASD. The experimental results showed that our proposed hybrid approach could determine ADHD children with coexisting ASD with a classification accuracy of 94.0%, a sensitivity of 89.7%, specificity of 97.8%, f1-score of 93.3%, and AUC of 0.938, respectively, for the PL predict task.

Human activity recognition is one of the most difficult tasks in computer vision. Due to the lack of time information, detecting human activities from still photos is more difficult than sensor-based or video-based techniques. Recently, various deep learning based solutions are being proposed one after another, and their performance is constantly improving. In this paper, we proposed a convolutional neural architecture by ensembling transfer learning based multi-channel attention networks. Here, four CNN branches were used to make feature fusion based ensembling and in each branch, an attention module was used to extract the contextual information from the feature map produced by existing pre-trained models. Finally, the extracted feature maps from four branches were concatenated and fed to fully connected network to produce the final recognition output. We considered 3 different datasets, Stanford 40 actions, BU-101 and Willow human actions datasets to evaluate our system. Experimental analysis showed that the proposed ensembled convolutional architecture outperformed previous works by a noteworthy margin.