Most of the available American Sign Language (ASL) words share similar characteristics. These characteristics are usually during sign trajectory which yields similarity issues and hinders ubiquitous application. However, recognition of similar ASL words confused translation algorithms, which lead to misclassification. In this paper, based on fast fisher vector (FFV) and bi-directional Long-Short Term memory (Bi-LSTM) method, a large database of dynamic sign words recognition algorithm called bidirectional long-short term memory-fast fisher vector (FFV-Bi-LSTM) is designed. This algorithm is designed to train 3D hand skeletal information of motion and orientation angle features learned from the leap motion controller (LMC). Each bulk features in the 3D video frame is concatenated together and represented as an high-dimensional vector using FFV encoding. Evaluation results demonstrate that the FFV-Bi-LSTM algorithm is suitable for accurately recognizing dynamic ASL words on basis of prosodic and angle cues. Furthermore, comparison results demonstrate that FFV-Bi-LSTM can provide better recognition accuracy of 98.6% and 91.002% for randomly selected ASL dictionary and 10 pairs of similar ASL words, in leave-one-subject-out cross-validation on the constructed dataset. The performance of our FFV-Bi-LSTM is further evaluated on ASL data set, leap motion dynamic hand gestures data set (LMDHG), and Semaphoric hand gestures contained in the Shape Retrieval Contest (SHREC) dataset. We improve the accuracy of the ASL data set, LMDHG, and SHREC data sets by 2%, 2%, and 3.19% respectively.

Complex hand gesture interactions among dynamic sign words may lead to misclassification, which affects the recognition accuracy of the ubiquitous sign language recognition system. This paper proposes to augment the feature vector of dynamic sign words with knowledge of hand dynamics as a proxy and classify dynamic sign words using motion patterns based on the extracted feature vector. In this method, some double-hand dynamic sign words have ambiguous or similar features across a hand motion trajectory, which leads to classification errors. Thus, the similar/ambiguous hand motion trajectory is determined based on the approximation of a probability density function over a time frame. Then, the extracted features are enhanced by transformation using maximal information correlation. These enhanced features of 3D skeletal videos captured by a leap motion controller are fed as a state transition pattern to a classifier for sign word classification. To evaluate the performance of the proposed method, an experiment is performed with 10 participants on 40 double hands dynamic ASL words, which reveals 97.98% accuracy. The method is further developed on challenging ASL, SHREC, and LMDHG data sets and outperforms conventional methods by 1.47%, 1.56%, and 0.37%, respectively.

One of the essential pre-processing steps of semantic video analysis is the video shot boundary detection (SBD). It is the primary step to segment the sequence of video frames into shots. Many SBD systems using supervised learning have been proposed for years; however, the training process still remains its principal limitation. In this paper, a multi-modal visual features-based SBD framework is employed that aims to analyze the behaviors of visual representation in terms of the discontinuity signal. We adopt a candidate segment selection that performs without the threshold calculation but uses the cumulative moving average of the discontinuity signal to identify the position of shot boundaries and neglect the non-boundary video frames. The transition detection is structurally performed to distinguish candidate segment into a cut transition and a gradual transition, including fade in/out and logo occurrence. Experimental results are evaluated using the golf video clips and the TREC2001 documentary video data set. Results show that the proposed SBD framework can achieve good accuracy in both types of video data set compared with other proposed SBD methods.

Inconsistent hand and body features pose barriers to sign language recognition and translation leading to unsatisfactory models. Existing recognition models are built up on the spatial-temporal depth S_p features. Finding suitable expert features for the S_p model is challenging especially for dynamic sign words because many inconsistent features exist across hand motions and shapes. In this article, we propose IDF-Sign: an efficient and consistent S_p model from a spatial-temporal multivariate pairwise consistency feature ranking (PairCFR) approach. The temporal features are obtained by computing the 3D position vector of skeletal hand joint coordinates, while the spatial features were obtained by taking every ten spatial coordinates in the 3D video frames and averaging it and doing so until the end of the frames. The PairCFR was used to rank and select the best S_p model features at different feature thresholds. We employed a threshold selection to compute a mid-point value of each ranked feature according to its weight. The receiver operating characteristics (ROC) scheme was employed to identify the relationship between the sensitive parameters and the S_p features, and the obtained values were utilized as modeling inputs. To verify the IDF-Sign, we design a real-life experiment with a leap motion sensor (LMS) consisting of ten signers with a total of ninety dynamic sign words. LMS provides the depth videos, since depth videos are too dense for the S_p model to treat directly, we read the depth videos in comma-separated files in real time. Extensive IDF-Sign evaluations using machine learning on ASL, GSL, DSG, and ASL-similar datasets prove the Optimized Forest achieved an average recognition performance of 95%, 78%, 65.07%, and 95.33% of the top-1, respectively.

The aim of this research was to develop a learning recommendation component in an intelligent tutoring system (ITS) that dynamically predicts and adapts to a learner's style. In order to develop a proper ITS, we present an improved knowledge base supporting adaptive learning, which can be achieved by a suitable knowledge construction. This process is illustrated by implementing a web-based online tutor system. In addition, our knowledge structure provides adaptive presentation and personalized learning with the proposed adaptive algorithm, to retrieve content according to individual learner characteristics. To demonstrate the proposed adaptive algorithm, pre-test and post-test were used to evaluate suggestion accuracy of the course in a class for adapting to a learner's style. In addition, pre- and post-testing were also used with students in a real teaching/learning environment to evaluate the performance of the proposed model. The results show that the proposed system can be used to help students or learners achieve improved learning.

To measure the calorie of food which are varied depending on its ingredients and volume in each cooking time, it is required to sense and calculate calories of food before consuming. Based on nutrition knowledge, ingredients that are components of food naturally have different calories. This paper proposes a method of ingredient-based food calorie measurement using nutrition knowledge and thermal information. In this method, an image of the food is first recognized as a type of food, and ingredients of the recognized food are retrieved from the database with their nutrition knowledge and pattern of brightness and thermal images. Simultaneously, the image is segmented into boundaries of ingredient candidates, and all boundaries are then classified into ingredients using fuzzy logic based on their heat pattern and intensities. The classified ingredients from all boundaries are finally calculated for total calories based on area ratio and nutrition knowledge. The performance of the proposed method was evaluated with ten types of Thai curry in which all ingredients were complicatedly mixed, and the results showed 2.21% for software error and 2.28% for hardware error.