Cyber-physical-social systems (CPSS) are expected to support telemedicine and will be the future of independent rehabilitation. However, a connection between the physical and cognitive as one of the social aspects has not been adequately supported. This paper proposes a multiscopic CPSS framework to investigate this issue by developing hand–object interaction (HOI) recognition with visual attention for the block-design test (BDT). We use three vision systems to extract hand features from block interaction. First, a hand-tracking vision is used to collect the hand-skeletal data and finger joint angle feature. We estimate the physical handling postures at the microscopic level. Second, an egocentric vision with an eye tracker is used to get hand and eye movement at the mesoscopic level. We analyze hand-eye coordination through hand gestures and the focus of visual attention when the hand interacts with the block. Third, an evaluation vision is used to classify the color features in each block at the macroscopic level. The system can recognize whether the design matches the task given. The eight-block design test with two scenarios demonstrates that the system can successfully measure human behavior from physical to cognitive during BDT. We investigate the relationship between physical and cognitive evaluation through the experiment in eight healthy persons. The regression and correlation analyses between the dominant and non-dominant hands show that evaluation indices – skewness-kurtosis of handling posture, attention to pattern, and attention to blocks – could indicate improvement during BDT. We believe this study will benefit clinicians and researchers by providing information unavailable in clinical settings. Code and datasets are available online at https://github.com/anom-tmu/bdt-multiscopic.
Most East Asian rehabilitation centers offer chopsticks manipulation tests (CMT). In addition to impaired hand function, approximately two-thirds of stroke survivors have visual impairment related to eye movement. This article investigates the significance of combining finger joint angle estimation and a visual attention measurement in CMT. We present a multiscopic framework that consists of microscopic, mesoscopic, and macroscopic levels. We develop a feature extraction technique to build the finger kinematic model at the microscopic level. At the mesoscopic level, we propose an active perception ability to detect the position and geometry of the finger on the chopsticks. The proposed framework estimates the proximal interphalangeal (PIP) joint angle on the index finger during CMT using fully connected cascade neural networks (FCC-NN). At the macroscopic level, we implement a cognitive ability by measuring visual attention during CMT. We further evaluate the proposed framework with a conventional test that counts the number of peanuts (NP) which are moved from one bowl to another using chopsticks within a particular time frame. We introduce three evaluation indices, namely joint angle estimation movement (JAEM), chopstick attention movement (CAM), and chopstick tip movement (CTM), by detecting the local minima and maxima of the time series data. According to the experiment results, the velocity of these three evaluation indices could indicate improvement in hand and eye function during CMT. We expect this study to benefit therapists and researchers by providing valuable information that is not accessible in the clinic. Code and datasets are available online at https://github.com/anom-tmu/cmt-attention .
1 July 2023 in International Journal of Advances in Intelligent Informatics
Computer vision-based cyber-physical-social systems (CPSS) are predicted to be the future of independent hand rehabilitation. However, there is a link between hand function and cognition in the elderly that this technology has not adequately supported. To investigate this issue, this paper proposes a multiscopic CPSS framework by developing hand–object interaction (HOI) based on visual attention. First, we use egocentric vision to extract features from hand posture at the microscopic level. With 94.87% testing accuracy, we use three layers of graph neural network (GNN) based on hand skeletal features to categorize 16 grasp postures. Second, we use a mesoscopic active perception ability to validate the HOI with eye tracking in the task-specific reach-to-grasp cycle. With 90.75% testing accuracy, the distance between the fingertips and the center of an object is used as input to a multi-layer gated recurrent unit based on recurrent neural network architecture. Third, we incorporate visual attention into the cognitive ability for classifying multiple objects at the macroscopic level. In two scenarios with four activities, we use GNN with three convolutional layers to categorize some objects. The outcome demonstrates that the system can successfully separate objects based on related activities. Further research and development are expected to support the CPSS application in independent rehabilitation.
19 April 2017 in IOP Conference Series: Materials Science and Engineering
This paper presents a robust control method to relieve the sloshing of liquid container transport using Active Force Control (AFC) method. A model of two degree-of-freedom (2-DOF) liquid container transfer was implemented in this research as the main dynamical system to be controlled. The surface of liquid is maintained in a flat position, so that changes the slope of liquid surface countered by changing the acceleration of container. The focus of this research is how to use AFC method being applied to the system, so that it can suppress liquid sloshing. The control scheme were simulated, compare between PID-AFC and pure PID. Simulations has been conducted, the results show that the PID-AFC have superior performance to suppress the sloshing compared with pure PID, especially if disturbance occurred.
23 October 2019 in International Journal on Advanced Science, Engineering and Information Technology