Accurate capture finger of movements for biomechanical assessments has typically been achieved within laboratory environments through the use of physical markers attached to a participant’s hands. However, such requirements can narrow the broader adoption of movement tracking for kinematic assessment outside these laboratory settings, such as in the home. Thus, there is the need for markerless hand motion capture techniques that are easy to use and accurate enough to evaluate the complex movements of the human hand. Several recent studies have validated lower-limb kinematics obtained with a marker-free technique, OpenPose. This investigation examines the accuracy of OpenPose, when applied to images from single RGB cameras, against a ‘gold standard’ marker-based optical motion capture system that is commonly used for hand kinematics estimation. Participants completed four single-handed activities with right and left hands, including hand abduction and adduction, radial walking, metacarpophalangeal (MCP) joint flexion, and thumb opposition. The accuracy of finger kinematics was assessed using the root mean square error. Mean total active flexion was compared using the Bland–Altman approach, and the coefficient of determination of linear regression. Results showed good agreement for abduction and adduction and thumb opposition activities. Lower agreement between the two methods was observed for radial walking (mean difference between the methods of 5.03°) and MCP flexion (mean difference of 6.82°) activities, due to occlusion. This investigation demonstrated that OpenPose, applied to videos captured with monocular cameras, can be used for markerless motion capture for finger tracking with an error below 11° and on the order of that which is accepted clinically.
This paper introduces a multi-class hand gesture recognition model developed to identify a set of hand gesture sequences from two-dimensional RGB video recordings, using both the appearance and spatiotemporal parameters of consecutive frames. The classifier utilizes a convolutional-based network combined with a long-short-term memory unit. To leverage the need for a large-scale dataset, the model deploys training on a public dataset, adopting a technique known as transfer learning to fine-tune the architecture on the hand gestures of relevance. Validation curves performed over a batch size of 64 indicate an accuracy of 93.95% (±0.37) with a mean Jaccard index of 0.812 (±0.105) for 22 participants. The fine-tuned architecture illustrates the possibility of refining a model with a small set of data (113,410 fully labelled image frames) to cover previously unknown hand gestures. The main contribution of this work includes a custom hand gesture recognition network driven by monocular RGB video sequences that outperform previous temporal segmentation models, embracing a small-sized architecture that facilitates wide adoption.
Human beings adapt their motor patterns in response to their surroundings, utilizing sensory modalities such as visual inputs. This context-informed adaptive motor behavior has increased interest in integrating computer vision algorithms into robotic assistive technologies, marking a shift towards context aware control. However, such integration has rarely been achieved so far, with current methods mostly relying on data-driven approaches. In this study, we introduce a novel control framework for a soft hip exosuit, employing instead a physics-informed computer vision method grounded on geometric modeling of the captured scene for assistance tuning during stairs and level walking. This approach promises to provide a viable solution that is more computationally efficient and does not depend on training examples. Evaluating the controller with six subjects on a path comprising level walking and stairs, we achieved an overall detection accuracy of $93.0\pm 1.1\%$. Computer vision-based assistance provided significantly greater metabolic benefits compared to non-vision-based assistance, with larger energy reductions relative to being unassisted during stair ascent ($-18.9 \pm 4.1\%$ vs. $-5.2 \pm 4.1\%$) and descent ($-10.1 \pm 3.6\%$ vs. $-4.7 \pm 4.8\%$). Such a result is a consequence of the adaptive nature of the device, enabled by the context aware controller, that allowed for more effective walking support: i.e. the assistive torque showed a significant increase while ascending stairs ($+33.9\pm 8.8\%$) and decrease while descending stairs ($-17.4\pm 6.0\%$) compared to a condition without assistance modulation enabled by vision. These results highlight the potential of the approach, promoting effective real-time embedded applications in assistive robotics.
In the field of wearable robotics, assistance needs to be individualized for the user to maximize benefit. Information from muscle fascicles automatically recorded from brightness mode (B-mode) ultrasound has been used to design assistance profiles that are proportional to the estimated muscle force of young individuals. There is also a desire to develop similar strategies for older adults who may have age-altered physiology. This study introduces and validates a ResNet + 2x-LSTM model for extracting fascicle lengths in young and older adults. The labeling was generated in a semimanual manner for young (40,696 frames) and older adults (34,262 frames) depicting B-mode imaging of the medial gastrocnemius. First, the model was trained on young and tested on both young (R2 = 0.85, RMSE = 2.36 ± 1.51 mm, MAPE = 3.6%, aaDF = 0.48 ± 1.1 mm) and older adults (R2 = 0.53, RMSE = 4.7 ± 2.51 mm, MAPE = 5.19%, aaDF = 1.9 ± 1.39 mm). Then, the performances were trained across all ages (R2 = 0.79, RMSE = 3.95 ± 2.51 mm, MAPE = 4.5%, aaDF = 0.67 ± 1.8 mm). Although age-related muscle loss affects the error of the tracking methodology compared to the young population, the absolute percentage error for individual fascicles leads to a small variation of 3–5%, suggesting that the error may be acceptable in the generation of assistive force profiles.
10 September 2024 in IEEE Robotics & Automation Magazine
Robot perception is the capability of a robot to estimate and understand its surroundings to the degree that enables it to navigate and interact with the environment. Right at the core of robot perception lies the problem of building an internal model of the robot’s surroundings using onboard sensor data and prior knowledge. Although the internal model of the environment can be purely geometric (e.g., a point cloud)—as in traditional simultaneous localization and mapping (SLAM)—it can also contain higher-level structures, such as objects and other semantic elements of the scene (e.g., buildings, roads, pedestrians). In this sense, robot perception is related to the topic of scene understanding in the computer vision literature. Due to its crucial role in enabling robotics applications, ranging from navigation to manipulation and human–robot interaction, robot perception has been at the center stage of robotics research for more than 50 years [1] . Similarly, perception has been a core topic in the computer vision community since its inception [2] .