Brain–computer interfaces (BCIs) have been widely employed to identify and estimate a user’s intention to trigger a robotic device by decoding motor imagery (MI) from an electroencephalogram (EEG). However, developing a BCI system driven by MI related to natural hand-grasp tasks is challenging due to its high complexity. Although numerous BCI studies have successfully decoded large body parts, such as the movement intention of both hands, arms, or legs, research on MI decoding of high-level behaviors such as hand grasping is essential to further expand the versatility of MI-based BCIs. In this study, we propose NeuroGrasp, a dual-stage deep learning framework that decodes multiple hand grasping from EEG signals under the MI paradigm. The proposed method effectively uses an EEG and electromyography (EMG)-based learning, such that EEG-based inference at test phase becomes possible. The EMG guidance during model training allows BCIs to predict hand grasp types from EEG signals accurately. Consequently, NeuroGrasp improved classification performance offline, and demonstrated a stable classification performance online. Across 12 subjects, we obtained an average offline classification accuracy of 0.68 (±0.09) in four-grasp-type classifications and 0.86 (±0.04) in two-grasp category classifications. In addition, we obtained an average online classification accuracy of 0.65 (±0.09) and 0.79 (±0.09) across six high-performance subjects. Because the proposed method has demonstrated a stable classification performance when evaluated either online or offline, in the future, we expect that the proposed method could contribute to different BCI applications, including robotic hands or neuroprosthetics for handling everyday objects.
17 October 2022 in IEEE Transactions on Cybernetics
Electroencephalogram (EEG)-based brain–machine interface (BMI) has been utilized to help patients regain motor function and has recently been validated for its use in healthy people because of its ability to directly decipher human intentions. In particular, neurolinguistic research using EEGs has been investigated as an intuitive and naturalistic communication tool between humans and machines. In this study, the human mind directly decoded the neural languages based on speech imagery using the proposed deep neurolinguistic learning. Through real-time experiments, we evaluated whether BMI-based cooperative tasks between multiple users could be accomplished using a variety of neural languages. We successfully demonstrated a BMI system that allows a variety of scenarios, such as essential activity, collaborative play, and emotional interaction. This outcome presents a novel BMI frontier that can interact at the level of human-like intelligence in real time and extends the boundaries of the communication paradigm.
It is important to operate devices with control panels and touch screens assisted by haptic feedback in mobile environments such as driving automobiles and electric power wheelchairs. A lot of consideration is needed to give accurate haptic feedback, especially, presenting clear touch feedback to the elderly and people with reduced sensation is a very critical issue from healthcare and safety perspectives. In this study, we aimed to identify the perceptual characteristics for the frequency and direction of haptic vibration on the touch screen with vehicle-driving vibration and to propose an efficient haptic system based on these characteristics. As a result, we demonstrated that the detection threshold shift decreased at frequencies above 210 Hz due to the contact pressure during active touch, but the detection threshold shift increased at below 210 Hz. We found that the detection thresholds were 0.30–0.45 gpeak with similar sensitivity in the 80–270 Hz range. The haptic system implemented by reflecting the experimental results achieved characteristics suitable for use scenarios in automobiles. Ultimately, it could provide practical guidelines for the development of touch screens to give accurate touch feedback in the real-world environment.
29 July 2024 in IEEE Transactions on Neural Systems and Rehabilitation Engineering
Robotic arms are increasingly being utilized in shared workspaces, which necessitates the accurate interpretation of human intentions for both efficiency and safety. Electroencephalogram (EEG) signals, commonly employed to measure brain activity, offer a direct communication channel between humans and robotic arms. However, the ambiguous and unstable characteristics of EEG signals, coupled with their broad distribution, make it challenging to collect sufficient data and hinder the calibration performance for new signals, thereby reducing the reliability of EEG-based applications. To address these issues, this study proposes an iteratively calibratable network aimed at enhancing the reliability and efficiency of EEG-based robotic arm control systems. The proposed method integrates feature inputs with network expansion techniques. This integration allows a network trained on an extensive initial dataset to adapt effectively for new users during calibration. Additionally, our approach combines motor imagery with speech imagery datasets to increase not only its intuitiveness but also the number of command classes. The evaluation is conducted in a real-time offline manner, with a robotic arm operating in real time to collect data, which is then analyzed offline. The evaluation results demonstrated that the proposed method outperformed the comparison group in 10 sessions and demonstrated competitive results when combining the two paradigms. Therefore, it was confirmed that the network can be calibrated and personalized using only the new data from new users.
18 February 2025 in IEEE Transactions on Cybernetics
Developing natural, intuitive, and human-centric input systems for mobile human-machine interaction (HMI) poses significant challenges. Existing gaze or gesture-based interaction systems are often constrained by their dependence on continuous visual engagement, limited interaction surfaces, or cumbersome hardware. To address these challenges, we propose MetaSkin, a novel neurohaptic interface that uniquely integrates neural signals with on-skin interaction for bare-handed, eyes-free interaction by exploiting human’s natural proprioceptive capabilities. To support the interface, we developed a deep learning framework that employs multiscale temporal-spectral feature representation and selective feature attention to effectively decode neural signals generated by on-skin touch and motion gestures. In experiments with 12 participants, our method achieved offline accuracies of 81.95% for touch location discrimination, 71.00% for motion type identification, and 46.08% for 10-class touch-motion classification. In pseudo-online settings, accuracies reached 99.43% for touch onset detection, and 80.34% and 67.02% for classification of touch location and motion type, respectively. Neurophysiological analyses revealed distinct neural activation patterns in the sensorimotor cortex, underscoring the efficacy of our multiscale approach in capturing rich temporal and spectral dynamics. Future work will focus on optimizing the system for diverse user populations and dynamic environments, with a long-term goal of advancing human-centered, neuroadaptive interfaces for next-generation HMI systems. This work represents a significant step toward a paradigm shift in design of brain-computer interfaces, bridging sensory and motor paradigms for building more sophisticated systems.