Objective: Humans are able to localize the source of a sound. This enables them to direct attention to a particular speaker in a cocktail party. Psycho-acoustic studies show that the sensory cortices of the human brain respond to the location of sound sources differently, and the auditory attention itself is a dynamic and temporally based brain activity. In this work, we seek to build a computational model which uses both spatial and temporal information manifested in EEG signals for auditory spatial attention detection (ASAD). Methods: We propose an end-to-end spatiotemporal attention network, denoted as STAnet, to detect auditory spatial attention from EEG. The STAnet is designed to assign differentiated weights dynamically to EEG channels through a spatial attention mechanism, and to temporal patterns in EEG signals through a temporal attention mechanism. Results: We report the ASAD experiments on two publicly available datasets. The STAnet outperforms other competitive models by a large margin under various experimental conditions. Its attention decision for 1-second decision window outperforms that of the state-of-the-art techniques for 10-second decision window. Experimental results also demonstrate that the STAnet achieves competitive performance on EEG signals ranging from 64 to as few as 16 channels. Conclusion: This study provides evidence suggesting that efficient low-density EEG online decoding is within reach. Significance: This study also marks an important step towards the practical implementation of ASAD in real life applications.

Powered exoskeletons are used to assist human walking, but it is inconvenient for a user to carry a bulky energy-supply system. During walking, human lower limbs accelerate and decelerate alternatively, during which period the human body does positive and negative work, respectively. Therefore, if the negative work performed during deceleration can be harnessed using some assisting device to then assist the acceleration movement of the lower limb, the total metabolic cost of the human body during walking can be reduced. In this work, an unpowered flexible lower-limb exoskeleton is proposed; it is worn in parallel to the lower limbs to assist human walking without consuming external electrical power. The flexible exoskeleton imitates the physiological structure of a human lower limb, consisting of elastic and damping components. When worn on the lower limb, the exoskeleton can partly replace the function of the lower limb muscles, which can scavenge kinetic energy during lower limb deceleration to assist the acceleration movement, so that the biomechanical power consumption can be reduced during walking. In addition to the walking assistance, the generator in the exoskeleton, serving as a damping component, can harvest kinetic energy to produce electricity for powering wearable electronics. A prototype of the flexible exoskeleton was developed, and experiments were carried out to validate the effectiveness. The experiments showed that the lower-limb exoskeleton could reduce the metabolic cost by 3.12% at the walking speed of 4.5 km/h, and a maximum electrical power of 6.47 W was generated at the walking speed of 5.1 km/h.

Objective: Despite recent advances, the decoding of auditory attention from brain signals remains a challenge. A key solution is the extraction of discriminative features from high-dimensional data, such as multi-channel electroencephalography (EEG). However, to our knowledge, topological relationships between individual channels have not yet been considered in any study. In this work, we introduced a novel architecture that exploits the topology of the human brain to perform auditory spatial attention detection (ASAD) from EEG signals. Methods: We propose EEG-Graph Net , an EEG-graph convolutional network, which employs a neural attention mechanism. This mechanism models the topology of the human brain in terms of the spatial pattern of EEG signals as a graph. In the EEG-Graph, each EEG channel is represented by a node, while the relationship between two EEG channels is represented by an edge between the respective nodes. The convolutional network takes the multi-channel EEG signals as a time series of EEG-graphs and learns the node and edge weights from the contribution of the EEG signals to the ASAD task. The proposed architecture supports the interpretation of the experimental results by data visualization. Results: We conducted experiments on two publicly available databases. The experimental results showed that EEG-Graph Net significantly outperforms the state-of-the-art methods in terms of decoding performance. In addition, the analysis of the learned weight patterns provides insights into the processing of continuous speech in the brain and confirms findings from neuroscientific studies. Conclusion: We showed that modeling brain topology with EEG-graphs yields highly competitive results for auditory spatial attention detection. Significance: The proposed EEG-Graph Net is more lightweight and accurate than competing baselines and provides explanations for the results. Also, the architecture can be easily transferred to other brain-computer interface (BCI) tasks.

The continuous control of rehabilitation robots based on surface electromyography (sEMG) is a natural control strategy that can ensure human safety and ease the discomfort of human-machine coupling. However, current models for estimating movement of the upper limb focus on two dimensions movement, and models of three dimensions movement are too complex. In this paper, a simple-structure temporal information-based model for upper limb motion was proposed. An experiment of the multijoint motion of the upper limb was carried out. We studied the touching motion and the compound task motion of the upper limbs. The touch motor task consists of three tasks, namely, shoulder abduction, shoulder forward bend and finger-nose. The compound tasks include driving and fetching objects. Three-dimensional upper limb movement data and sEMG signals of seven upper limb muscles were recorded from seven healthy subjects. Model training was carried out after data preprocessing and feature extraction. Then, 120 s of upper limb motion data was used to verify the performance of the model proposed in this paper. The estimated accuracy of the model for the touch tasks was 0.9171, and it was 0.8109 for compound tasks. Compared to the multilayer perception (MLP) model, a 13.57% reduction in the root-mean-square error (RSME) was observed. The results show that the model has good accuracy for estimating the angular motion of the upper limb and that it has the potential to be applied for three-dimensional motion control in an upper limb mirror-image therapy rehabilitation robot.

Humans have the ability to pay attention to one of the sound sources in a multispeaker acoustic environment. Auditory attention detection (AAD) seeks to detect the attended speaker from one’s brain signals that will enable many innovative human–machine systems. However, effective representation learning of electroencephalography (EEG) signals remains a challenge. In this article, we propose a neural attention mechanism that dynamically assigns differentiated weights to the subbands and the channels of EEG signals to derive discriminative representations for AAD. In the nutshell, we would like to build a computational attention mechanism, i.e., neural attention, to model the auditory attention in human brain. We incorporate the proposed neural attention into an AAD system, and validate the neural attention mechanism through comprehensive experiments on two publicly available datasets. The experimental results demonstrate that the proposed system significantly outperforms the state-of-the-art reference baselines.

Humans possess the remarkable ability to selectively attend to a single speaker amidst competing voices and background noise, known as selective auditory attention . Recent studies in auditory neuroscience indicate a strong correlation between the attended speech signal and the corresponding brain's elicited neuronal activities. In this work, we study such brain activities measured using affordable and non-intrusive electroencephalography (EEG) devices. We present NeuroHeed, a speaker extraction model that leverages the listener's synchronized EEG signals to extract the attended speech signal in a cocktail party scenario, in which the extraction process is conditioned on a neuronal attractor encoded from the EEG signal. We propose both an offline and an online NeuroHeed, with the latter designed for real-time inference. In the online NeuroHeed, we additionally propose an autoregressive speaker encoder, which accumulates past extracted speech signals for self-enrollment of the attended speaker information into an auditory attractor, that retains the attentional momentum over time. Online NeuroHeed extracts the current window of the speech signals with guidance from both attractors. Experimental results on KUL dataset two-speaker scenario demonstrate that NeuroHeed effectively extracts brain-attended speech signals with an average scale-invariant signal-to-noise ratio improvement (SI-SDRi) of 14.3 dB and extraction accuracy of 90.8% in offline settings, and SI-SDRi of 11.2 dB and extraction accuracy of 85.1% in online settings.