Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system’s excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

Background: Recent years have seen a surge of interest in dual-modality imaging systems that integrate functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to probe brain function. This review aims to explore the advancements and clinical applications of this technology, emphasizing the synergistic integration of fNIRS and EEG. Methods: The review begins with a detailed examination of the fundamental principles and distinctive features of fNIRS and EEG techniques. It includes critical technical specifications, data-processing methodologies, and analysis techniques, alongside an exhaustive evaluation of 30 seminal studies that highlight the strengths and weaknesses of the fNIRS-EEG bimodal system. Results: The paper presents multiple case studies across various clinical domains—such as attention-deficit hyperactivity disorder, infantile spasms, depth of anesthesia, intelligence quotient estimation, and epilepsy—demonstrating the fNIRS-EEG system’s potential in uncovering disease mechanisms, evaluating treatment efficacy, and providing precise diagnostic options. Noteworthy research findings and pivotal breakthroughs further reinforce the developmental trajectory of this interdisciplinary field. Conclusions: The review addresses challenges and anticipates future directions for the fNIRS-EEG dual-modal imaging system, including improvements in hardware and software, enhanced system performance, cost reduction, real-time monitoring capabilities, and broader clinical applications. It offers researchers a comprehensive understanding of the field, highlighting the potential applications of fNIRS-EEG systems in neuroscience and clinical medicine.

Multi-robot systems are significant in decision-making capabilities and applications, but avoiding collisions during movement remains a critical challenge. Existing decentralized obstacle avoidance strategies, while low in computational cost, often fail to ensure safety effectively. To address this issue, this paper leverages graph neural networks (GNNs) and deep reinforcement learning (DRL) to aggregate high-dimensional features as inputs for reinforcement learning (RL) to generate paths. Additionally, it introduces safety constraints through an artificial potential field (APF) to optimize these trajectories. Additionally, a constrained nonlinear optimization method further refines the APF-adjusted paths, resulting in the development of the GNN-RL-APF-Lagrangian algorithm. By combining APF and nonlinear optimization techniques, experimental results demonstrate that this method significantly enhances the safety and obstacle avoidance capabilities of multi-robot systems in complex environments. The proposed GNN-RL-APF-Lagrangian algorithm achieves a 96.43% success rate in sparse obstacle environments and 89.77% in dense obstacle scenarios, representing improvements of 59% and 60%, respectively, over baseline GNN-RL approaches. The method maintains scalability up to 30 robots while preserving distributed execution properties.

We investigated the effects of lexical frequency and homophone neighborhood density on the acoustic realization of two neutralizing falling tones in Dalian Mandarin Chinese. Monosyllabic morphemes containing the target tones (Tone 1 and Tone 4) were produced by 60 native speakers from two generations (middle-aged vs. young). The duration of tone-bearing syllable rhymes, as well as the F0 curves and velocity profiles of the lexical tones were quantitatively analyzedvialinear mixed-effects modeling and functional data analysis. Results showed no durational difference between T1 and T4. However, the F0 contours of the two falling tones were incompletely neutralized for both young and middle-aged speakers. Lexical frequency showed little effect on the incomplete tonal neutralization; there were significant differences in the turning point of the two falling tones in syllables with both high and low lexical frequency. However, homophone neighborhood density showed an effect on the incomplete neutralization between the two falling tones, reflected in significant differences in the slope and turning point of the F0 velocity profiles between the two tones carried by syllables with low density but not with high density. Moreover, homophone neighborhood density also affected the duration, the turning point of F0 curves, and velocity profiles of the T1- and T4-syllables. These results are discussed with consideration of social phonetic variations, the theory of Hypo- and Hyper-articulation (H&H), the Neighborhood Activation Model, and communication-based information-theoretic accounts. Collectively, these results broaden our understanding of the effects that lexical properties have on the acoustic details of lexical tone production and tonal sound changes.