Vision-and-language navigation (VLN) is a challenging task that requires an agent to navigate in real-world environments by understanding natural language instructions and visual information received in real time. Prior works have implemented VLN tasks on continuous environments or physical robots, all of which use a fixed-camera configuration due to the limitations of datasets, such as 1.5-m height, 90° horizontal field of view (HFOV), and so on. However, real-life robots with different purposes have multiple camera configurations, and the huge gap in visual information makes it difficult to directly transfer the learned navigation skills between various robots. In this brief, we propose a visual perception generalization strategy based on meta-learning, which enables the agent to fast adapt to a new camera configuration. In the training phase, we first locate the generalization problem to the visual perception module and then compare two meta-learning algorithms for better generalization in seen and unseen environments. One of them uses the model-agnostic meta-learning (MAML) algorithm that requires few-shot adaptation, and the other refers to a metric-based meta-learning method with a feature-wise affine transformation (AT) layer. The experimental results on the VLN-CE dataset demonstrate that our strategy successfully adapts the learned navigation skills to new camera configurations, and the two algorithms show their advantages in seen and unseen environments respectively.
This study uses production and perception experiments to explore tone 3 focus in Mandarin Chinese. Overall, contrastive focus in Mandarin is clearly marked with increased duration, intensity, and pitch range: in the experiments, listeners identified focused syllables correctly more than 90% of the time. However, a tone 3 syllable offers a smaller capacity for pitch range expansion under focus, and also yields less intensity increase; in addition, local dissimilation increases the duration, intensity, and pitch range of adjacent syllables within the same phrase as a focused tone 3 syllable. As a result, tone 3 focus was less well identified by listeners (77.1%). We suggest that the relatively poor identification of tone 3 focus is due to the smaller capacity for pitch range expansion, the confusion from within-phrase local dissimilatory effects, and the relatively weak intensity of tone 3. This study demonstrates that even within a language where purely prosodic marking of focus is clear, the location of prosodic focus can be difficult to identify in certain circumstances. Our results underline the conclusion, established in other work, that prosodic marking of focus is not universal, but is expressed through the prosodic system of each language.
This study reports experimental results on whether the acoustic realization of vocal emotions differs between Mandarin and English. Prosodic cues, spectral cues and articulatory cues generated by electroglottograph (EGG) of five emotions (anger, fear, happiness, sadness and neutral) were compared within and across Mandarin and English through a production experiment. Results of within-language comparison demonstrated that each vocal emotion had specific acoustic patterns in each language. Moreover, normalized data were used in the across-language comparison analysis. Results indicated that Mandarin and English showed different mechanisms of utilizing pitch for encoding emotions. The differences in pitch variation between neutral and other emotions were significantly larger in English than in Mandarin. However, the variations of speech rate and certain phonation cues (e.g., CPP (Cepstral Peak Prominence) and CQ (Contact quotient)) were significantly greater in Mandarin than in English. The differences in emotional speech between the two languages may be due to the restriction of pitch variation by the presence of lexical tones in Mandarin. This study reveals an interesting finding that occurs when a certain cue (e.g., pitch) is restricted in one language, other cues were strengthened to take on the responsibility of differentiating vocal emotions. Therefore, we posit that the acoustic realizations of emotional speech are multidimensional.
31 December 2014 in The Journal of the Acoustical Society of America
This study reports a finding about vocal expressions of emotion in Mandarin Chinese. Production and perception experiments used the same tone and mixed tone sequences to test whether pitch variation is restricted due to the presence of lexical tones. Results showed that the restriction of pitch variation occurred in all high level tone sequences (tone 1 group) with the expression of happiness but did not happen for other dynamic tone groups. However, perception analysis revealed that all the emotions in every tone group received high identification rates; this indicates that listeners used other cues for encoding happiness in the tone 1 group. This study demonstrates that the restriction of pitch variation does not affect the perception of vocal emotions.
Prior works in vision-and-language navigation (VLN) focus on using long short-term memory (LSTM) to carry the flow of information on either the navigation model (navigator) or the instruction generating model (speaker).The outstanding capability of LSTM to process intermodal interactions has been widely verified; however, LSTM neglects the intramodel interactions, leading to negative effect on either navigator or speaker. The performance of attention-based Transformer is satisfactory in sequence-to-sequence translation domains, but Transformer structure implemented directly in VLN has yet been satisfied. In this article, we propose novel Transformer-based multimodal frameworks for the navigator and speaker, respectively. In our frameworks, the multihead self-attention with the residual connection is used to carry the information flow. Specially, we set a switch to prevent them from directly entering the information flow in our navigator framework. In experiments, we verify the effectiveness of our proposed approach, and show significant performance advantages over the baselines.
Introduction: Language impairments often result from severe neurological disorders, driving the development of neural prosthetics utilizing electrophysiological signals to restore comprehensible language. Previous decoding efforts primarily focused on signals from the cerebral cortex, neglecting subcortical brain structures’ potential contributions to speech decoding in brain-computer interfaces.Methods: In this study, stereotactic electroencephalography (sEEG) was employed to investigate subcortical structures’ role in speech decoding. Two native Mandarin Chinese speakers, undergoing sEEG implantation for epilepsy treatment, participated. Participants read Chinese text, with 1–30, 30–70, and 70–150 Hz frequency band powers of sEEG signals extracted as key features. A deep learning model based on long short-term memory assessed the contribution of different brain structures to speech decoding, predicting consonant articulatory place, manner, and tone within single syllable.Results: Cortical signals excelled in articulatory place prediction (86.5% accuracy), while cortical and subcortical signals performed similarly for articulatory manner (51.5% vs. 51.7% accuracy). Subcortical signals provided superior tone prediction (58.3% accuracy). The superior temporal gyrus was consistently relevant in speech decoding for consonants and tone. Combining cortical and subcortical inputs yielded the highest prediction accuracy, especially for tone.Discussion: This study underscores the essential roles of both cortical and subcortical structures in different aspects of speech decoding.