15 May 2019 in The Journal of Neuroscience
The auditory system converts the physical properties of a sound waveform to neural activities and processes them for recognition. During the process, the tuning to amplitude modulation (AM) is successively transformed by a cascade of brain regions. To test the functional significance of the AM tuning, we conducted single-unit recording in a deep neural network (DNN) trained for natural sound recognition. We calculated the AM representation in the DNN and quantitatively compared it with those reported in previous neurophysiological studies. We found that an auditory-system-like AM tuning emerges in the optimized DNN. Better-recognizing models showed greater similarity to the auditory system. We isolated the factors forming the AM representation in the different brain regions. Because the model was not designed to reproduce any anatomical or physiological properties of the auditory system other than the cascading architecture, the observed similarity suggests that the AM tuning in the auditory system might also be an emergent property for natural sound recognition during evolution and development. SIGNIFICANCE STATEMENT This study suggests that neural tuning to amplitude modulation may be a consequence of the auditory system evolving for natural sound recognition. We modeled the function of the entire auditory system; that is, recognizing sounds from raw waveforms with as few anatomical or physiological assumptions as possible. We analyzed the model using single-unit recording, which enabled a fair comparison with neurophysiological data with as few methodological biases as possible. Interestingly, our results imply that frequency decomposition in the inner ear might not be necessary for processing amplitude modulation. This implication could not have been obtained if we had used a model that assumes frequency decomposition.
29 November 2020 in Quarterly Journal of Experimental Psychology
Sustained attention plays an important role in adaptive behaviours in everyday activities. As previous studies have mostly focused on vision, and attentional resources have been thought to be specific to sensory modalities, it is still unclear how mechanisms of attentional fluctuations overlap between visual and auditory modalities. To reduce the effects of sudden stimulus onsets, we developed a new gradual-onset continuous performance task (gradCPT) in the auditory domain and compared dynamic fluctuation of sustained attention in vision and audition. In the auditory gradCPT, participants were instructed to listen to a stream of narrations and judge the gender of each narration. In the visual gradCPT, they were asked to observe a stream of scenery images and indicate whether the scene was a city or mountain. Our within-individual comparison revealed that auditory and visual attention are similar in terms of the false alarm rate and dynamic properties including fluctuation frequency. Absolute timescales of the fluctuation in the two modalities were comparable, notwithstanding the difference in stimulus onset asynchrony. The results suggest that fluctuations of visual and auditory attention are underpinned by common principles and support models with a more central, modality-general controller.
20 March 2025 in Network Neuroscience
Sustained attention is essential for daily life and can be directed to information from different perceptual modalities, including audition and vision. Recently, cognitive neuroscience has aimed to identify neural predictors of behavior that generalize across datasets. Prior work has shown strong generalization of models trained to predict individual differences in sustained attention performance from patterns of fMRI functional connectivity. However, it is an open question whether predictions of sustained attention are specific to the perceptual modality in which they are trained. In the current study, we test whether connectome-based models predict performance on attention tasks performed in different modalities. We show first that a predefined network trained to predict adults’ visual sustained attention performance generalizes to predict auditory sustained attention performance in three independent datasets (N1 = 29, N2 = 60, N3 = 17). Next, we train new network models to predict performance on visual and auditory attention tasks separately. We find that functional networks are largely modality general, with both model-unique and shared model features predicting sustained attention performance in independent datasets regardless of task modality. Results support the supposition that visual and auditory sustained attention rely on shared neural mechanisms and demonstrate robust generalizability of whole-brain functional network models of sustained attention.
10 August 2023 in Cerebral Cortex
Attention levels fluctuate during the course of daily activities. However, factors underlying sustained attention are still unknown. We investigated mechanisms of sustained attention using psychological, neuroimaging, and neurochemical approaches. Participants were scanned with functional magnetic resonance imaging (fMRI) while performing gradual-onset, continuous performance tasks (gradCPTs). In gradCPTs, narrations or visual scenes gradually changed from one to the next. Participants pressed a button for frequent Go trials as quickly as possible and withheld responses to infrequent No-go trials. Performance was better for the visual gradCPT than for the auditory gradCPT, but the 2 were correlated. The dorsal attention network was activated during intermittent responses, regardless of sensory modality. Reaction-time variability of gradCPTs was correlated with signal changes (SCs) in the left fronto-parietal regions. We also used magnetic resonance spectroscopy (MRS) to measure levels of glutamate–glutamine (Glx) and γ-aminobutyric acid (GABA) in the left prefrontal cortex (PFC). Glx levels were associated with performance under undemanding situations, whereas GABA levels were related to performance under demanding situations. Combined fMRI–MRS results demonstrated that SCs of the left PFC were positively correlated with neurometabolite levels. These findings suggest that a neural balance between excitation and inhibition is involved in attentional fluctuations and brain dynamics.
25 April 2023 in Journal of Neuroscience
Natural sounds contain rich patterns of amplitude modulation (AM), which is one of the essential sound dimensions for auditory perception. The sensitivity of human hearing to AM measured by psychophysics takes diverse forms depending on the experimental conditions. Here, we address with a single framework the questions of why such patterns of AM sensitivity have emerged in the human auditory system and how they are realized by our neural mechanisms. Assuming that optimization for natural sound recognition has taken place during human evolution and development, we examined its effect on the formation of AM sensitivity by optimizing a computational model, specifically, a multilayer neural network, for natural sound (namely, everyday sounds and speech sounds) recognition and simulating psychophysical experiments in which the AM sensitivity of the model was assessed. Relatively higher layers in the model optimized to sounds with natural AM statistics exhibited AM sensitivity similar to that of humans, although the model was not designed to reproduce human-like AM sensitivity. Moreover, simulated neurophysiological experiments on the model revealed a correspondence between the model layers and the auditory brain regions. The layers in which human-like psychophysical AM sensitivity emerged exhibited substantial neurophysiological similarity with the auditory midbrain and higher regions. These results suggest that human behavioral AM sensitivity has emerged as a result of optimization for natural sound recognition in the course of our evolution and/or development and that it is based on a stimulus representation encoded in the neural firing rates in the auditory midbrain and higher regions.SIGNIFICANCE STATEMENTThis study provides a computational paradigm to bridge the gap between the behavioral properties of human sensory systems as measured in psychophysics and neural representations as measured in nonhuman neurophysiology. This was accomplished by combining the knowledge and techniques in psychophysics, neurophysiology, and machine learning. As a specific target modality, we focused on the auditory sensitivity to sound AM. We built an artificial neural network model that performs natural sound recognition and simulated psychophysical and neurophysiological experiments in the model. Quantitative comparison of a machine learning model with human and nonhuman data made it possible to integrate the knowledge of behavioral AM sensitivity and neural AM tunings from the perspective of optimization to natural sound recognition.
15 July 2024 in Philosophical Transactions of the Royal Society B: Biological Sciences
Sound serves as a potent medium for emotional well-being, with phenomena like the autonomous sensory meridian response (ASMR) showing a unique capacity for inducing relaxation and alleviating stress. This study aimed to understand how tingling sensations (and, for comparison, pleasant feelings) that such videos induce relate to acoustic features, using a broader range of ASMR videos as stimuli. The sound texture statistics and their timing predictive of tingling and pleasantness were identified through L1-regularized linear regression. Tingling was well-predicted ( r = 0.52), predominantly by the envelope of frequencies near 5 kHz in the 1500 to 750 ms period before the response: stronger tingling was associated with a lower amplitude around the 5 kHz frequency range. This finding was further validated using an independent set of ASMR sounds. The prediction of pleasantness was more challenging ( r = 0.26), requiring a longer effective time window, threefold that for tingling. These results enhance our understanding of how specific acoustic elements can induce tingling sensations, and how these elements differ from those that induce pleasant feelings. Our findings have potential applications in optimizing ASMR stimuli to improve quality of life and alleviate stress and anxiety, thus expanding the scope of ASMR stimulus production beyond traditional methods. This article is part of the theme issue ‘Sensing and feeling: an integrative approach to sensory processing and emotional experience’.