3 June 2022 in Science Advances
The integration of deep learning and theories of reinforcement learning (RL) is a promising avenue to explore novel hypotheses on reward-based learning and decision-making in humans and other animals. Here, we trained deep RL agents and mice in the same sensorimotor task with high-dimensional state and action space and studied representation learning in their respective neural networks. Evaluation of thousands of neural network models with extensive hyperparameter search revealed that learning-dependent enrichment of state-value and policy representations of the task-performance-optimized deep RL agent closely resembled neural activity of the posterior parietal cortex (PPC). These representations were critical for the task performance in both systems. PPC neurons also exhibited representations of the internally defined subgoal, a feature of deep RL algorithms postulated to improve sample efficiency. Such striking resemblance between the artificial and biological networks and their functional convergence in sensorimotor integration offers new opportunities to better understand respective intelligent systems.
1 April 2019 in Neuroscience Research
22 December 2022 in Nature Neuroscience
The ability to compose new skills from a preacquired behavior repertoire is a hallmark of biological intelligence. Although artificial agents extract reusable skills from past experience and recombine them in a hierarchical manner, whether the brain similarly composes a novel behavior is largely unknown. In the present study, I show that deep reinforcement learning agents learn to solve a novel composite task by additively combining representations of prelearned action values of constituent subtasks. Learning efficacy in the composite task was further augmented by the introduction of stochasticity in behavior during pretraining. These theoretical predictions were empirically tested in mice, where subtask pretraining enhanced learning of the composite task. Cortex-wide, two-photon calcium imaging revealed analogous neural representations of combined action values, with improved learning when the behavior variability was amplified. Together, these results suggest that the brain composes a novel behavior with a simple arithmetic operation of preacquired action-value representations with stochastic policies.
28 October 2023 in Nature Communications
Learning of adaptive behaviors requires the refinement of coordinated activity across multiple brain regions. However, how neural communications develop during learning remains poorly understood. Here, using two-photon calcium imaging, we simultaneously recorded the activity of layer 2/3 excitatory neurons in eight regions of the mouse dorsal cortex during learning of a delayed-response task. Across learning, while global functional connectivity became sparser, there emerged a subnetwork comprising of neurons in the anterior lateral motor cortex (ALM) and posterior parietal cortex (PPC). Neurons in this subnetwork shared a similar choice code during action preparation and formed recurrent functional connectivity across learning. Suppression of PPC activity disrupted choice selectivity in ALM and impaired task performance. Recurrent neural networks reconstructed from ALM activity revealed that PPC-ALM interactions rendered choice-related attractor dynamics more stable. Thus, learning constructs cortical network motifs by recruiting specific inter-areal communication channels to promote efficient and robust sensorimotor transformation.
1 January 2008 in THE JOURNAL OF JAPAN SOCIETY FOR CLINICAL ANESTHESIA