It has previously been shown that the relative reliability of model-based and model-free reinforcement-learning (RL) systems plays a role in the allocation of behavioral control between them. However, the role of task complexity in the arbitration between these two strategies remains largely unknown. Here, using a combination of novel task design, computational modelling, and model-based fMRI analysis, we examined the role of task complexity alongside state-space uncertainty in the arbitration process. Participants tended to increase model-based RL control in response to increasing task complexity. However, they resorted to model-free RL when both uncertainty and task complexity were high, suggesting that these two variables interact during the arbitration process. Computational fMRI revealed that task complexity interacts with neural representations of the reliability of the two systems in the inferior prefrontal cortex.
Recent studies in decision neuroscience revealed that two different strategies guide trial-by-trial choice behavior during reinforcement learning (RL): a goal-directed strategy and a habitual learning strategy. An increasing number of studies have provided evidence for neural substrates underlying respective strategies, suggesting that a single RL algorithm cannot make accurate predictions about individual choices. Despite rapid progress in predicting humans intentions such as movements or choices, there has been no attempt to read out human learning strategies. We proposed a novel brain-computer interface (BCI) framework for decoding human learning strategies from electroencephalography (EEG) data. To circumvent the issue that there is no gold standard for labeling learning strategies, we trained the proposed framework classifier to best match predictions of the computational model of the neural process underlying human learning strategies. The simulation used a 1D, 2D, and 3D convolutional neural network (CNN) on 18 subjects; the EEG data demonstrated that the proposed framework successfully read out human learning strategies with very high accuracy (98.4%). Subsequently, we examined whether those learning strategies labeled from the computational model exhibited distinctive EEG patterns. We used class activation mapping (CAM) for visualization, and we identified distinctive strategy-dependent patterns in the EEG feature space. We argue that the proposed framework has great potential for decoding high-level cognitive states.
Human reinforcement learning (RL) has been known to utilize two distinctive learning strategies, model-based (MB) and model-free (MF) RL. The process of arbitration between MB and MF is thought to be located in the ventrolateral prefrontal cortex and frontopolar cortex. These loci are near the cortex, so we expect the related information can be represented in EEG signals. However, EEG signal patterns considering the arbitration of RL has not been investigated. In this paper, we tested a EEG-based classification model to separate these two different types of trials, each of which is meant to promote MB and MF RL. We found, for the first time, firm evidence to indicate that information pertaining to learning strategies is represented in prefrontal EEG signals.
The analysis of EEG data has been widely utilized due to its non-invasive nature. However, the typically noisy characteristics of EEG signals necessitate artifact removal processes. These processes can involve excessive computation or additional devices, significantly compromising the high usability inherent to non-invasive techniques. Although deep learning was expected to overcome these limitations, it still requires artifact removal and thus becomes highly inefficient. Moreover, it suffers from a lack of interpretability due to its black-box nature. To address these challenges, we propose and validate a framework utilizing Graph Convolutional Network (GCN) and PGExplainer in this study. Specifically, to demonstrate the noise-robustness of our model, we applied it to classify hyperactive disorders such as Tic disorder and Attention deficit hyperactivity disorder (ADHD), which generate excessive noise due to movement. Using EEG data from patients with tics and ADHD, we demonstrated that our model predicts at least 20% higher accuracy than any other neural network models, while the analysis with PGExplainer showed that the same channels were identified as important whether or not noise was removed, indicating noise-robust interpretability of the model. Additionally, this method allowed us to increase computational efficiency, thus identifying a maximally efficient model, i.e., a model that balances accuracy with minimal complexity. The proposed framework is not limited to our case of hyperactive disorder, but is expandable to more general cases. This work demonstrates that GCN-based classification can provide a noise-robust and interpretable solution for EEG analysis.
We use efficient coding principles borrowed from sensory neuroscience to derive the optimal population of neurons to encode rewards from a probability distribution. We find that the response properties of dopaminergic reward prediction error neurons in a rodent and a primate data set are similar to those of the efficient code in many ways: the neurons have a broad distribution of midpoints covering the reward distribution; neurons with higher thresholds have higher gains, more convex tuning functions, and lower slopes; moreover, their slope is higher when the reward distribution is narrower. Furthermore, we derive learning rules that converge to this efficient code; the learning rule for the position of the neuron on the reward axis closely resembles the learning rule of distributional reinforcement learning. Thus, reward prediction error neuron responses may be optimized to broadcast an efficient reward signal, forming a connection between efficient coding and reinforcement learning, two of the most successful theories in computational neuroscience.
Continual reinforcement learning (CRL) agents face significant challenges when encountering distributional shifts. This paper formalizes these shifts into two key scenarios, namely virtual drift (domain switches), where object semantics change (e.g., walls becoming lava), and concept drift (task switches), where the environment’s structure is reconfigured (e.g., moving from object navigation to a door key puzzle). This paper demonstrates that while conventional convolutional neural networks (CNNs) struggle to preserve relational knowledge during these transitions, graph convolutional networks (GCNs) can inherently mitigate catastrophic forgetting by encoding object interactions through explicit topological reasoning. A unified framework is proposed that integrates GCN-based state representation learning with a proximal policy optimization (PPO) agent. The GCN’s message-passing mechanism preserves invariant relational structures, which diminishes performance degradation during abrupt domain switches. Experiments conducted in procedurally generated MiniGrid environments show that the method significantly reduces catastrophic forgetting in domain switch scenarios. While showing comparable mean performance in task switch scenarios, our method demonstrates substantially lower performance variance (Levene’s test, ), indicating superior learning stability compared to CNN-based methods. By bridging graph representation learning with robust policy optimization in CRL, this research advances the stability of decision-making in dynamic environments and establishes GCNs as a principled alternative to CNNs for applications requiring stable, continual learning.