24 May 2024 in Computational Brain & Behavior
A key feature of animal and human decision-making is to balance the exploration of unknown options for information gain (directed exploration) versus selecting known options for immediate reward (exploitation), which is often examined using restless bandit tasks. Recurrent neural network models (RNNs) have recently gained traction in both human and systems neuroscience work on reinforcement learning, due to their ability to show meta-learning of task domains. Here we comprehensively compared the performance of a range of RNN architectures as well as human learners on restless four-armed bandit problems. The best-performing architecture (LSTM network with computation noise) exhibited human-level performance. Computational modeling of behavior first revealed that both human and RNN behavioral data contain signatures of higher-order perseveration, i.e., perseveration beyond the last trial, but this effect was more pronounced in RNNs. In contrast, human learners, but not RNNs, exhibited a positive effect of uncertainty on choice probability (directed exploration). RNN hidden unit dynamics revealed that exploratory choices were associated with a disruption of choice predictive signals during states of low state value, resembling a win-stay-loose-shift strategy, and resonating with previous single unit recording findings in monkey prefrontal cortex. Our results highlight both similarities and differences between exploration behavior as it emerges in meta-learning RNNs, and computational mechanisms identified in cognitive and systems neuroscience work.
24 October 2025 in The Journal of Neuroscience
Disorders characterized by changes in dopamine (DA) neurotransmission are often linked to changes in the temporal discounting of future rewards. Likewise, pharmacological manipulations of DA neurotransmission in healthy individuals modulate temporal discounting, but there is considerable variability in the directionality of reported pharmacological effects, as enhancements and reductions of DA signaling have been linked to both increases and reductions of temporal discounting. This may be due to meaningful individual differences in drug effects and/or false-positive findings in small samples. To resolve these inconsistencies, we (1) revisited pharmacological effects of the DA precursor l -DOPA on temporal discounting in a large sample of N = 76 healthy participants ( n = 44 male) and (2) examined several putative proxy measures for DA to revisit the role of individual differences in a randomized, double-blind placebo–controlled preregistered study ( https://osf.io/a4k9j/ ). Replicating previous findings, higher rewards were discounted less (magnitude effect). Computational modeling using hierarchical Bayesian parameter estimation confirmed that the data in both drug conditions were best accounted for by a nonlinear temporal discounting drift diffusion model. In line with recent animal and human work, l -DOPA reliably reduced the discount rate with a small effect size, challenging earlier findings in substantially smaller samples. We found no credible evidence for effects of putative DA proxy measures on model parameters, calling into question the role of these measures in accounting for individual differences in DA drug effects.
29 August 2024 in bioRxiv
Disorders characterised by changes in dopamine (DA) neurotransmission are often linked to changes in the temporal discounting of future rewards. Likewise, pharmacological manipulations of DA neuro-transmission in healthy individuals modulate temporal discounting, but there is considerable variability in the directionality of reported pharmacological effects, as enhancements and reductions of DA signalling have been linked to both increases and reductions of temporal discounting. This may be due to meaningful individual differences in drug effects and/or false positive findings in small samples. To resolve these inconsistencies, we 1) revisited pharmacological effects of the DA precursor L-DOPA on temporal discounting in a large sample of N = 76 healthy participants (n = 44 male) and 2) examined several putative proxy measures for DA to revisit the role of individual differences in a randomised, double-blind placebo-controlled pre-registered study ( https://osf.io/a4k9j/ ). Replicating previous findings, higher rewards were discounted less (magnitude effect). Computational modelling using hierarchical Bayesian parameter estimation confirmed that the data in both drug conditions were best accounted for by a non-linear temporal discounting drift diffusion model. In line with recent animal and human work, L-DOPA reliably reduced the discount rate with a small effect size, challenging earlier findings in substantially smaller samples. We found no credible evidence for linear or quadratic effects of putative DA proxy measures on model parameters, calling into question the role of these measures in accounting for individual differences in DA drug effects.
28 October 2025 in Computational Brain & Behavior
Artificial neural networks constitute simplified computational models of neural circuits that might help understand how the biological brain solves and represents complex tasks. Previous research revealed that recurrent neural networks (RNNs) with 48 hidden units show human-level performance in restless four-armed bandit tasks but differ from humans with respect to the task strategy employed. Here we systematically examined the impact of network capacity (no. of hidden units) on computational mechanisms and performance. Computational modeling was applied to investigate and compare network behavior between capacity levels as well as between RNNs and human learners. Using a task frequently employed in human cognitive neuroscience work as well as in animal systems neuroscience work, we show that high-capacity networks displayed increased directed exploration and attenuated random exploration relative to low-capacity networks. RNNs with 576 hidden units approached “human-like” exploration strategies, but the overall switch rate and the level of perseveration still deviated from human learners. In the context of the resource-rational framework, which posits a trade-off between reward and policy complexity, human learners may devote more resources to solving the task, albeit without performance benefits over RNNs. Taken together, this work reveals the importance of network capacity on exploration strategies during reinforcement learning and therefore contributes to the goal of building neural networks that behave “human-like” to possibly gain insights into computational mechanisms in human brains.