A well-established notion in cognitive neuroscience proposes that multiple brain systems contribute to choice behaviour. These include: (1) a model-free system that uses values cached from the outcome history of alternative actions, and (2) a model-based system that considers action outcomes and the transition structure of the environment. The widespread use of this distinction, across a range of applications, renders it important to index their distinct influences with high reliability. Here we consider the two-stage task, widely considered as a gold standard measure for the contribution of model-based and model-free systems to human choice. We tested the internal/temporal stability of measures from this task, including those estimated via an established computational model, as well as an extended model using drift-diffusion. Drift-diffusion modeling suggested that both choice in the first stage, and RTs in the second stage, are directly affected by a model-based/free trade-off parameter. Both parameter recovery and the stability of model-based estimates were poor but improved substantially when both choice and RT were used (compared to choice only), and when more trials (than conventionally used in research practice) were included in our analysis. The findings have implications for interpretation of past and future studies based on the use of the two-stage task, as well as for characterising the contribution of model-based processes to choice behaviour. In this paper, we report a reliability analysis for the estimation of “model basedness”—a psychological construct that informs a wealth of studies in animal, human and clinical research. We consider an exemplar paradigm, the two-step task, widely used in the recent literature. We report low reliability for model-agnostic model-based estimates, as well as computational model parameter estimates. We suggest how a model-based/free trade-off might affect reaction-time variability in this task, and go on to suggest use of model parameter estimates based on a combination of choice and RT. Finally, we demonstrate that combining choice and RT estimates improves both model-agnostic and algorithmic model-based estimates.
Learning-based therapies, such as cognitive-behavioral therapy, are used worldwide, and their efficacy is endorsed by health and research funding agencies. However, the mechanisms behind both their strengths and their weaknesses are inadequately understood. Here we describe how advances in computational modeling may help formalize and test hypotheses regarding how patients make inferences, which are core postulates of these therapies. Specifically, we highlight the relevance of computations with regard to the development, maintenance, and therapeutic change in psychiatric disorders. A Bayesian approach helps delineate which apparent inferential biases and aberrant beliefs are in fact near-normative, given patients’ current concerns, and which are not. As examples, we formalize three hypotheses. First, high-level dysfunctional beliefs should be treated as beliefs over models of the world. There is a need to test how, and whether, people apply these high-level beliefs to guide the formation of lower level b...
18 July 2019 in Proceedings of the National Academy of Sciences
Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment’s structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challenge this assumption by providing evidence that a putative model-free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well-described 2-step reward decision task where stimulus identity but not spatial-motor aspects of the task predicted reward. We show that participants assigned value to spatial-motor representations despite it being outcome irrelevant. Strikingly, spatial-motor value associations affected behavior across all outcome-relevant features and stages of the task, consistent with credit assignment to low-level state-independent task representations. Individual difference analyses suggested that the impact of spatial-motor value formation was attenuated for individuals who showed greater deployment of goal-directed (model-based) strategies. Our findings highlight a need for a reconsideration of how model-free representations are formed and regulated according to the structure of the environment.
A characteristic of adaptive behavior is its goal-directed nature. An ability to act in a goal-directed manner is progressively refined during development, but this refinement can be impacted by the emergence of psychiatric disorders. Disorders of compulsivity have been framed computationally as a deficit in model-based control, and have been linked also to abnormal frontostriatal connectivity. However, the developmental trajectory of model-based control, including an interplay between its maturation and an emergence of compulsivity, has not been characterized. Availing of a large sample of healthy adolescents (n = 569) aged 14 to 24 y, we show behaviorally that over the course of adolescence there is a within-person increase in model-based control, and this is more pronounced in younger participants. Using a bivariate latent change score model, we provide evidence that the presence of higher compulsivity traits is associated with an atypical profile of this developmental maturation in model-based control. Resting-state fMRI data from a subset of the behaviorally assessed subjects (n = 230) revealed that compulsivity is associated with a less pronounced change of within-subject developmental remodeling of functional connectivity, specifically between the striatum and a frontoparietal network. Thus, in an otherwise clinically healthy population sample, in early development, individual differences in compulsivity are linked to the developmental trajectory of model-based control and a remodeling of frontostriatal connectivity. Significance Goal-directed behavior is impaired in disorders of compulsivity. Here, we characterize the developmental trajectory of model-based control and show a progressive strengthening from adolescence to early adulthood. We found that the presence of compulsivity traits impacts on this trajectory as well as on the degree of remodeling in functional connectivity within frontostriatal circuits. These findings have implications for understanding the interplay between compulsivity, the developmental trajectory of model-based planning, and functional connectivity in frontostriatal circuits.