Many large and small decisions we make in our daily lives—which ice cream to choose, what research projects to pursue, which partner to marry—require an exploration of alternatives before committing to and exploiting the benefits of a particular choice. Furthermore, many decisions require re-evaluation, and further exploration of alternatives, in the face of changing needs or circumstances. That is, often our decisions depend on a higher level choice: whether to exploit well known but possibly suboptimal alternatives or to explore risky but potentially more profitable ones. How adaptive agents choose between exploitation and exploration remains an important and open question that has received relatively limited attention in the behavioural and brain sciences. The choice could depend on a number of factors, including the familiarity of the environment, how quickly the environment is likely to change and the relative value of exploiting known sources of reward versus the cost of reducing uncertainty through exploration. There is no known generally optimal solution to the exploration versus exploitation problem, and a solution to the general case may indeed not be possible. However, there have been formal analyses of the optimal policy under constrained circumstances. There have also been specific suggestions of how humans and animals may respond to this problem under particular experimental conditions as well as proposals about the brain mechanisms involved. Here, we provide a brief review of this work, discuss how exploration and exploitation may be mediated in the brain and highlight some promising future directions for research.
Humans are adept at switching between goal-directed behaviors quickly and effectively. The prefrontal cortex (PFC) is thought to play a critical role by encoding, updating, and maintaining internal representations of task context in working memory. It has also been hypothesized that the encoding of context representations in PFC is regulated by phasic dopamine gating signals. Here we use multimodal methods to test these hypotheses. First we used functional MRI (fMRI) to identify regions of PFC associated with the representation of context in a working memory task. Next we used single-pulse transcranial magnetic stimulation (TMS), guided spatially by our fMRI findings and temporally by previous event-related EEG recordings, to disrupt context encoding while participants performed the same working memory task. We found that TMS pulses to the right dorsolateral PFC (DLPFC) immediately after context presentation, and well in advance of the response, adversely impacted context-dependent relative to context-independent responses. This finding causally implicates right DLPFC function in context encoding. Finally, using the same paradigm, we conducted high-resolution fMRI measurements in brainstem dopaminergic nuclei (ventral tegmental area and substantia nigra) and found phasic responses after presentation of context stimuli relative to other stimuli, consistent with the timing of a gating signal that regulates the encoding of representations in PFC. Furthermore, these responses were positively correlated with behavior, as well as with responses in the same region of right DLPFC targeted in the TMS experiment, lending support to the hypothesis that dopamine phasic signals regulate encoding, and thereby the updating, of context representations in PFC.
To test the hypothesis that the ability to actively represent and maintain context information in a central function of working memory and that a disturbance in this function contributes to cognitive deficits in schizophrenia, the authors modified 3 tasks--the AX version of the Continuous Performance Test, Stroop, and a lexical disambiguation task--and administered them to patients with schizophrenia as well as to depressed and healthy controls. The results suggest an accentuation of deficits in patients with schizophrenia in context-sensitive conditions and cross-task correlations of performance in these conditions. However, the results do not definitively eliminate the possibility of a generalized deficit. The significance of these findings is discussed with regard to the specificity of deficits in schizophrenia and the hypothesis concerning the neural and cognitive mechanisms that underlie these deficits.