Novelty and surprise play significant roles in animal behavior and in attempts to understand the neural mechanisms underlying it. They also play important roles in technology, where detecting observations that are novel or surprising is central to many applications, such as medical diagnosis, text processing, surveillance, and security. Theories of motivation, particularly of intrinsic motivation, place novelty and surprise among the primary factors that arouse interest, motivate exploratory or avoidance behavior, and drive learning. In many of these studies, novelty and surprise are not distinguished from one another: the words are used more-or-less interchangeably. However, while undeniably closely related, novelty and surprise are very different. The purpose of this article is first to highlight the differences between novelty and surprise and to discuss how they are related by presenting an extensive review of mathematical and computational proposals related to them, and then to explore the implications of this for understanding behavioral and neuroscience data. We argue that opportunities for improved understanding of behavior and its neural basis are likely being missed by failing to distinguish between novelty and surprise.
Humans and other biological agents are able to autonomously learn and cache different skills in the absence of any biological pressure or any assigned task. In this respect, Intrinsic Motivations (i.e. motivations not connected to reward-related stimuli) play a cardinal role in animal learning, and can be considered as a fundamental tool for developing more autonomous and more adaptive artificial agents. In this work, we provide an exhaustive analysis of a scarcely investigated problem: which kind of IM reinforcement signal is the most suitable for driving the acquisition of multiple skills in the shortest time? To this purpose we implemented an artificial agent with a hierarchical architecture that allows to learn and cache different skills. We tested the system in a setup with continuous states and actions, in particular, with a cinematic robotic arm that has to learn different reaching tasks. We compare the results of different versions of the system driven by several different intrinsic motivation signals. The results show a) that intrinsic reinforcements purely based on the knowledge of the system are not appropriate to guide the acquisition of multiple skills, and b) that the stronger the link between the IM signal and the competence of the system, the better the performance.
The effects of striatal dopamine (DA) on behavior have been widely investigated over the past decades, with ``phasic'' burst firings considered as the key expression of a reward prediction error responsible for reinforcement learning. Less well studied is tonic DA, where putative functions include the idea that it is a regulator of vigor, incentive salience, disposition to exert an effort and a modulator of approach strategies. We present a preliminary model combining tonic and phasic DA to show how different outflows triggered by either intrinsically or extrinsically motivating stimuli dynamically affect the basal ganglia by impacting on a selection process that this system performs on the inputs provided by the targeted cortex.The model, which has been tested on the simulated humanoid robot iCub in the interaction with a mechatronic board, shows the putative functions ascribed to DA emerging from the combination of a standard computational mechanism coupled to a differential sensitivity to the presence of DA across the striatum.
Like any other biological trait, communication can be studied from at least four perspectives: mechanistic, ontogenetic, functional, and phylogenetic. In this article, we focus on the following phylogenetic question: how can communication emerge, given that both signal-producing and signal-responding abilities seem to be adaptively neutral until the complementary ability is present in the population? We explore the problem of co-evolution of speakers and hearers with artificial life simulations: a population of artificial neural networks evolving a food call system. The core of the article is devoted to a careful analysis of the complex evolutionary dynamics demonstrated by our simple simulation. Our analyses reveal an important factor, which might solve the phylogenetic problem: the spontaneous production of good (meaningful) signals by speakers because of the need for organisms to categorize their experience in adaptively relevant ways. We discuss our results with respect both to previous simulative work and to the biological literature on the evolution of communication.