Coherent structures appear in a concentrated suspension of swimming bacteria. While transport phenomena in a suspension have been studied extensively, how energy is transported from the individual cell scale to the larger mesoscale remains unclear. In this study, we carry out the first successful measurement of the three-dimensional velocity field in a dense suspension of bacteria. The results show that most of the energy generated by individual bacteria dissipates on the cellular scale. Only a small amount of energy is transported to the mesoscale, but the gain in swimming velocity and mass transport due to mesoscale coherent structures is enormous. These results indicate that collective swimming of bacteria is efficient in terms of energy. This paper sheds light on how energy can be transported toward smaller wave numbers in the Stokes flow regime.

Conventional reinforcement learning algorithms have several parameters which determine the feature of learning process, called meta-parameters. In this study, we focus on the discount factor that influences the time scale of the tradeoff between immediate and delayed rewards. The discount factor is usually considered as a constant value, but we introduce the state-dependent discount function and a new optimization criterion for the reinforcement learning algorithm. We first derive a new algorithm under the criterion, named ExQ-learning and we prove that the algorithm converges to the optimal action-value function in the meaning of new criterion w.p.1. We then present a framework to optimize the discount factor and the discount function by using an evolutionary algorithm. In order to validate the proposed method, we conduct a simple computer simulation and show that the proposed algorithm can find an appropriate state-dependent discount function with which performs better than that with a constant discount factor.

Homeostasis is a self-regulatory process, wherein an organism maintains a specific internal physiological state. Homeostatic reinforcement learning (RL) is a framework recently proposed in computational neuroscience to explain animal behavior. Homeostatic RL organizes the behaviors of autonomous embodied agents according to the demands of the internal dynamics of their bodies, coupled with the external environment. Thus, it provides a basis for real-world autonomous agents, such as robots, to continually acquire and learn integrated behaviors for survival. However, prior studies have generally explored problems pertaining to limited size, as the agent must handle observations of such coupled dynamics. To overcome this restriction, we developed an advanced method to realize scaled-up homeostatic RL using deep RL. Furthermore, several rewards for homeostasis have been proposed in the literature. We identified that the reward definition that uses the difference in drive function yields the best results. We created two benchmark environments for homeostasis and performed a behavioral analysis. The analysis showed that the trained agents in each environment changed their behavior based on their internal physiological states. Finally, we extended our method to address vision using deep convolutional neural networks. The analysis of a trained agent revealed that it has visual saliency rooted in the survival environment and internal representations resulting from multimodal input.

One of the essential aspect in biological agents is dynamic stability. This aspect, called homeostasis, is widely discussed in ethology, neuroscience and during the early stages of artificial intelligence. Ashby’s homeostats are general-purpose learning machines for stabilizing essential variables of the agent in the face of general environments. However, despite their generality, the original homeostats couldn’t be scaled because they searched their parameters randomly. In this paper, first we re-define the objective of homeostats as the maximization of a multi-step survival probability from the view point of sequential decision theory and probabilistic theory. Then we show that this optimization problem can be treated by using reinforcement learning algorithms with special agent architectures and theoretically-derived intrinsic reward functions. Finally we empirically demonstrate that agents with our architecture automatically learn to survive in a given environment, including environments with visual stimuli. Our survival agents can learn to eat food, avoid poison and stabilize essential variables through theoretically-derived single intrinsic reward formulations.

The continual generation of behaviors that satisfy all conflicting demands that cannot be satisfied simultaneously, is a situation that is seen naturally in autonomous agents such as long-term operating household robots, and in animals in the natural world. Homeostatic reinforcement learning (homeostatic RL) is known as a bio-inspired framework that achieves such multiobjective control through behavioral optimization. Homeostatic RL achieves autonomous behavior optimization using only internal body information in complex environmental systems, including continuous motor control. However, it is still unknown whether the resulting behaviors actually have the similar long-term properties as real animals. To clarify this issue, this study focuses on the balancing of multiple nutrients in animal foraging as a situation in which such multiobjective control is achieved in animals in the natural world. We then focus on the nutritional geometry framework, which can quantitatively handle the long-term characteristics of foraging strategies for multiple nutrients in nutritional biology, and construct a similar verification environment to show experimentally that homeostatic RL agents exhibit long-term foraging characteristics seen in animals in nature. Furthermore, numerical simulation results show that the long-term foraging characteristics of the agent can be controlled by changing the weighting for the agent’s multiobjective motivation. These results show that the long-term behavioral characteristics of homeostatic RL agents that perform behavioral emergence at the motor control level can be predicted and designed based on the internal dynamics of the body and the weighting of motivation, which change in real time.