Energy consumption is expected to be reduced while maintaining high productivity for container handling. This paper investigates a new energy-efficient scheduling problem of automated container terminals, in which quay cranes (QCs) and lift automated guided vehicles (AGVs) cooperate to handle inbound and outbound containers. In our scheduling problem, operation times and task sequences are both to be determined. The underlying optimization problem is mixed-integer nonlinear programming (MINLP). To deal with its computational intractability, a customized and efficient genetic algorithm (GA) is developed to solve the studied MINLP problem, and lexicographic and weighted-sum strategies are further considered. An $\epsilon $ -constraint algorithm is also developed to analyze the Pareto frontiers. Comprehensive experiments are tested on a container handling benchmark system, and the results show the effectiveness of the proposed lexicographic GA, compared to results obtained with two commonly-used metaheuristics, a commercial MINLP solver, and two state-of-the-art methods.
Many methods for reinforcement learning use symbolic representations--nonemergent--such as Q-learning. We use emergent representations here, without human handcrafted symbolic states (i.e., each state corresponds to a different location). This paper models reinforcement learning for hidden neurons in emergent networks for sequential tasks. In this paper, their influences on sequential tasks (e.g., robot navigation in different scenarios) are investigated where the learned value and results of a behavior rely on not only the current experience just like in a pattern recognition (episodic) but also the prediction of future experiences (e.g., delayed rewards) and environments (e.g., previously learned navigational trajectories). We show that this new model of motivated learning amounts to the computation of the maximum-likelihood estimate through ``life'' where punishment and reward have increased weights. This new formulation avoids the greediness of time-discount in Q-learning. Its complex nonlinear sequential optimization has been solved in a closed-form procedure under the condition of the limited computational resources and limited learning experience so far, because we convert it into a simpler problem of incremental and linear estimation. The experimental results showed that the serotonin and dopamine systems speed up learning for sequential tasks, because not all events are equally important. As far as we know, this is the first work that studies the influences of reinforcers (via serotonin and dopamine) on hidden neurons (Y neurons) for sequential tasks in dynamic scenarios using emergent representations.
10 February 2022 in IEEE Transactions on Cognitive and Developmental Systems
How to achieve flexible behavioral decision-making in dynamic environment is an important prerequisite for a mobile robot to execute various tasks. Traditional methods suffer from discontinuous and unstable learning ability. This paper brings the mobile robot continuous and stable learning ability in the dynamic environment, through improving the binding manner between the cerebellum and basal ganglia in the neuromodulatory system proposed in our previous work, thus to enhance its flexible behavioral decision-making performance. Moreover, a more biological significance index, i.e., the curiosity index, Cur, is designed to mimic the activity switch of the locus coeruleus between the tonic and phasic mode, to modulate the explorationexploitation tradeoff in the reinforcement learning of the basal ganglia. Influence of varying learning rate and varying discount rate on the performance of the reinforcement learning is also investigated. Experiment results in static and dynamic environments, as well as the comparative experiment in complex maze environment and the complex dynamic environment, demonstrate the potential of the proposed neuromodulatory system.
4 November 2020 in IEEE Transactions on Cognitive and Developmental Systems
Online model-free reinforcement learning approaches play a crucial role in coping with the real-world applications such as the behavioral decision-making in robotics. How to balance the exploration and exploitation processes is a central problem in reinforcement learning. A balanced ratio of exploration/exploitation has a great influence on the total learning time and the quality of the learned strategy. Therefore, various action selection policies have been presented to obtain a balance between the exploration and exploitation procedures. However, these approaches are rarely automatically and dynamically regulated to the environment variations. One of the most amazing self-adaptation mechanisms in animals is their capacity to dynamically switch between exploration and exploitation strategies. This paper proposes a novel neurophysiologically motivated model which simulates the role of medial prefrontal cortex (MPFC) and lateral prefrontal cortex (LPFC) in behavior decision. The sensory input is transmitted to the MPFC, then the ventral tegmental area (VTA) receives a reward and calculates a dopaminergic reinforcement signal, and the feedback categorization neurons in anterior cingulate cortex (ACC) calculate the vigilance according to the dopaminergic reinforcement signal. Then the vigilance is transformed to LPFC to regulate the exploration rate, finally the exploration rate is transmitted to thalamus to calculate the corresponding action probability. This action selection mechanism is introduced to the actor-critic model of the basal ganglia, combining with the cerebellum model based on the developmental network to construct a new hybrid neuromodulatory model to select the action of the agent. Both the simulation comparison with other four traditional action selection policies and the physical experiment results demonstrate the potential of the proposed neuromodulatory model in action selection.