Multi-robot task assignment is one of the main processes in an intelligent warehouse. This paper models multi-robot task assignment in an intelligent warehouse as an open-path multi-depot asymmetric traveling salesman problem (OP-MATSP). A two-objective integer linear programming (ILP) model for solving OP-MDTSP is proposed. The theoretical bound on the computational time complexity of this model is $O(n!)$. We can solve the small multi-robot task assignment problem by solving the two-objective ILP model using the Gurobi solver. The multi-chromosome coding-based genetic algorithm has a smaller search space, so we use it to solve large-scale problems. The experiment results reveal that the two-objective ILP model is very good at solving small-scale problems. For large-scale problems, both EGA and NSGA3 genetic algorithms can efficiently obtain suboptimal solutions. It demonstrates that this paper’s multi-robot work assignment methods are helpful in an intelligent warehouse.

Reinforcement learning (RL) still suffers from the problem of sample inefficiency and struggles with the exploration issue, particularly in situations with long-delayed rewards, sparse rewards, and deep local optimum. Recently, learning from demonstration (LfD) paradigm was proposed to tackle this problem. However, these methods usually require a large number of demonstrations. In this study, we present a sample efficient teacher-advice mechanism with Gaussian process (TAG) by leveraging a few expert demonstrations. In TAG, a teacher model is built to provide both an advice action and its associated confidence value. Then, a guided policy is formulated to guide the agent in the exploration phase via the defined criteria. Through the TAG mechanism, the agent is capable of exploring the environment more intentionally. Moreover, with the confidence value, the guided policy can guide the agent precisely. Also, due to the strong generalization ability of Gaussian process, the teacher model can utilize the demonstrations more effectively. Therefore, substantial improvement in performance and sample efficiency can be attained. Considerable experiments on sparse reward environments demonstrate that the TAG mechanism can help typical RL algorithms achieve significant performance gains. In addition, the TAG mechanism with soft actor-critic algorithm (TAG-SAC) attains the state-of-the-art performance over other LfD counterparts on several delayed reward and complicated continuous control environments.

Multi-agent path finding (MAPF) is an essential issue for warehouse automation, where multiple agents plan collision-free paths from the start to goal positions. Reinforcement learning (RL) has been employed to develop partially observable distributed MAPF methods that can be scaled to any number of agents. However, existing RL-based MAPF methods still have some limitations in handling redundant information and avoiding deadlock, resulting in a low success rate or longer makespan. This paper proposes a Priority-aware Communication & Experience learning method (PCE), which combines RL with a novel priority-aware multi-agent communication and a new priority-aware deadlock experience replay to tackle this challenge. To be specific, our innovation encompasses two-fold. Our proposed communication mechanism aims to handle redundant information, which establishes a dynamic communication topology based on agents' priorities and proposes a two-head priority-aware graph attention network to aggregate information. In order to help the agent avoid deadlock, we prioritize the expert experience that solves the deadlock when performing experience replay. We conduct multiple simulation experiments on warehouse-like structured grid maps. Compared with the state-of-the-art RL-based MAPF methods, PCE performs significantly better with a higher success rate and lower makespan in small and large MAPF and higher average throughput in the lifelong MAPF, which can further improve the efficiency of warehouse automation. Finally, we validate PCE using three Turtlebot3-Burger robots, which shows that PCE can be applied in real warehouse automation scenarios.

In value-based deep reinforcement learning (RL), value function approximation errors lead to suboptimal policies. Temporal difference (TD) learning is one of the most important methodologies to approximate state-action ( $Q$ ) value function. In TD learning, it is critical to estimate $Q$ values of greedy actions more accurately because a more accurate target $Q$ value enhances the estimation accuracy of $Q$ value. To improve the estimation accuracy of $Q$ value, we propose an action-ranked TD learning method to enhance the performance of deep RL by weighting each TD error according to the rank of its corresponding state-action pair's value among all the $Q$ values on a state. The proposed method can provide more accurate target values for TD learning, making the estimation of the $Q$ value more accurate. We apply the proposed method to a representative value-based deep RL algorithm, and results show that the proposed method outperforms baselines on 31 out of 40 Atari games. Furthermore, we extend the proposed method to multi-agent deep RL. To adaptively determine the hyperparameter in action-ranked TD learning, we propose a meta action-ranked TD learning. A series of experiments quantitatively verify that our methods outperform baselines on Atari games, StarCraft-II, and Grid World environments.