In this letter, a sparse Markov decision process (MDP) with novel causal sparse Tsallis entropy regularization is proposed. The proposed policy regularization induces a sparse and multimodal optimal policy distribution of a sparse MDP. The full mathematical analysis of the proposed sparse MDP is provided. We first analyze the optimality condition of a sparse MDP. Then, we propose a sparse value iteration method that solves a sparse MDP and then prove the convergence and optimality of sparse value iteration using the Banach fixed-point theorem. The proposed sparse MDP is compared to soft MDPs that utilize causal entropy regularization. We show that the performance error of a sparse MDP has a constant bound, while the error of a soft MDP increases logarithmically with respect to the number of actions, where this performance error is caused by the introduced regularization term. In experiments, we apply sparse MDPs to reinforcement learning problems. The proposed method outperforms existing methods in terms of the convergence speed and performance.

In this letter, we propose the Interactive Text2Pickup (IT2P) network for human-robot collaboration that enables an effective interaction with a human user despite the ambiguity in user's commands. We focus on the task where a robot is expected to pick up an object instructed by a human, and to interact with the human when the given instruction is vague. The proposed network understands the command from the human user and estimates the position of the desired object first. To handle the inherent ambiguity in human language commands, a suitable question which can resolve the ambiguity is generated. The user's answer to the question is combined with the initial command and given back to the network, resulting in more accurate estimation. The experiment results show that given unambiguous commands, the proposed method can estimate the position of the requested object with an accuracy of 98.49% based on the test dataset. Given ambiguous language commands, we show that the accuracy of the pick up task increases by 1.94 times after incorporating the information obtained from the interaction.

This paper presents a cost-effective motion-planning method for robots operating in complex and realistic environments. While sampling-based path-planning algorithms, such as a rapidly exploring random tree (RRT) and its variants, have been highly effective for general path-planning problems, it is still difficult to find the minimum cost path in a complex space efficiently since RRT-based algorithms extend a search tree locally, requiring a large number of samples before finding a good solution. This paper presents an efficient nonmyopic path-planning algorithm by combining RRT ^* and a stochastic optimization method, called cross entropy. The proposed method constructs two RRT trees: The first tree is a standard RRT ^* tree, which is used to determine the nearest node in the tree to be extended to a randomly chosen point, and the second tree contains the first tree with additional long extensions. By maintaining two separate trees, we can grow the search tree nonmyopically to improve the efficiency of the algorithm while ensuring the asymptotic optimality of RRT ^* . From an extensive set of simulations and experiments using mobile and humanoid robots, we demonstrate that the proposed method consistently finds low-cost paths faster than the existing algorithms.

In this paper, we focus on the problem of learning from demonstration (LfD) where demonstrations with different proficiencies are provided without labeling. To this end, we model multiple policies with different qualities as correlated Gaussian processes and present a leverage optimization method that estimates the leverage of each policy where the difference between two leverages defines the correlation between the corresponding policies. To recover a single policy function of an expert, we present a sparsity constraint on the leverage parameters. We first show that the proposed leverage optimization method can recover the correlations between sensory fields where the fields are realized from correlated Gaussian processes and sensor measurements are collected from the fields. Furthermore, we applied the proposed method to autonomous driving experiments, where demonstrations are collected from three different driving modes. While the driving policies are not realized from correlated processes, the proposed method assigns reasonable leverages to the driving demonstrations. The estimated driving policy of an expert, which incorporates the optimized leverages, outperforms previous LfD methods in terms of both safety and driving quality.

As safety is of paramount importance in robotics, reinforcement learning that reflects safety, called safe RL , has been studied extensively. In safe RL, we aim to find a policy which maximizes the desired return while satisfying the defined safety constraints. There are various types of constraints, among which constraints on conditional value at risk (CVaR) effectively lower the probability of failures caused by high costs since CVaR is a conditional expectation obtained above a certain percentile. In this paper, we propose a trust region-based safe RL method with CVaR constraints, called TRC . We first derive the upper bound on CVaR and then approximate the upper bound in a differentiable form in a trust region. Using this approximation, a subproblem to get policy gradients is formulated, and policies are trained by iteratively solving the subproblem. TRC is evaluated through safe navigation tasks in simulations with various robots and a sim-to-real environment with a Jackal robot from Clearpath. Compared to other safe RL methods, the performance is improved by 1.93 times while the constraints are satisfied in all experiments.