Offline reinforcement learning leverages previously collected offline datasets to learn optimal policies with no necessity to access the real environment. Such a paradigm is also desirable for multi-agent reinforcement learning (MARL) tasks, given the combinatorially increased interactions among agents and with the environment. However, in MARL, the paradigm of offline pre-training with online fine-tuning has not been studied, nor even datasets or benchmarks for offline MARL research are available. In this paper, we facilitate the research by providing large-scale datasets and using them to examine the usage of the decision transformer in the context of MARL. We investigate the generalization of MARL offline pre-training in the following three aspects: 1) between single agents and multiple agents, 2) from offline pretraining to online fine tuning, and 3) to that of multiple downstream tasks with few-shot and zero-shot capabilities. We start by introducing the first offline MARL dataset with diverse quality levels based on the StarCraftII environment, and then propose the novel architecture of multi-agent decision transformer (MADT) for effective offline learning. MADT leverages the transformer’s modelling ability for sequence modelling and integrates it seamlessly with both offline and online MARL tasks. A significant benefit of MADT is that it learns generalizable policies that can transfer between different types of agents under different task scenarios. On the StarCraft II offline dataset, MADT outperforms the state-of-the-art offline reinforcement learning (RL) baselines, including BCQ and CQL. When applied to online tasks, the pre-trained MADT significantly improves sample efficiency and enjoys strong performance in both few-short and zero-shot cases. To the best of our knowledge, this is the first work that studies and demonstrates the effectiveness of offline pre-trained models in terms of sample efficiency and generalizability enhancements for MARL.

In this work, a robust insertion control method for precision peg-in-hole assembly is developed by visually supervising the deformation of the passive compliance based on microscopic vision and force information. The advantage of the proposed method is the assembly task can be conducted with pose alignment errors since the passive compliance is supervised by microscopic vision. This significantly decreases the precision requirement for parts alignment and thus contributes to the precision assembly. Specifically, the relationship between the contact status among parts and the deformations of the passive compliance is firstly analyzed. Afterwards, the vision based method to estimate the 3D deformation of the passive compliance is discussed. Finally, a hybrid insertion controller is proposed combining both the force and position feedbacks. Experiments were designed to validate the effectiveness of the proposed method and the significance of introducing vision supervision into the insertion process.

Generating explanations for reinforcement learning (RL) is challenging as actions may produce long-term effects on the future. In this paper, we develop a novel framework for explainable RL by learning a causal world model without prior knowledge of the causal structure of the environment. The model captures the influence of actions, allowing us to interpret the long-term effects of actions through causal chains, which present how actions influence environmental variables and finally lead to rewards. Different from most explanatory models which suffer from low accuracy, our model remains accurate while improving explainability, making it applicable in model-based learning. As a result, we demonstrate that our causal model can serve as the bridge between explainability and learning.

The problem of simultaneous precision assembly of multiple objects is quite practical one to form compact physical structures and functionalities in mechatronics and advanced robotics. The core research aspects facing the problem are the contact status perception between each two object and the motion planning of each separate object. These two aspects mutually affect each other and cannot be discussed separately. In this paper, we first strategically discuss the possible approaches to solve the simultaneous assembly problem and analyze their advantages and drawbacks. Then, a probabilistic control method is developed based on the incomplete perceived information of the assembly process, which can achieve the highest assembly efficiency from the strategic perspective. Specifically, by fully utilizing the mechanical properties of materials in micrometer scale, the interaction between objects is first characterized as stochastic state-transition process. Second, adopting the simultaneous feeding strategy instead of serial feeding, the current contact status between each two object is determined based on the state-transition equation as a probability distribution along a hyperline. Finally, the motion planning technique is designed taking all possible radial forces on every contact surface into consideration. The experimental results demonstrate the effectiveness of the proposed method.

The inherent hysteresis property of piezoelectric actuator (PEA) brings challenges to its modeling and control. This paper proposes a model learning method that is suitable for both forward and inverse PEA models. The hysteresis property is learned based on least squares support vector machines (LS-SVMs). A larger dataset is used for training LS-SVM to guarantee a good generalization performance. Support vectors pruning is utilized to reduce the model complexity. The rate-dependent property of PEA is identified as a linear dynamic submodel. Moreover, a pointing control system with two dual-PEA-axis steering mirrors is developed, which can regulate the 4-degree-of-freedom pose of a laser beam. The coordinated control of four PEAs is realized based on the Jacobian matrix. The learned inverse PEA models are used for the feedforward compensation of each PEA's nonlinearity. A series of experiments were conducted to evaluate the proposed method's effectiveness.