In order to implement more accurate and fine-grained monitoring of robot-environment interaction (REI) in real time, this article extends the concept of digital twin (DT) to REI description and proposes a novel semantic-enhanced DT system with exteroceptive and proprioceptive modalities of robots as mechanisms for states monitoring and semantics inference associated with interactions. We create a 3-D generic graphic model embedded with contact dynamics based on domain descriptions and sensor data to mirror a real interactive scene and simultaneously extract the appearance and physical properties of real entities. In addition, we present a multifeature fused visual relations detector and introduce a lightweight spatial relation data set to measure the visual relationships between entity pairs. To keep the DT consistent with the response of a real system, the model is updated with interactive subprocess-triggered control. In our system, the whole framework is developed with virtual reality techniques, which provides a high degree of observability. The experiments on the objects sorting task in a kitchen scene are conducted, and the abilities for system monitoring in real time and semantic reasoning are evaluated; the results demonstrate the effectiveness and feasibility of our proposal.

Learning to achieve a user-specified objective from a random position in unseen environments is challenging for image-guided navigation agents. The abilities of long-horizon reasoning and semantic understanding are still lacking. Inspired by the human memory mechanism, we introduce a neural multi-store memory network to the reinforcement learning framework for target-driven visual navigation. The proposed memory network utilizes three temporal stages of memory to build time dependency for better scene understanding. Sensory memory encodes observations and embeds transient information into working memory, which is short-term and realized by a gated recurrent neural network (RNN). Then, the long-term memory stores the latent state from each step of the RNN into a single slot. Finally, a self-attention reading mechanism is designed to retrieve goal-related information from long-term memory. In addition, to improve the scene generalization capability of the agent, we facilitate training of the visual representation with a self-supervised auxiliary task and image augmentation. This method can navigate agents in unknown visual-realistic environments using only egocentric observations, without the need for any position sensors or pretrained models. The evaluation results on the Matterport3D dataset through the Habitat simulator demonstrate that our method outperforms the state-of-the-art approaches.

The flexible deployment and strong adaptability of unmanned aerial vehicles (UAVs) have great significance in improving communication quality and expanding communication network coverage. However, due to the complex constraints and time-varying characteristics of the communication environment, the dynamic deployment of multi-UAV is challenging to ensure the reliable operation of the UAV-assisted communication systems. To address this challenge, we present a novel approach called long-term communication coverage for ground users through jointly optimizing the multi-UAV deployment and user association (LTCC-UDUA) using deep reinforcement learning (DRL). Firstly, we model the multi-UAV deployment and user association as a decentralized partially observable Markov decision process. Subsequently, we formulate a reward function that accounts for both the communication fairness index for ground users (GUs) and the total system throughput. Finally, continuous optimization of the multi-UAV’s movement trajectories is performed in a centralized training and distributed execution manner. Simulation results demonstrate that our proposed LTCC-UDUA scheme significantly outperforms two commonly used baseline methods in terms of GUs associated, fairness index, and total throughput.

Service robots across industrial, medical, and domestic environments face significant challenges in flattening garments, due to their nearly infinite degrees of freedom and complex dynamic properties. Traditional quasi-static methods for handling fabrics require extensive interactions and are inefficient. While high-velocity dynamic actions enable efficient unfolding, they are unable to perform fine-grained adjustments. To address this, we propose a self-supervised learning framework combining dynamic (Fling) and static (pick and place and pick and drag) manipulation primitives using a dual-arm robot. This method unfolds crumpled garments efficiently without expert demonstrations or hand-labeled data. We introduce a novel factorized reward function based on garment coverage, shape matching, and smoothness. In addition, we employ a spatial action maps network, leveraging a Vision Transformer, to accurately determine grasping points. Our approach significantly reduces interaction steps and improves fabric coverage. We conducted simulations and real-world experiments on 14 garments across six categories. Our approach quickly achieved over 80% coverage within just two actions on long sleeves, surpassing the state-of-the-art by about 9%, and reached 88% –92.9% coverage across all categories within five actions. Our zero-shot sim-to-real transfer has successfully been demonstrated on UR5 robots, proving the effectiveness of the model on various fabrics.

Unmanned aerial vehicles (UAVs) are widely used in disaster or remote areas to provide ubiquitous service. Due to the limited energy and communication range of UAVs, and the operation of UAVs is subject to high uncertainty, current coverage path planning algorithms are not sufficient. Therefore, autonomous dynamic and energy-efficient path planning is still an important research direction for improving coverage efficiency, especially involving multiagent. To address this problem, we introduce a novel trace pheromone into multi-agent reinforcement learning framework for energy-efficient UAV dynamic coverage control, which is termed trace pheromone-based UAV energy-efficient dynamic coverage (TP-EDC). First, we combine multi-agent deep deterministic policy gradient (MADDPG) with a trace pheromone model to serve as a strong tool for building our TP-EDC framework. Meanwhile, the trace pheromones model is integrated into stigmergy mechanism to simulate natural pheromones, which enhances the inner indirect communications among distributed UAVs and avoids network delay. Finally, the intensive simulation results demonstrate that the proposed method can maximize the coverage efficiency by comprehensively considering the coverage rate and energy consumption. Our method also shows significant dynamic coverage performance compared to two well-known baselines methods.

In recent years, data-driven learning methods have been widely studied for autonomous robot skill learning. However, these methods rely on large amounts of robot-environment interaction data for training, which largely prevents them from being applied to real-world robots. To address this problem, this paper proposes a novel simulation-reality closed-loop learning framework for autonomous robot skill learning that can improve data efficiency, enhance policy stability and achieve effective policy simulation-to-reality (sim2real) transfer. First, a hybrid control model combining the asymmetric deep deterministic policy gradients (Asym-DDPG) model and the forward prediction control (FPC) model is proposed to learn vision-based manipulation policies in simulations, which can decompose complex tasks to improve learning efficiency. Second, a novel pixel-level domain adaptation method named Position-CycleGAN is designed to translate real images to simulated images while also preserving the task-related information. The policy trained in simulations can be directly migrated into real robots in a reverse reality-to-simulation manner using the Position-CycleGAN model. The experimental results validate the effectiveness of the proposed framework. This work provides an efficient and feasible path for achieving autonomous skill learning.