The view planning (VP) problem in robotic active vision enables a robot system to automatically perform object reconstruction tasks. Lacking prior knowledge, next-best-view (NBV) methods are typically used to plan a view sequence, with the goal of covering as many object surface areas as possible in an unknown environment. However, such methods have two problems: (1) they are unable to perform global path planning; and (2) the reconstruction process is inefficient because of time-consuming ray casting and high movement cost. We propose a neural network, SCVP, to pre-learn prior knowledge via set covering (SC) based training so as to achieve one-shot view planning. The SCVP network takes the volumetric occupancy grid as input and directly predicts a small (ideally minimum) number of views that cover all surface areas. Given object 3D models as a priori geometric knowledge, the training dataset is automatically labeled by the set covering optimization method. We propose a global path planning method to reconstruct objects without redundant movement. Comparative experiments on multiple datasets of 3D models show that, under the condition of similar or better surface coverage, the proposed method can outperform state-of-the-art NBV methods in terms of movement cost and inference time. Real-world experiments confirm that the proposed method can achieve faster object reconstruction than other methods.

Many robot tasks, such as grasping and inspection, may require complete 3D models with enough surface details. Fully autonomous environment exploration and 3D reconstruction of unknown objects are challenging for a robot when little or no knowledge about an object is known as a priori. Previous work updated sensor measurements from a greedy ordered set of views into a map with an unchangeable voxel size, leading to the lack of details on the object surface, sampling problems, and no adaption to small objects. We propose a global max-flow-based multi-resolution next-best-view (NBV) method to improve performance on these problems. In particular, it utilizes a max-flow-based global view quality function to obtain optimal NBVs, and a multi-resolution strategy to optimize the reconstruction quality and efficiency. Results of comparative simulation experiments with state-of-the-art (SOTA) methods show that our method achieves a higher voxel coverage under the same resolution. Parametric and ablation studies confirm that the global view function and multi-resolution strategy are effective. Results of real-world experiments show that our method achieves complete reconstruction of small objects with only 6–8 views.

In this letter, we address the problem of behavior-based cooperative navigation of mobile robots usingsafe multi-agent reinforcement learning (MARL). Our work is the first to focus on cooperative navigation without individual reference targets for the robots, using a single target for the formation's centroid. This eliminates the complexities involved in having several path planners to control a team of robots. To ensure safety, our MARL framework uses model predictive control (MPC) to prevent actions that could lead to collisions during training and execution. We demonstrate the effectiveness of our method in simulation and on real robots, achieving safe behavior-based cooperative navigation without using individual reference targets, with zero collisions, and faster target reaching compared to baselines. Finally, we study the impact of MPC safety filters on the learning process, revealing that we achieve faster convergence during training and we show that our approach can be safely deployed on real robots, even during early stages of the training.

Existing view planning systems either adopt an iterative paradigm using next-best views (NBV) or a one-shot pipeline relying on the set-covering view-planning (SCVP) network. However, neither of these methods can concurrently guarantee both high-quality and high-efficiency reconstruction of 3D unknown objects. To tackle this challenge, we introduce a crucial hypothesis: with the availability of more information about the unknown object, the prediction quality of the SCVP network improves. There are two ways to provide extra information: (1) leveraging perception data obtained from NBVs, and (2) training on an expanded dataset of multiview inputs. In this work, we introduce a novel combined pipeline that incorporates a single NBV before activating the proposed multiview-activated (MA-)SCVP network. The MA-SCVP is trained on a multiview dataset generated by our long-tail sampling method, which addresses the issue of unbalanced multiview inputs and enhances the network performance. Extensive simulated experiments substantiate that our system demonstrates a significant surface coverage increase and a substantial 45% reduction in movement cost compared to state-of-the-art systems. Real-world experiments justify the capability of our system for generalization and deployment.

In this study we present an autonomous grasping system that uses a vision-guided hand–eye coordination policy with closed-loop vision-based control to ensure a sufficient task success rate while maintaining acceptable manipulation precision. When facing a diversity of tasks with complex environments, an autonomous robot should use the concept of task precision, including the accuracy of perception and precision of manipulation, as opposed to just the grasping success rate typically used in previous works. Task precision combines the advantages of grasping behaviors observed in humans and a grasping method applied in existing works. A visual servoing approach and a subtask decomposition strategy are proposed here to obtain the desired level of task precision. Our system performs satisfactorily on a tangram puzzle task. The experiments highlight the accuracy of perception, precision of manipulation, and robustness of the system. Moreover, the system is of great significance for improving the adaptability and flexibility of autonomous robots.