Robotic manipulation in cluttered environments is one of the challenges roboticists are currently facing. When the objects to handle are delicate fresh fruits, grasping is even more challenging. Detecting and localizing fruits with the accuracy necessary to grasp them is very difficult due to the large variability in the aspect and dimensions of each item. This paper proposes a solution that exploits a state-of-the-art neural network and a novel enhanced 6D pose estimation method that integrates the depth map with the neural network output. Even with an accurate localization, grasping fruits with a suitable force to avoid slippage and damage at the same time is another challenge. This work solves this issue by resorting to a grasp controller based on tactile sensing. Depending on the specific application scenario, grasping a fruit might be impossible without colliding with other objects or other fruits. Therefore, a non-prehensile manipulation action is here proposed to push items hindering the grasp of a detected fruit. The pushing from an initial location to a target one is performed by a model predictive controller taking into account the unavoidable delay in the perception and computing pipeline of the robotic system. Experiments with real fresh fruits demonstrate that the overall proposed approach allows a robot to successfully grasp apples in various situations.

This paper presents an experimental comparison between two existing methods representative of two categories of 6D pose estimation algorithms nowadays commonly used in the robotics community. The first category includes purely deep learning methods, while the second one includes hybrid approaches combining learning pipelines and geometric reasoning. The hybrid method considered in this paper is a pipeline of an instance-level deep neural network based on RGB data only and a geometric pose refinement algorithm based on the availability of the depth map and the CAD model of the target object. Such a method can handle objects whose dimensions differ from those of the CAD. The pure learning method considered in this comparison is DenseFusion, a consolidated state-of-the-art pose estimation algorithm selected because it uses the same input data, namely, RGB image and depth map. The comparison is carried out by testing the success rate of fresh food pick-and-place operations. The fruit-picking scenario has been selected for the comparison because it is challenging due to the high variability of object instances in appearance and dimensions. The experiments carried out with apples and limes show that the hybrid method outperforms the pure learning one in terms of accuracy, thus allowing the pick-and-place operation of fruits with a higher success rate. An extensive discussion is also presented to help the robotics community select the category of 6D pose estimation algorithms most suitable to the specific application.

Robotic manipulation of objects in cluttered dynamic scenes is challenging for a twofold reason. Object detection and localization are complex due to partial occlusions and high variability in the object classes and manipulation in tight spaces is difficult due to potential collisions. The present letter focuses on the low-level control of the non-prehensile pushing action aimed at moving planar objects of generic shape along a given path with an assigned time law. Based on the continuous and nonlinear dynamics of the system, we propose a nonlinear model predictive controller (NMPC), which avoids the need for linearization and, thus, the hybrid dynamics arising from it. An extensive comparison with a state-of-the-art linear MPC demonstrates that the NMPC can successfully react to more general disturbances, outperforming the linear one. Experimental results confirm the effectiveness of the method in a task where a robot is required to grasp fruits in a container with other obstructing objects (shown in the attached video).

This letter proposes a control framework for dual-arm robotic manipulation of a common workpiece, with the objective of tracking a reference trajectory for the object while minimizing the internal stress on it using pose/force controllers. The visual-tactile observations provided by the individual robots are fused, using a discrete-time Extended Kalman Filter designed on the $SE(3)$ manifold, to online calibrate their relative pose and estimate both the object pose and the internal forces at the same time. This algorithm ensures consistency of the estimated quantities evolving in $SE(3)$ without any simplifying assumptions. The cooperative robots exhibit a high degree of redundancy owing to the suitable modulation of grasp forces enabling dexterous in-hand manipulation actions such as pivoting or sliding. The redundant degrees of freedom can easily be handled at kinematic level, owing to the soft contact suitably modeled with an elastic displacement in $SE(3)$ to keep kinetostatic consistency with the robot motions taking place in this manifold. Experimental results on a team of two heterogeneous robots demonstrate the effectiveness of the proposed method.