Flexible pick-and-place is a fundamental yet challenging task within robotics, in particular due to the need of an object model for a simple target pose definition. In this work, the robot instead learns to pick-and-place objects using planar manipulation according to a single, demonstrated goal state. Our primary contribution lies within combining robot learning of primitives, commonly estimated by fully-convolutional neural networks, with one-shot imitation learning. Therefore, we define the place reward as a contrastive loss between real-world measurements and a task-specific noise distribution. Furthermore, we design our system to learn in a self-supervised manner, enabling real-world experiments with up to 25 000 pick-and-place actions. Then, our robot is able to place trained objects with an average placement error of (2.7 $\pm$ 0.2) mm and (2.6 $\pm$ 0.8) $^\circ$ . As our approach does not require an object model, the robot is able to generalize to unknown objects while keeping a precision of (5.9 $\pm$ 1.1) mm and (4.1 $\pm$ 1.2) $^\circ$ . We further show a range of emerging behaviors: The robot naturally learns to select the correct object in the presence of multiple object types, precisely inserts objects within a peg game, picks screws out of dense clutter, and infers multiple pick-and-place actions from a single goal state.

Robotic grasping in cluttered environments is often infeasible due to obstacles preventing possible grasps. Then, pre-grasping manipulation like shifting or pushing an object becomes necessary. We developed an algorithm that can learn, in addition to grasping, to shift objects in such a way that their grasp probability increases. Our research contribution is threefold: First, we present an algorithm for learning the optimal pose of manipulation primitives like clamping or shifting. Second, we learn non-prehensible actions that explicitly increase the grasping probability. Making one skill (shifting) directly dependent on another (grasping) removes the need of sparse rewards, leading to more data-efficient learning. Third, we apply a real-world solution to the industrial task of bin picking, resulting in the ability to empty bins completely. The system is trained in a self-supervised manner with around 25 000 grasp and 2500 shift actions. Our robot is able to grasp and file objects with 274±3 picks per hour. Furthermore, we demonstrate the system’s ability to generalize to novel objects.

Task-oriented grasping models aim to predict a suitable grasp pose on an object to fulfill a task. These systems have limited generalization capabilities to new tasks, but have shown the ability to generalize to novel objects by recognizing affordances. This object generalization comes at the cost of being unable to recognize the object category being grasped, which could lead to unpredictable or risky behaviors. To overcome these generalization limitations, we contribute a novel system for task-oriented grasping called the One-Shot Task-Oriented Grasping (OS-TOG) framework. OS-TOG comprises four interchangeable neural networks that interact through dependable reasoning components, resulting in a single system that predicts multiple grasp candidates for a specific object and task from multi-object scenes. Embedded one-shot learning models leverage references within a database for OS-TOG to generalize to novel objects and tasks more efficiently than existing alternatives. Additionally, the paper presents suitable candidates for the framework's neural components, covering essential adjustments for their integration and evaluative comparisons to state-of-the-art. In physical experiments with novel objects, OS-TOG recognizes 69.4% of detected objects correctly and predicts suitable task-oriented grasps with 82.3% accuracy, having a physical grasp success rate of 82.3%.

The accurate tracking of patients during a surgery is an essential requirement of computer assisted surgery. Many tracking systems are based on permanently installed infrared camera systems to detect reflective spheres. These tracking concepts need a certain amount of installation effort and are associated with high investments. An alternative are planar fiducial markers, which can be tracked only through RGB data and can therefore be used with different camera systems. The objective of this work is to introduce a set of similarity metrics to compare fiducial markers for pose estimation. We propose eight different similarity metrics to unify the process of evaluation and comparison of marker systems. These are the size and outer margin of the marker, the distance to the camera, the pose estimation accuracy, the runtime of the algorithm, the robustness against external influences, the affection to the sensor system and the number of used markers. We also describe the methodology for evaluating these metrics. We then apply these metrics to compare the ArUco and AprilTag open source marker systems. Our tests conclude that the optical tracking of open source fiducial markers is possible at submillimeter range at distances up to one meter. In addition, the tracking result can be greatly improved by using multiple markers. Accuracy is increased and fluctuations are minimized. The similarity metrics presented by us are suitable for evaluating and comparing marker systems in detail. This can serve as a basis for selecting a suitable system for a specific medical procedure.

We present an end-to-end trainable multi-task model that locates and retrieves target objects from multi-object scenes. The model is an extension of the Siamese Mask R-CNN, which combines the components of Siamese Neural Networks (SNNs) and Mask R-CNN for performing one-shot instance segmentation. The proposed network, called Grasping Siamese Mask R-CNN (GSMR-CNN), extends Siamese Mask R-CNN by adding an additional branch for grasp detection in parallel to the previous object detection head branches. This allows our model to identify a target object with a suitable grasp simultaneously, as opposed to other approaches that require the training of separate models to achieve the same task. The inherent SNN properties enable the proposed model to generalize and recognize new object categories that were not present during training, which is beyond the capabilities of standard object detectors. Moreover, an end-to-end solution uses shared features entailing less model parameters. The model achieves grasp accuracy scores of 92.1 % and 90.4% on the OCID grasp dataset on image-wise and object-wise splits. Physical experiments show that the model achieves a grasp success rate of 76.4 % when correctly identifying the object. Code and models are available at https://github.com/valerijah/grasping_siamese_mask_rcnn

This paper presents a description logic based system to store and retrieve knowledge used in models for autonomous probabilistic decision making by multimodal service robots. These models are mainly generated by observation and analysis of humans performing tasks, the programming by demonstration methodology. As formal model representation, partially observable Markov decision processes (POMDPs) are utilized as they are a well understood formal framework for decision making considering real world uncertainty in both perception and execution. The approach presented here deals with aspects of organizing knowledge which cannot be retrieved from user demonstrations or which is valid beyond a single task. It is shown how use it in the process of model generation on a real service robot.