This work focuses on the operation of picking an object on a table with a mobile manipulator. We use deep reinforcement learning (DRL) to learn a positioning policy for the robot’s base by considering the reachability constraints of the arm. This work extends our first proof-of-concept with the ultimate goal of validating the method on a real robot. Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is used to model the base controller, and is optimised using the feedback from the MoveIt! based arm planner. The idea is to encourage the base controller to position itself in areas where the arm reaches the object. Following a simulation-to-reality approach, first we create a realistic simulation of the robotic environment in Unity, and integrate it in Robot Operating System (ROS). The drivers for both the base and the arm are also implemented. The DRL-based agent is trained in simulation and, both the robot and target poses are randomised to make the learnt base controller robust to uncertainties. We propose a task-specific setup for TD3, which includes state/action spaces, reward function and neural architectures. We compare the proposed method with the baseline work and show that the combination of TD3 and the proposed setup leads to a $$11\%$$11% higher success rate than with the baseline, with an overall success rate of $$97\%$$97% . Finally, the learnt agent is deployed and validated in the real robotic system where we obtain a promising success rate of $$75\%$$75% .

In the context of industrial production, a worker that wants to program a robot using the hand-guidance technique needs that the robot is available to be programmed and not in operation. This means that production with that robot is stopped during that time. A way around this constraint is to perform the same manual guidance steps on a holographic representation of the digital twin of the robot, using augmented reality technologies. However, this presents the limitation of a lack of tangibility of the visual holograms that the user tries to grab. We present an interface in which some of the tangibility is provided through ultrasound-based mid-air haptics actuation. We report a user study that evaluates the impact that the presence of such haptic feedback may have on a pick-and-place task of the wrist of a holographic robot arm which we found to be beneficial.

Computer vision based visual inspection systems are gaining enormous importance for manufacturing quality control in recent years due to the advent of Convolutional neural networks (CNN) and transformer-based (vision) models. CNN based models attempt to extract global features by gradually increasing the receptive field, while long-range dependencies are ignored. Therefore, CNN recognizes objects based on the texture instead of the shape. Transformer models, on the other hand, enable modeling long range dependencies using self-attention mechanism. But learning ability of spatial information inside each patch is limited, which means it can disregard a significant spatial local pattern, such as texture. In this work, we propose to combine transformer-based and CNN-based models to take advantage of the strengths of both methods. To meet inference time constraints of real time defect classification tasks, we exploit knowledge distillation method (KD) using softened logits of ensemble model as supervision to train a lightweight CNN model (Resnet18). The study showed that the proposed vision transformer-based KD approach overcome the requirements of limited computational resources and can be deployed on low-power and resource limited devices. The experimental results also showed that proposed framework outperforms in terms of mean accuracy on the test datasets compared to stand-alone CNN methods.

The MASTER project introduces an open Extended Reality (XR) platform designed to enhance human-robot collaboration and train workers in robotics within manufacturing settings. It includes modules for creating safe workspaces, intuitive robot programming, and user-friendly human-robot interactions (HRI), including eye-tracking technologies. The development of the platform is supported by two open calls targeting technical SMEs and educational institutes to enhance and test its functionalities. By employing the learning-by-doing methodology and integrating effective teaching principles, the MASTER platform aims to provide a comprehensive learning environment, preparing students and professionals for the complexities of flexible and collaborative manufacturing settings.

Manual guidance of a collaborative robot arm (i.e., moving the robot arm by hand) is a technique to program the robot without coding and in the context of the job. However, it requires the robot to be available and not in operation. As a workaround, we propose performing manual guidance on a hologram of that same robot, the feasibility of which we are investigating. A potential limitation of this approach is the lack of tangibility of the hologram, for which we are investigating the contribution that mid-air haptics (MAH) can make. Early results suggest beneficial effects from tactile feedback on AR pick-and-place.