This paper presents a novel method for online tool recognition in manual assembly processes. The goal was to develop and implement a method that can be integrated with existing Human Action Recognition (HAR) methods in collaborative tasks. We examined the state-of-the-art for progress detection in manual assembly via HAR-based methods, as well as visual tool-recognition approaches. A novel online tool-recognition pipeline for handheld tools is introduced, utilizing a two-stage approach. First, a Region Of Interest (ROI) was extracted by determining the wrist position using skeletal data. Afterward, this ROI was cropped, and the tool located within this ROI was classified. This pipeline enabled several algorithms for object recognition and demonstrated the generalizability of our approach. An extensive training dataset for tool-recognition purposes is presented, which was evaluated with two image-classification approaches. An offline pipeline evaluation was performed with twelve tool classes. Additionally, various online tests were conducted covering different aspects of this vision application, such as two assembly scenarios, unknown instances of known classes, as well as challenging backgrounds. The introduced pipeline was competitive with other approaches regarding prediction accuracy, robustness, diversity, extendability/flexibility, and online capability.

Globally interconnecting machines, processes, and resources driven by exploring and advancing new technologies defined Industry 4.0 (I4.0), resulting in, e.g., Cyber-Physical Production Systems (CPPS). The aircraft industry particularly struggled with transforming production and Maintenance, Repair, and Overhaul (MRO) processes, replacing humans with machines and automating as well as digitalizing significant parts of their value- and non-value-adding activities. However, in the face of current social and environmental challenges, future industries will need to shift from purely technology-driven to value-driven, working sustainably with resources, including human capital. Together, these approaches constitute the idea of Industry 5.0 (I5.0). On the one hand, the aviation industry faces the challenge that even I4.0 concepts and technologies are not yet fully exploited or implemented. On the other hand, due to the specific characteristics of aircraft production and MRO as well as the environmental impact of the product, a tremendous potential arises regarding placing human well-being back into the center of adding value and decreasing environmental footprint while building an industry that is resilient and fortified against disruptions of this era. In line with the I5.0 terminology, in this work, we outline the challenges and opportunities of integrating I5.0 principles into the aircraft production and MRO industries, focusing specifically on the scope of selected use cases.

Obtaining annotated data for proper training of AI image classifiers remains a challenge for successful deployment in industrial settings. As a promising alternative to handcrafted annotations, synthetic training data generation has grown in popularity. However, in most cases the pipelines used to generate this data are not of universal nature and have to be redesigned for different domain applications. This requires a detailed formulation of the domain through a semantic scene grammar. We aim to present such a grammar that is based on domain knowledge for the production-supplying transport of components in intralogistic settings. We present a use-case analysis for the domain of production supplying logistics and derive a scene grammar, which can be used to formulate similar problem statements in the domain for the purpose of data generation. We demonstrate the use of this grammar to feed a scene generation pipeline and obtain training data for an AI based image classifier.

Machine vision solutions can perform within a wide range of applications and are commonly used to verify the operation of production systems. They offer the potential to automatically record assembly states and derive information, but simultaneously require a high effort of planning, configuration and implementation. This generally leads to an iterative, expert based implementation with long process times and sets major barriers for many companies. Furthermore the implementation is task specific and needs to be repeated with every variation of product, environment or process. Therefore a novel concept of a simulation-based process chain for both—configuration and enablement—of machine vision systems is presented in this paper. It combines related work of sensor planning algorithms with new methods of training data generation and detailed task specific analysis for assembly applications.

Robotic bin picking remains a main challenge for the wide enablement of industrial robotic tasks. While AI-enabled picking approaches are encouraging they repeatedly face the problem of data availability. The scope of this paper is to present a method that combines analytical grasp research with the field of synthetic data creation to generate individual training data for use-cases in intralogistics transportation scenarios. Special attention is given to systematic grasp finding for new objects and unknown geometries in transportation bins and to match the generated data to a real two-finger parallel gripper. The presented approach includes a grasping simulation in Pybullet to investigate the general tangibility of objects under uncertainty and combines these findings with a previously reported virtual scene generator in Blender, which generates AI-images of fully packed transport boxes, including depth maps and necessary annotations. This paper, therefore, contributes a synthesizing and cross-topic approach that combines different facets of bin-picking research such as geometric analysis, determination of tangibility of objects, grasping under uncertainty, finding grasps in dynamic and restricted bin-environments, and automation of synthetic data generation. The approach is utilized to generate synthetic grasp training data and to train a grasp-generating convolutional neural network (GG-CNN) and demonstrated on real-world objects.