Modern ultralight aircraft in general aviation are equipped with an onboard Pilot Assistance System (PAS) as a companion system, meant to guide the pilot in decision-making, e.g. with plan suggestions, especially in critical situations. For more meaningful guidance, the PAS must possess a continuous understanding of the context, i.e. the pilot’s intention, so that decision-making support is relevant. However, in realistic settings, the pilot’s intention is not communicated manually, but can only be proactively monitored by the PAS. This paper explores the possibility of embedding domain expertise using Hierarchical Task Network (HTN) planning to track the pilot’s intention, by recognising the pilot’s current goal task judging from the pilot’s actions. Furthermore, by leveraging probability theory for state estimation, we derive belief values to be associated with the recognised goal task, inferred from already executed actions which are in turn inferred from in-cockpit observable measurement data. Statistical evaluation using data collected from human-in-the-loop tests shows that our method for tracking the pilot’s intention is reliable enough to provide the PAS with a contextual understanding in real time.

Modern manufacturing processes are characterized by growing product diversities and complexities alike. As a result, the demand for fast and flexible process automation is ever increasing. However, higher individuality and smaller batch sizes hamper the use of standard robotic automation systems, which are well suited for repetitive tasks but struggle in unknown environments. Modern manipulators, such as collaborative industrial robots, provide extended capabilities for flexible automation. In this paper, an adaptive ROS-based end-to-end toolchain for vision-guided robotic process automation is presented. The processing steps comprise several consecutive tasks: CAD-based object registration, pose generation for sensor-guided applications, trajectory generation for the robotic manipulator, the execution of sensor-guided robotic processes, test and the evaluation of the results. The main benefits of the ROS framework are readily applicable tools for digital twin functionalities and established interfaces for various manipulator systems. To prove the validity of this approach, an application example for surface reconstruction was implemented with a 3D vision system. In this example, feature extraction is the basis for viewpoint generation, which, in turn, defines robotic trajectories to perform the inspection task. Two different feature point extraction algorithms using neural networks and Voronoi covariance measures, respectively, are implemented and evaluated to demonstrate the versatility of the proposed toolchain. The results showed that complex geometries can be automatically reconstructed, and they outperformed a standard method used as a reference. Hence, extensions to other vision-controlled applications seem to be feasible.