This paper introduces the novel Augmented Reality (AR) assistance system AVIKOM, a joint endeavour of three research groups together with four small and medium-sized enterprises (SME) as well as network partners and a diaconal institution. In particular, we investigate how AR-enabled assistance systems can be tailored to individual requirements of workers with diverse cognitive and physical capabilities for today's real-world industrial applications in the areas of (manual) assembly, logistics and operation of industrial machinery. We combine best practices from the domains of artificial intelligence, machine learning, user experience engineering, ethics research, and cognitive science with state-of-the-art insights for multi-modal system development to create a cognitive action assistance system that recognizes and adapts to individual users in various situational contexts, such as picking and training. Proven work and organizational psychology methods and worth-related evaluations will accompany the system introduction into working environments. Using user- and worth-centred system design and change management strategies (e.g. information and participation) right from the beginning of such a technological development facilitates proper involvement of future users in the development process. This can lead to better congruence of technology features with workers' requirements and positively shape future users' attitudes towards the system.

In this paper we introduce two new methods for real-time sonification of head movements and head gestures. Head gestures such as nodding or shaking the head are important non-verbal back-channelling signals which facilitate coordination and alignment of communicating interaction partners. Visually impaired persons cannot interpret such non-verbal signals, same as people in mediated communication (e.g. on the phone), or cooperating users whose visual attention is focused elsewhere. We introduce our approach to tackle these issues, our sensing setup and two different sonification methods. A first preliminary study on the recognition of signals shows that subjects understand the gesture type even without prior explanation and can estimate gesture intensity and frequency with no or little training.

Contemporary assistance systems support a broad variety of tasks. When they provide information or instruction, the way they do it has an implicit and often not directly graspable impact on the user. System design often forces static roles onto the user, which can have negative side effects when system errors occur or unique and previously unknown situations need to be tackled. We propose an adjustable augmented reality-based assistance infrastructure that adapts to the user’s individual cognitive task proficiency and dynamically reduces its active intervention in a subtle, not consciously noticeable way over time to spare attentional resources and facilitate independent task execution. We also introduce multi-modal mechanisms to provide context-sensitive assistance and argue why system architectures that provide explainability of concealed automated processes can improve user trust and acceptance.

Multimodal research in human interaction has to consider a variety of factors, ranging from local short-time phenomena to complex interaction patterns. As of today, no single discipline engaged in communication research offers the methods and tools to investigate the full complexity continuum in a time-efficient way. A synthesis of qualitative and quantitative analysis is required to merge insights about micro-sequential structures with big data patterns. Using the example of a co-present dyadic negotiation analysis to combine methods offered by Conversation Analysis and Data Mining, we show how such a partnership can benefit each discipline and lead to insights as well as new hypotheses evaluation opportunities.