Manual assembly tasks can be difficult and tedious without assistance. Here, augmented reality (AR) can help to ease the task load and lower the time and error rates by interactive in-situ instructions. Such approaches are often implemented by video or optical see-through AR. In order to test a system that is as low-threshold as possible and easy to implement, we developed a prototype with a mirror display supplemented by a RGB-D camera and evaluated it against an well-established AR HMD (HoloLens 2). For this purpose, in our study 10 participants placed 3D printed bricks via both technologies. The quantitative and qualitative analysis revealed that the fulfillment of this task with a professional, established product AR HMD yet remains unmatched in terms of the time required, error rate, usability and task load due to the prototype status and the remaining technical shortcomings. However, the mirror display setup prototype met with interest from the participants as a novel, but unfamiliar and thus more difficult to use way of interaction. Furthermore, we report implementation challenges and advises. The empirical insights of our prototype and the first-time comparison to an established AR HMD aims to foster refining future work with half-silvered AR mirrors as an little researched field and many different fields of application.

This paper presents a new approach for providing contextual information in real-world environments. Our approach is consciously designed to be low-threshold; by using mirrors as augmented reality surfaces, no devices such as AR glasses or smartphones have to be worn or held by the user. It enables technical and non-technical objects in the environment to be visually highlighted and thus subtly draw the attention of people passing by. The presented technology enables the provision of information that can be viewed in more detail by the user if required by slowing down their movement. Users can decide whether this is relevant to them or not. A prototype system was implemented and evaluated through a user study. The results show a high level of acceptance and intuitive usability of the system, with participants being able to reliably perceive and process the information displayed. The technology thus offers promising potential for the unobtrusive and context-sensitive provision of information in various application areas. The paper discusses limitations of the system and outlines future research directions to further optimize the technology and extend its applicability.

For people with visual impairments, navigating and orienting themselves in public spaces is a daily challenge. Although a wide variety of electronic assistive devices already exist, in the past these provided only fairly abstract guidance. With large language models (LLMs), natural-language descriptions of image content are now possible. While there are already approaches that capture images via smart glasses and forward the image content to external large language models to generate image descriptions, these methods raise concerns regarding data privacy and availability. This paper therefore presents a novel approach in which data is processed exclusively locally on the user’s device. To do this, the user captures a camera image via smart glasses, which is forwarded to a connected smartphone. Inference takes place directly on the smartphone using a reduced, local LLM that has been adapted through LoRA fine-tuning using data relevant to people with visual impairments. The generated image description is then played through a speaker in the temple of the smart glasses. For the task-specific fine-tuning, a comprehensive survey of visually impaired people was conducted, and professional mobility trainers were consulted. The study demonstrated the feasibility of fully local processing using smart glasses in combination with a smartphone. The evaluation showed that fine-tuning the small, local model yields improved image descriptions compared to the non-adapted model, some of which outperform a significantly larger external language model.