Background: Gait and balance impairments are common in neurological diseases, including stroke, and negatively affect patients’ quality of life. Improving balance and gait are among the main goals of rehabilitation. Rehabilitation is mainly performed in clinics, which lack context specificity; therefore, training in the patient’s home environment is preferable. In the last decade, developed rehabilitation technologies such as virtual reality and augmented reality (AR) have enabled gait and balance training outside clinics. Here, we propose a new method for gait rehabilitation in persons who have had a stroke in which mobile AR technology and a sensor-based motion capture system are combined to provide fine-grained feedback on gait performance in real time. Objective: The aims of this study were (1) to investigate manipulation of the gait pattern of persons who have had a stroke based on virtual augmentation during overground walking compared to walking without AR performance feedback and (2) to investigate the usability of the AR system. Methods: We developed the ARISE (Augmented Reality for gait Impairments after StrokE) system, in which we combined a development version of HoloLens 2 smart glasses (Microsoft Corporation) with a sensor-based motion capture system. One patient with chronic minor gait impairment poststroke completed clinical gait assessments and an AR parkour course with patient-centered performance gait feedback. The movement kinematics during gait as well as the usability and safety of the system were evaluated. Results: The patient changed his gait pattern during AR parkour compared to the pattern observed during the clinical gait assessments. He recognized the virtual objects and ranked the usability of the ARISE system as excellent. In addition, the patient stated that the system would complement his standard gait therapy. Except for the symptom of exhilaration, no adverse events occurred. Conclusions: This project provided the first evidence of gait adaptation during overground walking based on real-time feedback through visual and auditory augmentation. The system has potential to provide gait and balance rehabilitation outside the clinic. This initial investigation of AR rehabilitation may aid the development and investigation of new gait and balance therapies.

Three decades after the first set of work on Medical Augmented Reality (MAR) was presented to the international community, and ten years after the deployment of the first MAR solutions into operating rooms, its exact definition, basic components, systematic design, and validation still lack a detailed discussion. This paper defines the basic components of any Augmented Reality (AR) solution and extends them to exemplary Medical Augmented Reality Systems (MARS). We use some of the original MARS applications developed at the Chair for Computer Aided Medical Procedures and deployed into medical schools for teaching anatomy and into operating rooms for telemedicine and surgical guidance throughout the last decades to identify the corresponding basic components. In this regard, the paper is not discussing all past or existing solutions but only aims at defining the principle components and discussing the particular domain modeling for MAR and its design-development-validation process, and providing exemplary cases through the past in-house developments of such solutions.

Medical teleconsultation was among the initial use cases for early telepresence research projects since medical treatment often requires timely intervention by highly specialized experts. When remote medical experts support interventions, a holistic view of the surgical site can increase situation awareness and improve team communication. A possible solution is the concept of immersive telepresence, where remote users virtually join the operating theater that is transmitted based on a real-time reconstruction of the local site. Enabled by the availability of RGB-D sensors and sufficient computing capability, it becomes possible to capture such a site in real time using multiple stationary sensors. The 3D reconstruction and simplification of textured surface meshes from the point clouds of a dynamic scene in real time is challenging and becomes infeasible for increasing capture volumes. This work presents a tightly integrated, stateless 3D reconstruction pipeline for dynamic, room-scale environments that generates simplified surface meshes from multiple RGB-D sensors in real time. Our algorithm operates directly on the fused, voxelized point cloud instead of populating signed-distance volumes per frame and using a marching cube variant for surface reconstruction. We extend the formulation of the dual contouring algorithm to work for point cloud data stored in an octree and interleave a vertex-clustering-based simplification before extracting the surface geometry. Our 3D reconstruction pipeline can perform a live reconstruction of six incoming depth videos at their native frame rate of 30 frames per second, enabling the reconstruction of smooth movement. Arbitrarily complex scene changes are possible since we do not store persistent information between frames. In terms of mesh quality and hole filling, our method falls between the direct mesh reconstruction and expensive global fitting of implicit functions.

Our world is full of cameras, whether they are installed in the environment or integrated into mobile devices such as mobile phones or head-mounted displays. Displaying external camera views in our egocentric view with a picture-in-picture approach allows us to understand their view; however, it would not allow us to correlate their viewpoint with our perceived reality. We introduce Projective Bisector Mirrors for visualizing a camera view comprehensibly in the egocentric view of an observer with the metaphor of a virtual mirror. Our concept projects the image of a capturing camera onto the bisecting plane between the capture and the observer camera. We present extensive mathematical descriptions of this novel paradigm for multi-view visualization, discuss the effects of tracking errors and provide concrete implementation for multiple exemplary use-cases.

A core principle of modern health care is the compliance of hygienic and aseptic techniques in areas that are sensitive to contamination through bacteria, dust, aerosols, and fallout, primarily in operating theatres or around patients with contagious diseases. Keeping track of potentially contaminated surfaces in an environment is a major concern, especially when protecting from COVID-19. This work proposes a novel concept in using 3D sensing technology to track human movement within an indoor area and identifying high-risk contaminated surfaces in real-time. It combines recent Augmented Reality display technology, which allows keeping track of decontaminated surfaces during the cleaning process using an interactive visualization method. The proposed concept of Clean- AR is implemented in a clinical environment used for observation in COVID-19 scenarios. We discuss key challenges and outline further research direction in effectively reducing the risk of contamination using the proposed concept.