With innovations in the field of gaze and eye tracking, a new concentration of research in the area of gaze-tracked systems and user interfaces has formed in the field of Extended Reality (XR). Eye trackers are being used to explore novel forms of spatial human–computer interaction, to understand human attention and behavior, and to test expectations and human responses. In this article, we review gaze interaction and eye tracking research related to XR that has been published since 1985, which includes a total of 215 publications. We outline efforts to apply eye gaze for direct interaction with virtual content and design of attentive interfaces that adapt the presented content based on eye gaze behavior and discuss how eye gaze has been utilized to improve collaboration in XR. We outline trends and novel directions and discuss representative high-impact papers in detail.

Augmented reality (AR) and robotic-assisted surgery (RAS) are both rapidly evolving technologies in recent years. RAS systems, such as the da Vinci® Surgical System, aim to improve surgical precision and dexterity, as well as access to minimally-invasive procedures, while AR provides an advanced interface to enhance user perception. Combining the features of both, AR-integrated RAS has become an appealing concept with increased interest among the academic community. In this paper, we review the existing literature about AR-integrated RAS. We discuss the hardware components, application paradigms and clinical relevance of the literature. The concept of AR-integrated RAS has been shown to be feasible for various procedures. Encouraging preliminary results include reduced sight diversion and improved situation awareness. Special techniques, e.g., activation-on-demand, are taken into consideration to address visual clutter of the AR interface and ensure that the system is fail-safe. Although AR-integrated RAS is not yet mature, we believe that if the current trend of development continues, it will soon demonstrate its clinical value.

In robot‐assisted laparoscopic surgery, the first assistant (FA) is responsible for tasks such as robot docking, passing necessary materials, manipulating hand‐held instruments, and helping with trocar planning and placement. The performance of the FA is critical for the outcome of the surgery. The authors introduce ARssist , an augmented reality application based on an optical see‐through head‐mounted display, to help the FA perform these tasks. ARssist offers (i) real‐time three‐dimensional rendering of the robotic instruments, hand‐held instruments, and endoscope based on a hybrid tracking scheme and (ii) real‐time stereo endoscopy that is configurable to suit the FA's hand–eye coordination when operating based on endoscopy feedback. ARssist has the potential to help the FA perform his/her task more efficiently, and hence improve the outcome of robot‐assisted laparoscopic surgeries.

Stereo visual servo based surgical instruments tracking enables quicker adjustment of the view and depth of the robotic stereo flexible endoscope (RSFE). However, there are two main drawbacks: 1) when the lesion is covered by organs, the observation angles, from which the endoscope observes it, need to be adjusted independently. But this cannot be achieved by only tracking surgical instruments; 2) the field of view is restricted by the shared area of the two cameras, which limits the tracking area. Therefore, a 6-DOF augmented reality visualizing robotic stereo flexible endoscope (ARSFE), which integrates a pair of HoloLens with the RSFE, is proposed. The Hololens is used for tracking the surgeon’s head and providing head-mounted display. In the ARSFE, the endoscope can be controlled in two modes: 1) The view and depth of the endoscope can be controlled by tracking surgical instruments; 2) The observation angles can be adjusted independently by tracking the surgeon’s head. Modes switching is realized via the foot pedals. Besides, a hybrid control method for tracking instruments is proposed, which significantly enlarges the control field for instruments tracking. In the experiment section, the performance of the proposed hybrid visual servoing control and the ARSFE is verified.

Head-mounted loupes can increase the user’s visual acuity to observe the details of an object. On the other hand, optical see-through head-mounted displays (OST-HMD) are able to provide virtual augmentations registered with real objects. In this article, we propose AR-Loupe, combining the advantages of loupes and OST-HMDs, to offer augmented reality in the user’s magnified field-of-vision. Specifically, AR-Loupe integrates a commercial OST-HMD, Magic Leap One, and binocular Galilean magnifying loupes, with customized 3D-printed attachments. We model the combination of user’s eye, screen of OST-HMD, and the optical loupe as a pinhole camera. The calibration of AR-Loupe involves interactive view segmentation and an adapted version of stereo single point active alignment method (Stereo-SPAAM). We conducted a two-phase multi-user study to evaluate AR-Loupe. The users were able to achieve sub-millimeter accuracy ( $0.82\;\mathrm{mm}$ ) on average, which is significantly ( $p < 0.001$ ) smaller compared to normal AR guidance ( $1.49\;\mathrm{mm}$ ). The mean calibration time was $268.46\;s$ . With the increased size of real objects through optical magnification and the registered augmentation, AR-Loupe can aid users in high-precision tasks with better visual acuity and higher accuracy.