Eye-tracking plays a crucial role in the development of virtual reality devices, neuroscience research, and psychology. Despite its significance in numerous applications, achieving an accurate, robust, and fast eye-tracking solution remains a considerable challenge for current state-of-the-art methods. While existing reflection-based techniques (e.g., “glint tracking") are considered to be very accurate, their performance is limited by their reliance on sparse 3D surface data acquired solely from the cornea surface. In this paper, we rethink the way how specular reflections can be used for eye tracking: We propose a method for accurate and fast evaluation of the gaze direction that exploits teachings from single-shot phase-measuring-deflectometry. In contrast to state-of-the-art reflection-based methods, our method acquires dense 3D surface information of both cornea and sclera within only one single camera frame (single-shot). For a typical measurement, we acquire >3000× more surface reflection points ("glints”) than conventional methods. We show the feasibility of our approach with experimentally evaluated gaze errors on a realistic model eye below only 0.13°. Moreover, we demonstrate quantitative measurements on real human eyes in vivo, reaching accuracy values between only 0.46° and 0.97°.

Eye tracking is an important tool in many scientific and commercial domains. State-of-the-art eye tracking methods are either reflection-based and track reflections of sparse point light sources, or image-based and exploit 2D features of the acquired eye image. In this work, we attempt to significantly improve reflection-based methods by utilizing pixel-dense deflectometric surface measurements in combination with optimization-based inverse rendering algorithms. Utilizing the known geometry of our deflectometric setup, we develop a differentiable rendering pipeline based on PyTorch3D that simulates a virtual eye under screen illumination. Eventually, we exploit the image-screen-correspondence information from the captured measurements to find the eye's rotation , translation , and shape parameters with our renderer via gradient descent. We demonstrate real-world experiments with evaluated mean relative gaze errors below 0.45 ${}^{\circ }$ at a precision better than 0.11 ${}^{\circ }$ . Moreover, we show an improvement of 6X over a representative reflection-based state-of-the-art method in simulation. In addition, we demonstrate a special variant of our method that does not require a specific pattern and can work with arbitrary image or video content from every screen (e.g., in a VR headset).

We present an easy and flexible calibration approach for small field of view Deflectometry systems to be used in VR headsets. Our method replaces all required calibration targets and mirrors with a small (phone) screen to perform intrinsic and extrinsic camera calibration as well as extrinsic calibration of the whole setup.

Event-based structured light systems have recently emerged as an exciting alternative to frame-based triangulation for 3D measurement of diffuse surfaces, offering high dynamic range and fast capture speed, but at the cost of reduced data quality. Existing event-based and frame-based 3D imaging systems are typically tailored to a single surface reflectance type, diffuse or specular, and therefore struggle with mixed reflectance scenes. In this work, we present an event-based structured light system for fast and accurate 3D imaging of mixed reflectance scenes. Using epipolar constraints on the captured events, we decompose reflections into diffuse, two-bounce specular, and other multi-bounce reflections. Diffuse surfaces are reconstructed via triangulation and repurposed as a virtual screen for deflectometry to evaluate specular surfaces, requiring only a scanning laser and an event camera. Our system achieves motion-robust 3D reconstructions at < 600 μm depth accuracy and introduces a fast diffuse-only capture mode operating at 250 Hz.

We propose a 3D mesh-based integration method for iterative surface reconstruction in phase measuring deflectometry, enabling accurate reconstruction of specular surfaces from non-uniformly sampled data.

We introduce a new technique and experiments for deflectometric eye tracking on real human eyes in vivo. Our technique utilizes the teachings of our previous eye-tracking research, combining the captured deflectometric information in a novel fashion.