In this paper we present Drone Brush, a prototype mixed reality interface for immersive planning of drone paths for tasks such as collaborative photogrammetry and inspection. This interface employs Microsoft's HoloLens 2 to allow users to draw paths for drone navigation in 3D using hand gestures. Users can place waypoints with a simple pinch gesture, and similarly, delete and move existing waypoints. To validate paths, we leverage the HoloLens spatial map to check for potential collisions ahead of time, greatly reducing the likelihood of a collision during drone navigation. Paths are simplified and cleaned up using density-based clustering to prevent complex or redundant drone movement. In this Late-Breaking Report, we present the design and implementation of our system that integrates mixed reality, natural hand gestures, and drone path planning, which we plan to evaluate in a user study in the near future.
This work aims to leverage medical augmented reality (AR) technology to counter the shortage of medical experts in low-resource environments. We present a complete and cross-platform proof-of-concept AR system that enables remote users to teach and train medical procedures without expensive medical equipment or external sensors. By seeing the 3D viewpoint and head movements of the teacher, the student can follow the teacher’s actions on the real patient. Alternatively, it is possible to stream the 3D view of the patient from the student to the teacher, allowing the teacher to guide the student during the remote session. A pilot study of our system shows that it is easy to transfer detailed instructions through this remote teaching system and that the interface is easily accessible and intuitive for users. We provide a performant pipeline that synchronizes, compresses, and streams sensor data through parallel efficiency.