4 December 2024 in IEEE Robotics and Automation Letters
The accurate reconstruction of dynamic scenes with neural radiance fields is significantly dependent on the estimation of camera poses. Widely used structure-from-motion pipelines encounter difficulties in accurately tracking the camera trajectory when faced with separate dynamics of the scene content and the camera movement. To address this challenge, we propose Dyna mic Mo tion-Aware Fast and Robust Camera Localization for Dynamic N eural Radiance Fields (DynaMoN). DynaMoN utilizes semantic segmentation and generic motion masks to handle dynamic content for initial camera pose estimation and statics-focused ray sampling for fast and accurate novel-view synthesis. Our novel iterative learning scheme switches between training the NeRF and updating the pose parameters for an improved reconstruction and trajectory estimation quality. The proposed pipeline shows significant acceleration of the training process. We extensively evaluate our approach on two real-world dynamic datasets, the TUM RGB-D dataset and the BONN RGB-D Dynamic dataset. DynaMoN improves over the state-of-the-art both in terms of reconstruction quality and trajectory accuracy. We plan to make our code public to enhance research in this area. Code available: https://hannahhaensen.github.io/DynaMoN/ .
27 November 2024 in Research Square
Background Mixed Reality (MR) enables intuitive and tangible visualization of patient data in interventional environments without further exposure to radiation, while most state-of-the-art navigation systems lack proper immersive visualization and rely on the use of markers as well as additional intrasurgical imaging for digital-physical registration. Methods We develop and validate a modular and immersive system combining state of the art mixed reality hardware, high-fidelity tracking/scanning technology, and automated surface registration algorithms to generate an accurate MR holographic projection of surgical tools and patient anatomy. Results The system exhibits reliable real-time performance with latencies below 50 ms. The accuracy of the holographic projection was measured for registration and MR precision, revealing a combined average deviation of 1.73 mm on a patient model. Conclusion Overall system accuracy provides a strong foundation for immersive surgical or interventional guidance. Ongoing studies aim to investigate clinical value and usability in real-world scenarios.