Trunk anatomy education is critical in the training of surgeon. Most trunk anatomy education systems use a personal or synthetic anatomical model. It remains difficult to obtain appropriate population anatomy information and create individualized anatomy customization based on a quantity of diverse data. Furthermore, the naked-eye virtual reality modality provides advantages in immersive learning and meta-information transmission. In this study, we present a novel VR torso anatomy teaching environment with naked-eye 3D visualization and free-hand interaction. This work allows for comprehensive modeling, visualization and interaction of torso anatomy for human anatomical variation education. In particular, we derive genuine shape of Chinese male and female torso organs based on the statistical shape model generated from 129 healthy Chinese individuals. The following visual information is given with interaction: (1) Anatomy for human torso organs from CT imaging: geometric modeling, visualization and interaction of male and female torso organs, including skin, muscle, torso skeleton, heart, lungs, liver, spleen and kidneys; (2) Morphological differences of trunk organs from meta-information: motion deformation modeling, visualization and interaction of male and female deformable torso model; (3) Instant personalized customization: real-time interaction and visualization for personalized customization with shape mode coefficients and soft body simulation.
12 April 2022 in Journal of the Optical Society of America A
We present anl omnidirectional 3D autostereoscopic aerial display with continuous parallax. Integral photography (IP) combined with polyhedron-shaped aerial imaging plates (AIPs) is utilized to achieve an extended view angle of 3D aerial images. With optical theoretical analysis and an aerial in situ rotation design, a 3D aerial display with an enlarged viewing angle is realized. In particular, the proposed 3D aerial display can realize any assigned angle within 360 deg. We also optimize the aerial display with artifact image removal and floating image brightness analysis. Experiments are performed to prove the 3D aerial display with full-motion parallax, continuous viewpoints, and multiplayer interaction. The proposed system is an attractive prospect of non-contact interaction and multi-person collaboration.
1 September 2023 in IEEE Journal of Translational Engineering in Health and Medicine
Objective: During minimal invasive surgery (MIS), the laparoscope only provides a single viewpoint to the surgeon, leaving a lack of 3D perception. Many works have been proposed to obtain depth and 3D reconstruction by designing a new optical structure or by depending on the camera pose and image sequences. Most of these works modify the structure of the conventional laparoscopes and cannot provide 3D reconstruction of different magnification views. Methods: In this study, we propose a laparoscopic system based on double liquid lenses, which provide doctors with variable magnification rates, near observation, and real-time monocular 3D reconstruction. Our system composes of an optical structure that can obtain auto magnification change and autofocus without any physically moving element, and a deep learning network based on the Depth from Defocus (DFD) method, trained to suit inconsistent camera intrinsic situations and estimate depth from images of different focal lengths. The optical structure is portable and can be mounted on conventional laparoscopes. The depth estimation network estimates depth in real-time from monocular images of different focal lengths and magnification rates. Results: Experiments show that our system provides a 0.68-1.44x zoom rate and can estimate depth from different magnification rates at 6fps. Monocular 3D reconstruction reaches at least 6mm accuracy. The system also provides a clear view even under 1mm close working distance. Conclusion: Ex-vivo experiments and implementation on clinical images prove that our system provides doctors with a magnified clear view of the lesion, as well as quick monocular depth perception during laparoscopy, which help surgeons get better detection and size diagnosis of the abdomen during laparoscope surgeries.
2 August 2024 in IEEE Transactions on Human-Machine Systems
In traditional clinical practice, doctors often have to deal with 3D information based on 2D-displayed medical images. There is a considerable mismatch between the 2D and 3D dimensions in image interaction during clinical diagnosis, making image manipulation challenging and time-consuming. In this study, we explored 3D sketching for medical images using true 2D-3D interlinked visualization and interaction, presenting a novel AR environment named ARMedicalSketch. It supports image display enhancement preprocessing and 3D interaction tasks for original 3D medical images. Our interaction interface, based on 3D autostereoscopic display technology, provides both floating 3D display and 2D tablet display while enabling glasses-free visualization. We presented a method of 2D-3D interlinked visualization and interaction, employing synchronized projection visualization and a virtual synchronized interactive plane to establish an integrated relationship between 2D and 3D displays. Additionally, we utilized gesture sensors and a 2D touch tablet to capture the user's hand information for convenient interaction. We constructed the prototype and conducted a user study involving 23 students and 2 clinical experts. The controlled study compared our proposed system with a 2D display prototype, showing enhanced efficiency in interacting with medical images while maintaining 2D interaction accuracy, particularly in tasks involving strong 3D spatial correlation. In the future, we aim to further enhance the interaction precision and application scenarios of ARMedicalSketch.
17 November 2019 in Journal of the Society for Information Display