10 April 2025 in Medical Image Analysis
Medical imaging maps tissue characteristics into image intensity values, enhancing human perception. However, comprehending this data, especially in high-stakes scenarios such as surgery, is prone to errors. Additionally, current multimodal methods do not fully leverage this valuable data in their design. We introduce "From Tissue to Sound," a new paradigm for medical sonic interaction design. This paradigm establishes a comprehensive framework for mapping tissue characteristics to auditory displays, providing dynamic and intuitive access to medical images that complement visual data, thereby enhancing multimodal perception. "From Tissue to Sound" provides an advanced and adaptable framework for the interactive sonification of multimodal medical imaging data. This framework employs a physics-based sound model composed of a network of multiple oscillators, whose mechanical properties-such as friction and stiffness-are defined by tissue characteristics extracted from imaging data. This approach enables the representation of anatomical structures and the creation of unique acoustic profiles in response to excitations of the sound model. This method allows users to explore data at a fundamental level, identifying tissue characteristics ranging from rigid to soft, dense to sparse, and structured to scattered. It facilitates intuitive discovery of both general and detailed patterns with minimal preprocessing. Unlike conventional methods that transform low-dimensional data into global sound features through a parametric approach, this method utilizes model-based unsupervised mapping between data and an anatomical sound model, enabling high-dimensional data processing. The versatility of this method is demonstrated through feasibility experiments confirming the generation of perceptually discernible acoustic signals. Furthermore, we present a novel application developed based on this framework for retinal surgery. This new paradigm opens up possibilities for designing multisensory applications for multimodal imaging data. It also facilitates the creation of interactive sonification models with various auditory causality approaches, enhancing both directness and richness.
19 May 2026 in IEEE Access
Percutaneous epicardial access (PEA), performed on a beating heart under fluoroscopy, enables arrhythmia treatment. However, advancing a needle toward the thin, moving pericardium remains highly challenging and risky. To address this challenge, we present a novel physics-driven sonification method for Extended Reality (XR)-based multisensory navigation to enhance user perception regarding critically sensitive needle landing phase in PEA. Dynamic cardiac anatomy from 4DCTA was reconstructed and registered to a real-world coordinate system. Real-time needle tracking provided data of the needle tip relative to moving cardiac structures, driving the audio-visual feedback module. The visual display presents navigational cues and dynamic cardiac structures, whereas the auditory display encodes physiological cardiac states into a rigorous mapping through a multilayered physical membrane. To prove this novel concept, a phantom study was conducted with twelve cardiologists performing needle insertions under visual-only and multisensory feedback. Results show that the proposed multisensory method improved the navigation safety (χ2 = 11.30, p < 0.01), reducing myocardial contact (3.64% vs. 7.27%) and increasing correct access (90.91% vs. 52.73%). Needle placement accuracy improved, with closer membrane proximity (Cliff’s δ = 0.19) and reduced variability (Fligner p < 0.05). Execution time remained comparable, but time–accuracy correlations differed significantly between modalities (Fisher’s p < 0.01). NASA-TLX showed lower cognitive load with multisensory guidance (t = −4.02, p < 0.01). In conclusion, the multisensory system is associated with improved spatiotemporal awareness and suggests potential benefits for surgical navigation, while indicating reduced cognitive load. These findings demonstrate the feasibility of physics-driven sonification for supporting user-centered interaction and motivate further validation in realistic and clinical environments.
1 January 2026 in IEEE Access