It is a challenge to automatically and accurately segment the liver and tumors in computed tomography (CT) images, as the problem of over-segmentation or under-segmentation often appears when the Hounsfield unit (Hu) of liver and tumors is close to the Hu of other tissues or background. In this paper, we propose the spatial channel-wise convolution, a convolutional operation along the direction of the channel of feature maps, to extract mapping relationship of spatial information between pixels, which facilitates learning the mapping relationship between pixels in the feature maps and distinguishing the tumors from the liver tissue. In addition, we put forward an iterative extending learning strategy, which optimizes the mapping relationship of spatial information between pixels at different scales and enables spatial channel-wise convolution to map the spatial information between pixels in high-level feature maps. Finally, we propose an end-to-end convolutional neural network called Channel-UNet, which takes UNet as the main structure of the network and adds spatial channel-wise convolution in each up-sampling and down-sampling module. The network can converge the optimized mapping relationship of spatial information between pixels extracted by spatial channel-wise convolution and information extracted by feature maps and realizes multi-scale information fusion. The proposed ChannelUNet is validated by the segmentation task on the 3Dircadb dataset. The Dice values of liver and tumors segmentation were 0.984 and 0.940, which is slightly superior to current best performance. Besides, compared with the current best method, the number of parameters of our method reduces by 25.7%, and the training time of our method reduces by 33.3%. The experimental results demonstrate the efficiency and high accuracy of Channel-UNet in liver and tumors segmentation in CT images.

Haptic-based tissue stiffness perception is essential for palpation training system, which can provide the surgeon haptic cues for improving the diagnostic abilities. However, current haptic devices, such as Geomagic Touch, fail to provide immersive and natural haptic interaction in virtual surgery due to the inherent mechanical friction, inertia, limited workspace and flawed haptic feedback. To tackle this issue, we design a novel magnetic levitation haptic device based on electromagnetic principles to augment the tissue stiffness perception in virtual environment. Users can naturally interact with the virtual tissue by tracking the motion of magnetic stylus using stereoscopic vision so that they can accurately sense the stiffness by the magnetic stylus, which moves in the magnetic field generated by our device. We propose the idea that the effective magnetic field (EMF) is closely related to the coil attitude for the first time. To fully harness the magnetic field and flexibly generate the specific magnetic field for obtaining required haptic perception, we adopt probability clouds to describe the requirement of interactive applications and put forward an algorithm to calculate the best coil attitude. Moreover, we design a control interface circuit and present a self-adaptive fuzzy proportion integration differentiation (PID) algorithm to precisely control the coil current. We evaluate our haptic device via a series of quantitative experiments which show the high consistency of the experimental and simulated magnetic flux density, the high accuracy (0.28 mm) of real-time 3D positioning and tracking of the magnetic stylus, the low power consumption of the adjustable coil configuration, and the tissue stiffness perception accuracy improvement by 2.38 percent with the self-adaptive fuzzy PID algorithm. We conduct a user study with 22 participants, and the results suggest most of the users can clearly and immersively perceive different tissue stiffness and easily detect the tissue abnormality. Experimental results demonstrate that our magnetic levitation haptic device can provide accurate tissue stiffness perception augmentation with natural and immersive haptic interaction.

This paper presents a novel mixed reality (MR) guidance method for liver tumours radiofrequency ablation (RFA). Compared with traditional CT-guided method, our system can provide a more natural and intuitive surgical mode for surgeons. In essence, our system is a holographic navigation platform, which projects a mixed-reality overlay onto the patient via HoloLens during RFA. We first reconstruct the patient-specific anatomy structure from the CT images of abdominal phantom. Then, a tailored precise registration method is employed to map the virtual-real spatial information. In addition, considering that tumor shifting during biopsy severely impacts the accuracy of RFA, our guidance system involves a motion compensation computation through data-driven physicallybased modeling in holographic environment. In experiments, we conduct a user study on the comparison trial between MR-guided and CT-guided biopsy. User feedback demonstrates that our MR guidance method for needle placement procedure has the potential to simplify the operation, reduce the operation difficulty, shorten the operation time and raise the operation precision.

This paper presents a novel augmented reality (AR)-based neurosurgical training simulator which provides a very natural way for surgeons to learn neurosurgical skills. Surgical simulation with bimanual haptic interaction is integrated in this work to provide a simulated environment for users to achieve holographic guidance for pre-operative training. To achieve the AR guidance, the simulator should precisely overlay the 3D anatomical information of the hidden target organs in the patients in real surgery. In this regard, the patient-specific anatomy structures are reconstructed from segmented brain magnetic resonance imaging. We propose a registration method for precise mapping of the virtual and real information. In addition, the simulator provides bimanual haptic interaction in a holographic environment to mimic real brain tumor resection. In this study, we conduct AR-based guidance validation and a user study on the developed simulator, which demonstrate the high accuracy of our AR-based neurosurgery simulator, as well as the AR guidance mode’s potential to improve neurosurgery by simplifying the operation, reducing the difficulty of the operation, shortening the operation time, and increasing the precision of the operation.