HyperSpectral (HS) images have been successfully used for brain tumor boundary detection during resection operations. Nowadays, these classification maps coexist with other technologies such as MRI or IOUS that improve a neurosurgeon’s action, with their incorporation being a neurosurgeon’s task. The project in which this work is framed generates an unified and more accurate 3D immersive model using HS, MRI, and IOUS information. To do so, the HS images need to include 3D information and it needs to be generated in real-time operating room conditions, around a few seconds. This work presents Graph cuts Reference depth estimation in GPU (GoRG), a GPU-accelerated multiview depth estimation tool for HS images also able to process YUV images in less than 5.5 s on average. Compared to a high-quality SoA algorithm, MPEG DERS, GoRG YUV obtain quality losses of −0.93 dB, −0.6 dB, and −1.96% for WS-PSNR, IV-PSNR, and VMAF, respectively, using a video synthesis processing chain. For HS test images, GoRG obtains an average RMSE of 7.5 cm, with most of its errors in the background, needing around 850 ms to process one frame and view. These results demonstrate the feasibility of using GoRG during a tumor resection operation.

Purpose: Magnetic resonance imaging (MRI) is a common technique in image-guided neurosurgery (IGN). Recent research explores the integration of methods like ultrasound and tomography, among others, with hyperspectral (HS) imaging gaining attention due to its non-invasive real-time tissue classification capabilities. The main challenge is the registration process, often requiring manual intervention. This work introduces an automatic, markerless method for aligning HS images with MRI. Methods: This work presents a multimodal system that combines RGB-Depth (RGBD) and HS cameras. The RGBD camera captures the patient’s facial geometry, which is used for registration with the preoperative MR through ICP. Once MR-depth registration is complete, the integration of HS data is achieved using a calibrated homography transformation. The incorporation of external tracking with a novel calibration method allows camera mobility from the registration position to the craniotomy area. This methodology streamlines the fusion of RGBD, HS and MR images within the craniotomy area. Results: Using the described system and an anthropomorphic phantom head, the system has been characterised by registering the patient’s face in 25 positions and 5 positions resulted in a fiducial registration error of 1.88 ± 0.19 mm and a target registration error of 4.07 ± 1.28 mm, respectively. Conclusions: This work proposes a new methodology to automatically register MR and HS information with a sufficient accuracy. It can support the neurosurgeons to guide the diagnosis using multimodal data over an augmented reality representation. However, in its preliminary prototype stage, this system exhibits significant promise, driven by its cost-effectiveness and user-friendly design.

Hyperspectral imaging (HSI) has a crucial role in material classification tasks. The primary type of sensors in this field are linescan sensors, offering a high spatial and spectral resolution and improving the quality of the captured hypercubes. However, these cameras have inherent problems such as the need of dedicated hardware to move the camera over the scene and the necessity of pre-processing the raw captures in order to obtain the actual hypercube. This work makes an analysis and an adaptation of stitching techniques to apply them to the medical area. Also, it includes different methods to post-process the hypercubes obtained and it analyzes the SURF algorithm to improve processing times. The results obtained show that a stitching technique based on SURF is suitable for the medical area.

Hyperspectral (HS) imaging (HSI) is a powerful image technique that allows capturing spatial and spectral information, being able to characterize materials, tissues, and elements in a non-invasive manner. HSI technology is well established at the macroscopic level, but there are still technical challenges to overcome before it can be applied to the microscopic world, such as the lack of standardized characterization methodologies to HS microscopic systems that allow the correct data acquisition as well as ensure the repeatability of the experiments. In this work, we propose a comprehensive roadmap for characterizing and validating such systems, integrating essential parameters highlighted in the current state of the art. Furthermore, we provide a list of the materials needed for their characterization and testing the methodology on two different HS microscopic systems chosen as representative of common configurations in the field, where a HS camera is integrated into a bright-field microscope. Our proposed roadmap assesses the following parameters: dynamic range, noise quantification, pixel size, spatial frequency response, spatial scanning accuracy, spatial repeatability, flat-field correction, tone transfer, and spectral sensitivity. We address the challenge of unifying these parameters into a unified and standardized roadmap. All data used to characterize both systems have been captured by the authors. In summary, this comprehensive analysis provides a guideline for the scientific community to develop and characterize HS microscopic systems to ensure reliability, efficiency, and accuracy.

Purpose: This study proposes a generalization of markerless patient registration in image-guided neurosurgery based on depth information. The work builds on previous research to evaluate the performance of a range of commercial depth cameras and two different registration algorithms in this context. Methods: A multimodal experimental setup was used, testing five depth cameras in seven configurations. Fiducial registration error (FRE) and target registration error (TRE) metrics were calculated using iterative closest point (ICP) and deep global registration (DGR) algorithms. A phantom head model was used to simulate clinical conditions, with cameras positioned to capture the face and craniotomy regions. Results: The best-performing cameras, such as the D405 and Zed-M+, achieved TRE values as low as 2.36 ± 0.46 mm and 2.49 ± 0.35 mm, respectively, compared to manual registration that obtains a 1.37 mm error. Cameras equipped with texture projectors or enhanced depth refinement demonstrated improved performance. The proposed methodology effectively characterized the suitability of the camera for the registration tasks. Conclusion: This study validates an adaptable and reproducible framework to evaluate depth cameras in neurosurgical scenarios, highlighting D405 and Zed-M + as reliable options. Future work will focus on improving depth quality through hardware and algorithmic improvements. The experimental data and the accompanying code were made publicly available to ensure reproducibility.