Jonas Hein

Jonas Hein

Auto-generated profile, based on publicly available information

Jonas Hein

University Hospital Balgrist, University of Zurich, Zurich, Switzerland

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
21 April 2021 in International Journal of Computer Assisted Radiology and Surgery
Verified
Journal Article
Open access
PDF file
JHJonas Hein
Jonas Hein
MSMatthias Seibold
Matthias Seibold
FBFederica Bogo
Federica Bogo
MFMazda Farshad
Mazda Farshad
MPMarc Pollefeys
Marc Pollefeys
NNNassir Navab
Nassir Navab
Purpose: : Tracking of tools and surgical activity is becoming more and more important in the context of computer assisted surgery. In this work, we present a data generation framework, dataset and baseline methods to facilitate further research in the direction of markerless hand and instrument pose estimation in realistic surgical scenarios. Methods: : We developed a rendering pipeline to create inexpensive and realistic synthetic data for model pretraining. Subsequently, we propose a pipeline to capture and label real data with hand and object pose ground truth in an experimental setup to gather high-quality real data. We furthermore present three state-of-the-art RGB-based pose estimation baselines. Results: : We evaluate three baseline models on the proposed datasets. The best performing baseline achieves an average tool 3D vertex error of 16.7 mm on synthetic data as well as 13.8 mm on real data which is comparable to the state-of-the art in RGB-based hand/object pose estimation. Conclusion: : To the best of our knowledge, we propose the first synthetic and real data generation pipelines to generate hand and object pose labels for open surgery. We present three baseline models for RGB based object and object/hand pose estimation based on RGB frames. Our realistic synthetic data generation pipeline may contribute to overcome the data bottleneck in the surgical domain and can easily be transferred to other medical applications.

10 May 2025 in Medical Image Analysis
Verified
Journal Article
Open access
JHJonas Hein
Jonas Hein
NCNicola Cavalcanti
Nicola Cavalcanti
DSDaniel Suter
Daniel Suter
LZLukas Zingg
Lukas Zingg
Fabio CarrilloFabio Carrillo
Fabio Carrillo
Philipp FürnstahlPhilipp Fürnstahl
Philipp Fürnstahl
State-of-the-art research of traditional computer vision is increasingly leveraged in the surgical domain. A particular focus in computer-assisted surgery is to replace marker-based tracking systems for instrument localization with pure image-based 6DoF pose estimation using deep-learning methods. However, state-of-the-art single-view pose estimation methods do not yet meet the accuracy required for surgical navigation. In this context, we investigate the benefits of multi-view setups for highly accurate and occlusion-robust 6DoF pose estimation of surgical instruments and derive recommendations for an ideal camera system that addresses the challenges in the operating room. Our contributions are threefold. First, we present a multi-view RGB-D video dataset of ex-vivo spine surgeries, captured with static and head-mounted cameras and including rich annotations for surgeon, instruments, and patient anatomy. Second, we perform an extensive evaluation of three state-of-the-art single-view and multi-view pose estimation methods, analyzing the impact of camera quantities and positioning, limited real-world data, and static, hybrid, or fully mobile camera setups on the pose accuracy, occlusion robustness, and generalizability. Third, we design a multi-camera system for marker-less surgical instrument tracking, achieving an average position error of 1.01mm and orientation error of 0.89° for a surgical drill, and 2.79mm and 3.33° for a screwdriver under optimal conditions. Our results demonstrate that marker-less tracking of surgical instruments is becoming a feasible alternative to existing marker-based systems.

12 December 2024 in Computers in Biology and Medicine
Verified
Journal Article
Open access
LWLuohong Wu
Luohong Wu
MSMatthias Seibold
Matthias Seibold
NCNicola A. Cavalcanti
Nicola A. Cavalcanti
JHJonas Hein
Jonas Hein
TGTatiana Gerth
Tatiana Gerth
PFPhilipp Fürnstahl
Philipp Fürnstahl
Total Hip Arthroplasty (THA) is a well-established and common orthopedic surgery. Due to the complexity involved in THA, orthopedic surgeons require rigorous training. However, the current gold standard, the tutor-guided and -evaluated apprenticeship model is time-consuming, costly, and poses risks to patients. There is a pressing need for additional training resources to enhance the efficiency and safety of the training process. In this work, we present a novel Augmented Reality (AR)-based simulator designed for THA that helps enable a new self-paced training and learning paradigm without the need for instructors. The simulator reduces the need for instructors by integrating an AR guidance module and an automated performance evaluation module. Three types of AR guidance were developed: Overlay, Virtual Twin, and Sectional Views. A feasibility study was conducted with five resident surgeons and two senior surgeons to compare these guidance methods quantitatively and qualitatively. The automated performance evaluation module was assessed against manual performance evaluation using Bland-Altman analysis with limits of agreement (LoA) and Mann-Whitney U tests. The quantitative feasibility results indicate the efficacy of the developed AR guidance, characterized by mean transitional and rotational deviation errors below 3 mm and 3 degrees. Based on the qualitative results, we provide recommendations for efficient AR guidance designs. The Bland-Altman analysis results (0.22±1.32mm with LoA -2.37 to 2.81 mm for distance deviation, 0.94±2.41 degrees with LoA -3.78 to 5.66 degrees for yaw deviation, -0.34±1.30 degrees with LoA -2.90 to 2.22 degrees for pitch deviation) and p-values of Mann-Whitney U tests (0.64 for distance deviation, 0.12 for yaw deviation, 0.11 for pitch deviation) indicate no statistically significant differences between the automated and manual performance evaluation at a significance level of 0.05. This work shows the potential of AR-based simulators in introducing a novel, data-driven approach to open surgery training in orthopedics, enabling surgeons to individually assess and improve their progress.

4 June 2025 in Computers in Biology and Medicine
Verified
Journal Article
Open access
LWLuohong Wu
Luohong Wu
NCNicola A. Cavalcanti
Nicola A. Cavalcanti
MSMatthias Seibold
Matthias Seibold
GLGiuseppe Loggia
Giuseppe Loggia
LRLisa Reissner
Lisa Reissner
PFPhilipp Fürnstahl
Philipp Fürnstahl
Ultrasound-based bone surface segmentation is crucial in computer-assisted orthopedic surgery. However, ultrasound images have limitations, including a low signal-to-noise ratio, acoustic shadowing, and speckle noise, which make interpretation difficult. Existing deep learning models for bone segmentation rely primarily on costly manual labeling by experts, limiting dataset size and model generalizability. Additionally, the complexity of ultrasound physics and acoustic shadow makes the images difficult for humans to interpret, leading to incomplete labels in low-intensity and anechoic regions and limiting model performance. To advance the state-of-the-art in ultrasound bone segmentation and establish effective model benchmarks, larger and higher-quality datasets are needed. We propose a methodology for collecting ex-vivo ultrasound datasets with automatically generated bone labels, including anechoic regions. The proposed labels are derived by accurately superimposing tracked bone Computed Tomography (CT) models onto the tracked ultrasound images. These initial labels are refined to account for ultrasound physics. To clinically evaluate the proposed method, an expert physician from our university hospital specialized in orthopedic sonography assessed the quality of the generated bone labels. A neural network for bone segmentation is trained on the collected dataset and its predictions are compared to expert manual labels, evaluating accuracy, completeness, and F1-score. We collected UltraBones100k, the largest known dataset comprising 100k ex-vivo ultrasound images of human lower limbs with bone annotations, specifically targeting the fibula, tibia, and foot bones. A Wilcoxon signed-rank test with Bonferroni correction confirmed that the bone alignment after our optimization pipeline significantly improved the quality of bone labeling (p<0.001). The model trained on UltraBones100k consistently outperforms manual labeling in all metrics, particularly in low-intensity regions (at a distance threshold of 0.5 mm: 320% improvement in completeness, 27.4% improvement in accuracy, and 197% improvement in F1 score) CONCLUSION:: This work is promising to facilitate research and clinical translation of ultrasound imaging in computer-assisted interventions, particularly for applications such as 2D bone segmentation, 3D bone surface reconstruction, and multi-modality bone registration.

31 May 2025 in Computer Assisted Surgery
Verified
Journal Article
Open access
AMAli Bahari Malayeri
Ali Bahari Malayeri
MSMatthias Seibold
Matthias Seibold
NCNicola A. Cavalcanti
Nicola A. Cavalcanti
JHJonas Hein
Jonas Hein
SJSascha Jecklin
Sascha Jecklin
Philipp FürnstahlPhilipp Fürnstahl
Philipp Fürnstahl
This study advances surgical phase recognition in arthroscopic procedures, specifically Anterior Cruciate Ligament (ACL) reconstruction, by introducing the first arthroscopy dataset and a novel transformer-based model. We establish a benchmark for arthroscopic surgical phase recognition by leveraging spatio-temporal features to address challenges such as limited field of view, occlusions, and visual distortions. We developed the ACL27 dataset, comprising 27 videos of ACL surgeries, each labeled with surgical phases. Our model employs a transformer-based architecture, utilizing temporal-aware frame-wise feature extraction through ResNet-50 and transformer layers. This approach integrates spatio-temporal features and introduces a Surgical Progress Index (SPI) to quantify surgery progression. The model's performance was evaluated using accuracy, precision, recall, and Jaccard Index on the ACL27 and Cholec80 datasets. The proposed model achieved an overall accuracy of 72.9% on the ACL27 dataset. On the Cholec80 dataset, the model achieved performance comparable to state-of-the-art methods, with an accuracy of 92.4%. The SPI demonstrated an output error of 10.6% and 9.8% on ACL27 and Cholec80 datasets, respectively, indicating reliable surgery progression estimation. This study introduces a significant advancement in surgical phase recognition for arthroscopy, providing a comprehensive dataset and robust transformer-based model. The results validate the model's effectiveness and generalizability, highlighting its potential to improve surgical training, real-time assistance, and operational efficiency in orthopedic surgery. The publicly available dataset and code will facilitate future research in this critical field. Word Count: 6490.

17 September 2024 in arXiv
Verified
Preprint
Open access
PDF file
JHJonas Hein
Jonas Hein
JGJan Grunder
Jan Grunder
LCLilian Calvet
Lilian Calvet
FGFrédéric Giraud
Frédéric Giraud
NCNicola Alessandro Cavalcanti
Nicola Alessandro Cavalcanti
PFPhilipp Fürnstahl
Philipp Fürnstahl
Virtual Reality technology, when integrated with Surgical Digital Twins (SDTs), offers significant potential in medical training and surgical planning. We present SurgTwinVR, a VR application that immerses users within an SDT and enables them to navigate a high-fidelity virtual replica of the surgical environment. SurgTwinVR is the first VR application to utilize a dynamic 3D environment that is a clone of a real surgery, encompassing the entire surgical scene, including the surgeon, anatomy, and instruments. Our system utilizes a SDT with important improvements for real-time rendering and features to showcase the potential benefits of such an application in surgical education.