Surgical activity recognition and prediction can help provide important context in many Robot-Assisted Surgery (RAS) applications, for example, surgical progress monitoring and estimation, surgical skill evaluation, and shared control strategies during teleoperation. Transformer models were first developed for Natural Language Processing (NLP) to model word sequences and soon the method gained popularity for general sequence modeling tasks. In this paper, we propose the novel use of a Transformer model for three tasks: gesture recognition, gesture prediction, and trajectory prediction during RAS. We modify the original Transformer architecture to be able to generate the current gesture sequence, future gesture sequence, and future trajectory sequence estimations using only the current kinematic data of the surgical robot end-effectors. We evaluate our proposed models on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) and use Leave-One-User-Out (LOUO) cross validation to ensure generalizability of our results. Our models achieve up to 89.3% gesture recognition accuracy, 84.6% gesture prediction accuracy (1 second ahead) and 2.71mm trajectory prediction error (1 second ahead). Our models are comparable to and able to outperform state-of-the-art methods while using only the kinematic data channel. This approach can enable near-real time surgical activity recognition and prediction.

Increased levels of stress can impair surgeon performance and patient safety during surgery. The aim of this study is to investigate the effect of short term stressors on laparoscopic performance through analysis of kinematic data. Thirty subjects were randomly assigned into two groups in this IRB-approved study. The control group was required to finish an extended-duration peg transfer task (6 minutes) using the FLS trainer while listening to normal simulated vital signs and while being observed by a silent moderator. The stressed group finished the same task but listened to a period of progressively deteriorating simulated patient vitals, as well as critical verbal feedback from the moderator, which culminated in 30 seconds of cardiac arrest and expiration of the simulated patient. For all subjects, video and position data using electromagnetic trackers mounted on the handles of the laparoscopic instruments were recorded. A statistical analysis comparing time-series velocity, acceleration, and jerk data, as well as path length and economy of volume was conducted. Clinical stressors lead to significantly higher velocity, acceleration, jerk, and path length as well as lower economy of volume. An objective evaluation score using a modified OSATS technique was also significantly worse for the stressed group than the control group. This study shows the potential feasibility and advantages of using the time-series kinematic data to identify the stressful conditions during laparoscopic surgery in near-real-time. This data could be useful in the design of future robot-assisted algorithms to reduce the unwanted effects of stress on surgical performance.

Surgical movements have an important stylistic quality that individuals without formal surgical training can use to identify expertise. In our prior work, we sought to characterize quantitative metrics associated with surgical style and developed a near-real-time detection framework for stylistic deficiencies using a commercial haptic device. In this paper, we implement bimanual stylistic detection on the da Vinci Research Kit (dVRK) and focus on one stylistic deficiency, “Anxious”, which may describe movements under stressful conditions. Our goal is to potentially correct these “Anxious” movements by exploring the effects of three different types of haptic cues (time-variant spring, damper, and spring-damper feedback) on performance during a basic surgical training task using the da Vinci Research Kit (dVRK). Eight subjects were recruited to complete peg transfer tasks using a randomized order of haptic cues and with baseline trials between each task. Overall, all cues lead to a significant improvement over baseline economy of volume and time-variant spring haptic cues lead to significant improvements in reducing the classified “Anxious” movements and corresponded with significantly lower path length and economy of volume for the non-dominant hand. This work is the first step in evaluating our stylistic detection model on a surgical robot and could lay the groundwork for future methods to reduce the negative effect of stress actively and adaptively in the operating room.

Digital twin (DT) technology is recognized for its transformative potential to enhance efficiency in the construction process. However, the full potential of DT in construction practices remains largely unrealised. Moreover, few studies explore how DT success factors affect efficiency improvement (EI), energy optimization (EO), and cost control (CC) in the context of economic strength (ES). The study applied a hybrid research method to examine the impact of key DT success factors on EI, EO, and CC under the moderation of ES. After a critical literature review, five key DT success factors were identified. Then, 490 valid questionnaires were analyzed with the Partial Least Squares Structural Equation Model (PLS-SEM) to assess how success factors affect DT effectiveness. This is complemented using extreme gradient boosting (XGBoost) to assess prediction accuracy and understand which factors most influenced EI, EO, and CC. Research shows that ES exerts a significant positive influence on the relationships between most success factors and performance outcomes. High levels of ES enhance the contribution of success factors to performance in EI, EO, and CC. Resource management (RM) has a strong influence on EI and EO, but a weaker influence on CC; process optimization (PO) has the strongest influence on EO, a moderate influence on CC, and the weakest influence on EI; real-time monitoring (R-Tm) primarily affects EI; sustainable design (SD) has a comprehensive and significant regulatory effect on EI, EO, and CC; and predictive maintenance (PM) has a strong influence on both EI and CC. In practice, it offers practical guidance for implementing DT and supports policy and resource planning for building stakeholders.

It has been shown that intraoperative stress can have a negative effect on surgeon surgical skills during laparoscopic procedures. For novice surgeons, stressful conditions can lead to significantly higher velocity, acceleration, and jerk of the surgical instrument tips, resulting in faster but less smooth movements. However, it is still not clear which of these kinematic features (velocity, acceleration, or jerk) is the best marker for identifying the normal and stressed conditions. Therefore, in order to find the most significant kinematic feature that is affected by intraoperative stress, we implemented a spatial attention-based Long Short-Term Memory (LSTM) classifier. In a prior IRB approved experiment, we collected data from medical students performing an extended peg transfer task who were randomized into a control group and a group performing the task under external psychological stresses. In our prior work, we obtained “representative” normal or stressed movements from this dataset using kinematic data as the input. In this study, a spatial attention mechanism is used to describe the contribution of each kinematic feature to the classification of normal/stressed movements. We tested our classifier under Leave-One-User-Out (LOUO) cross-validation, and the classifier reached an overall accuracy of 77.11% for classifying “representative” normal and stressed movements using kinematic features as the input. More importantly, we also studied the spatial attention extracted from the proposed classifier. Velocity and acceleration on both sides had significantly higher attention for classifying a normal movement ($p\le 0.0001$ ); Velocity ($p\le 0.015$ ) and jerk ($p\le 0.001$ ) on nondominant hand had significant higher attention for classifying a stressed movement, and it is worthy noting that the attention of jerk on nondominant hand side had the largest increment when moving from describing normal movements to stressed movements ($p=0.0000$ ). In general, we found that the jerk on nondominant hand side can be used for characterizing the stressed movements for novice surgeons more effectively.