24 June 2024 in IEEE Transactions on Biomedical Engineering
Brain-Computer Interface (BCI) has gained remarkable prominence in biomedical community. While BCI holds vast potential across diverse domains, the implantation of neural electrodes poses multifaceted challenges to fully explore the power of BCI. Conventional rigid electrodes face the problem of foreign body reaction induced by mechanical mismatch to biological tissue, while flexible electrodes, though more preferential, lack controllability during implantation. Researchers have explored various strategies, from assistive shuttle to biodegradable coatings, to strike a balance between implantation rigidity and post-implantation flexibility. Yet, these approaches may introduce complications, including immune response, inflammations, and raising intracranial pressure. To this end, this paper proposes a novel nanorobot-based technique for direct implantation of flexible neural electrodes, leveraging the high controllability and repeatability of robotics to enhance the implantation quality. This approach features a dual-arm nanorobotic system equipped with stereo microscope, by which a flexible electrode is first visually aligned to the target neural tissue to establish contact and thereafter implanted into brain with well controlled insertion direction and depth. The key innovation is, through dual-arm coordination, the flexible electrode maintains straight along the implantation direction. With this approach, we implanted CNTf electrodes into cerebral cortex of mouse, and captured standard spiking neural signals.
5 June 2023 in IEEE Transactions on Automation Science and Engineering
Despite of the promising advancements in the last decade, the majority of scanning electron microscope (SEM)-based nanomanipulation tasks remain manually performed. Even in automated tasks, human intervention is required, at least during the task preparatory stage, where both the object of interest and robot end-effector are adequately enclosed in the field of view (FOV) of SEM at moderate magnification. This paper proposes a fully automated visual servo control method for workspace navigation of nanorobot end-effector that actively maintains the end-effector position offset from the center of FOV during passive working scene zooming and translation operations. We also propose using the scaling image Jacobian matrix theory to adaptively establish the hand-eye relationship of the nanorobotic system at uncalibrated magnifications, without the need for hardware regulation. This proposed method for workspace navigation is applicable to almost all commercial nanomanipulation systems. To the authors’ best knowledge, there has been no dedicated research on this problem in the literature. Experiments show that the proposed method significantly improves workspace navigation efficiency by about two-thirds, even for a skilled operator. Note to Practitioners —Workspace navigation refers to the process of moving the FOV to visually enclose the object or feature of interest at a moderate magnification, which is prerequisite for initializing either an automated or manual nanomanipulation task. This process involves consecutive zooming of the SEM in and out, as well as translating the sample stage, which can cause the end-effector to move out of the FOV. Therefore, it is necessary to move the robot end-effector with visual assistance along workspace navigation to avoid unexpected collisions. Typically, skilled operators perform this process manually in almost all nanomanipulation tasks, which is tedious and time-consuming. Therefore, developing an automated visual servo control method for workspace navigation of end-effector will significantly contribute to the nanomanipulation community.
13 June 2023 in IEEE Transactions on Nanotechnology
Nanorobotic manipulation inside Scanning Electron Microscope (SEM) is widely used in nanomaterial characterization and assembly tasks. However, tedious and time-consuming hardware regulations, like axis alignment, are still critical and mandatory to keep the functionality of nanorobot systems. This paper proposes a novel zooming-free hand-eye self-calibration method for nanorobotic manipulation inside SEM to circumvent the hardware regulation problem, which is applicable to a wide range of system configurations. The hand-eye relationship calibration method is based on the active incremental motion of the nanorobot without referring to expensive and complicated calibration patterns. Both perspective projection at low magnification and affine projection at high magnification are modelled, and unified by the image Jacobian matrix technique. With several times active calibration at different magnification factors in advance, the hand-eye relationship can be, thereafter, determined from the model at arbitrary status. The calibration model can be applied to both accurate SEM visual measurement and precision visual servo control. The corresponding methods regarding two-dimensional (2D) measurement, in-plane positioning, and constrained three-dimension (3D) positioning based on the proposed hand-eye relationship model are also presented in this paper. Experiments show that practical contact between robot end-effector and silicon nanowire under 1500× is well established with standard deviation less than one pixel.
15 January 2025 in IEEE Transactions on Biomedical Engineering
Knot tying is a critical task in robotic surgery, which is considerably important for surgical success and postoperative recovery. Despite of the well-established protocols and significant progress using medical robots at macro scale, the need for automatedly tying mechanically robust knots on mini-incisions remains largely unmet, particularly with relieved suture deformation, avoided suture slippage, reduced workspace consumption, and enhanced precision and biomechanical compatibility. Here, we propose an innovative dual-arm nanorobotic system featured by stereo microscope and additional rotation degree of freedom (DOF) mounted on each arm, enabling automated, precise, and controllable knot tying on mini-incisions. With this system, a synchronous rotation-based knot tying (SRKT) trajectory is designed to mitigate undesired suture deformation, suture slippage, and minimize workspace consumption. The theoretical and simulation analysis of the topology and tension force distribution on suture validated the efficacy of SRKT trajectory. The experiments demonstrated that the dual-arm nanorobotic system with SRKT trajectory can be used for tying various knot types, using different micro-sutures, conducting on mini-incisions of different tissues, especially keeping suture slippage avoidance, minimal workspace, and robust mechanical strength of the tied knots. The proposed dual-arm nanorobotic system and SRKT trajectory render a significant potential for various practical medical applications.
25 September 2023 in IEEE Transactions on Instrumentation and Measurement
The paper investigates the problem of visual tracking on moving object in nanomanipulation inside scanning electron microscope (SEM). Image noise is a primary concern when dealing with the problem, which includes the inherent statistical noise and that induced by the motion distortion. A visual tracking method with SEM image denoising algorithm is proposed. The denoising algorithm is well incorporated with robot motion by innovatively leveraging the image Jacobian matrix technique. The denosing algorithm removes image noise and provides more realistic image. On the basis, template matching is utilized to achieve visual tracking on image plane. Experimental results show that the visual tracking performance on image plane was well improved by 31.4% for point feature, 60.6% for line feature, and 52.1% for area feature in terms of root mean square (RMSE) compared to that without motion distortion removal. Comparison experiments validate the state-of-the-art performance achieved by the proposed method and thus the superiority.
24 March 2025 in IEEE Transactions on Instrumentation and Measurement
Acoustic reconstruction aims to recreate target acoustic fields by spatially modulating acoustic waves and has significant application potential. On basis of acoustic holography, this paper focuses on low-cost acoustic reconstruction technique using binary amplitude-only hologram (BAOH). A novel physics-incorporated deep learning framework trained with a two-stage strategy is proposed, achieving outstanding accuracy and real-time performance in BAOH prediction. Specifically, we introduce the Binary U-net (BU-net) architecture, which combines the classical U-net with a customized Binary Layer. With the unique design, BU-net is capable to yield binary results without being hindered by the gradient invalidation. By integrating the acoustic wave propagation model, BU-net is trained to learn the inverse mapping from the target acoustic field to the corresponding source BAOH, also eliminating the need for labor-intensive annotation collection. The simulation experiments show that the proposed method can reduce the influence of quality degradation from binarized source holograms and achieved satisfactory reconstruction quality. Comparison experiments further demonstrate that the superiority of our proposed method over state-of-the-art (SOTA) method in the aspects of both accuracy and real-time performance. Finally, physical experiments confirm the alignment with simulation results, showcasing promising potential for real-world applications.