Acoustic holography (AH) is a promising technique for precise noncontact micro-nano robotic manipulation. It encodes a three-dimensional (3D) acoustic field acting as a virtual end-effector into a two-dimensional (2D) hologram, whereby the desired acoustic field reconstruction is made possible. Most traditional methods to implement AH, such as 3D printed holographic lens and phased array of transducers (PAT), have limitations of dynamic and dexterous manipulation. Furthermore, existing iterative optimization algorithms to calculate 2D holograms have inadequate accuracy and real-time performance. To address these issues, this paper proposes a physics-based deep learning method with a novel training framework for phase-only hologram (POH) calculation enabling further pushing forward the PAT-based AH for noncontact robotic manipulation. By implementing independent control of each channel on PAT referring real-time calculated POH by a well-trained network, the desired acoustic field can be reconstructed in real-time with high fidelity. The results both on a simulated dataset and a real dataset demonstrate that our method supports accurate and dynamic reconstruction of desired acoustic field with distinct morphologies, with an average reconstruction error of 0.085 and average POH computing time of 47 milliseconds on GPU. Indeed, this work shows the future potential of AH in the field of noninvasive medical therapy, exogenous material delivery, and miniaturized industrial assembly. Note to Practitioners —This paper addresses the challenge of noncontact micro-nano robotic manipulation by PAT-based AH, an intriguing technique in bioengineering, micro-assembly, and material characterization. However, existing approaches have limited precision and real-time performance. To overcome these limitations, this paper proposes a physics-based deep learning method with a novel training framework. Our method achieves excellent accuracy and real-time performance, enabling efficient reconstruction of various complicated acoustic field morphologies for precise and dynamic acoustic manipulation. Experimental results demonstrate its high manipulation flexibility due to the independent modulation of each channel of PAT and real-time precise control due to the ultrafast calculation of the proposed deep learning method, though the method has not yet been deployed into an acoustic manipulation system and tested in practice. Future research will focus on designing physical experiments for further evaluation. Overall, the proposed method provides a novel and promising basis for desired acoustic field generation.

Brain-Computer Interface (BCI) has gained remarkable prominence in biomedical community. While BCI holds vast potential across diverse domains, the implantation of neural electrodes poses multifaceted challenges to fully explore the power of BCI. Conventional rigid electrodes face the problem of foreign body reaction induced by mechanical mismatch to biological tissue, while flexible electrodes, though more preferential, lack controllability during implantation. Researchers have explored various strategies, from assistive shuttle to biodegradable coatings, to strike a balance between implantation rigidity and post-implantation flexibility. Yet, these approaches may introduce complications, including immune response, inflammations, and raising intracranial pressure. To this end, this paper proposes a novel nanorobot-based technique for direct implantation of flexible neural electrodes, leveraging the high controllability and repeatability of robotics to enhance the implantation quality. This approach features a dual-arm nanorobotic system equipped with stereo microscope, by which a flexible electrode is first visually aligned to the target neural tissue to establish contact and thereafter implanted into brain with well controlled insertion direction and depth. The key innovation is, through dual-arm coordination, the flexible electrode maintains straight along the implantation direction. With this approach, we implanted CNTf electrodes into cerebral cortex of mouse, and captured standard spiking neural signals.

Acoustic holography (AH) provides a promising technique for arbitrary acoustic field reconstruction, supporting many applications like robotic micro-nano manipulation, neuromodulation, volumetric imaging, and virtual reality. In AH, three-dimensional (3D) acoustic fields quantified with complex-valued acoustic pressures are reconstructed by virtue of two-dimensional (2D) acoustic holograms. Phase-only hologram (POH) is recently regarded as an energy-efficient way for AH, which is typically implemented by a dynamically programmable phased array of transducers (PATs). As a result, spatiotemporal precise acoustic field reconstruction is enabled by precise, dynamic, and individual actuation of PAT. Thus, 2D POH is required per arbitrary acoustic fields, which can be viewed as a physical inverse problem. However, solving the aforementioned physical inverse problem in numerical manners poses challenges due to its non-linear, high-dimensional, and complex coupling natures. The existing iterative algorithms like the iterative angular spectrum approach (IASA) and iterative backpropagation (IB) still suffer from speed-accuracy trade-offs. Hence, this paper explores a novel physics-iterative-reinforced deep learning method, in which frequency-argument contrastive learning is proposed facilitated by the inherent physical nature of AH, and the energy conservation law is under consideration. The experimental results demonstrate the effectiveness of the proposed method for acoustic field reconstruction, highlighting its significant potential in the domain of acoustics, and pushing forward the combination of physics into deep learning.

Despite of the promising advancements in the last decade, the majority of scanning electron microscope (SEM)-based nanomanipulation tasks remain manually performed. Even in automated tasks, human intervention is required, at least during the task preparatory stage, where both the object of interest and robot end-effector are adequately enclosed in the field of view (FOV) of SEM at moderate magnification. This paper proposes a fully automated visual servo control method for workspace navigation of nanorobot end-effector that actively maintains the end-effector position offset from the center of FOV during passive working scene zooming and translation operations. We also propose using the scaling image Jacobian matrix theory to adaptively establish the hand-eye relationship of the nanorobotic system at uncalibrated magnifications, without the need for hardware regulation. This proposed method for workspace navigation is applicable to almost all commercial nanomanipulation systems. To the authors’ best knowledge, there has been no dedicated research on this problem in the literature. Experiments show that the proposed method significantly improves workspace navigation efficiency by about two-thirds, even for a skilled operator. Note to Practitioners —Workspace navigation refers to the process of moving the FOV to visually enclose the object or feature of interest at a moderate magnification, which is prerequisite for initializing either an automated or manual nanomanipulation task. This process involves consecutive zooming of the SEM in and out, as well as translating the sample stage, which can cause the end-effector to move out of the FOV. Therefore, it is necessary to move the robot end-effector with visual assistance along workspace navigation to avoid unexpected collisions. Typically, skilled operators perform this process manually in almost all nanomanipulation tasks, which is tedious and time-consuming. Therefore, developing an automated visual servo control method for workspace navigation of end-effector will significantly contribute to the nanomanipulation community.

Nanorobotic manipulation inside Scanning Electron Microscope (SEM) is widely used in nanomaterial characterization and assembly tasks. However, tedious and time-consuming hardware regulations, like axis alignment, are still critical and mandatory to keep the functionality of nanorobot systems. This paper proposes a novel zooming-free hand-eye self-calibration method for nanorobotic manipulation inside SEM to circumvent the hardware regulation problem, which is applicable to a wide range of system configurations. The hand-eye relationship calibration method is based on the active incremental motion of the nanorobot without referring to expensive and complicated calibration patterns. Both perspective projection at low magnification and affine projection at high magnification are modelled, and unified by the image Jacobian matrix technique. With several times active calibration at different magnification factors in advance, the hand-eye relationship can be, thereafter, determined from the model at arbitrary status. The calibration model can be applied to both accurate SEM visual measurement and precision visual servo control. The corresponding methods regarding two-dimensional (2D) measurement, in-plane positioning, and constrained three-dimension (3D) positioning based on the proposed hand-eye relationship model are also presented in this paper. Experiments show that practical contact between robot end-effector and silicon nanowire under 1500× is well established with standard deviation less than one pixel.

Selective contactless manipulation of organisms with intrinsic mobility from heterogeneous mixture is essential for biomedical engineering and microbiology. Acoustic manipulation, compared to its optical, magnetic, and electrostatic counterparts, provides superior bio-compatibility and additive-free properties. In this study, we present an acoustic manipulation system capable of selectively trapping, translating, rotating, and orienting individual organisms from in-Petri dish organism mixture using a phased transducer array and microscope, by dynamically steering the acoustic field. Specifically, using brine shrimp and zebrafish populations as example, the to-be-manipulated organisms with different sizes or morphologies can be manually designated by the user in microscopic image and interactively localized. Thereafter, the selected organisms can be automatically trapped from the heterogeneous mixture using a multiple focal point-based acoustic field steering method. Finally, the trapped organisms can be translated, rotated, and oriented in regard to the user's distinct manipulation objectives in instant response. In different tasks, closed-loop positioning and real-time motion planning control are performed, highlighting the innovation in terms of automation and accuracy of our manipulation technique. The results demonstrate that our acoustic manipulation system and acoustic field steering method enable selective, stable, precision, real-time, and in-Petri dish manipulation of organisms from heterogeneous mixture.