Place recognition is an essential spatial intelligence capability for robots to understand and navigate the world. However, recognizing places in natural environments remains a challenging task for robots because of resource limitations and changing environments. In contrast, humans and animals can robustly and efficiently recognize hundreds of thousands of places in different conditions. Here, we report a brain-inspired general place recognition system, dubbed NeuroGPR, that enables robots to recognize places by mimicking the neural mechanism of multimodal sensing, encoding, and computing through a continuum of space and time. Our system consists of a multimodal hybrid neural network (MHNN) that encodes and integrates multimodal cues from both conventional and neuromorphic sensors. Specifically, to encode different sensory cues, we built various neural networks of spatial view cells, place cells, head direction cells, and time cells. To integrate these cues, we designed a multiscale liquid state machine that can process and fuse multimodal information effectively and asynchronously using diverse neuronal dynamics and bioinspired inhibitory circuits. We deployed the MHNN on Tianjic, a hybrid neuromorphic chip, and integrated it into a quadruped robot. Our results show that NeuroGPR achieves better performance compared with conventional and existing biologically inspired approaches, exhibiting robustness to diverse environmental uncertainty, including perceptual aliasing, motion blur, light, or weather changes. Running NeuroGPR as an overall multi–neural network workload on Tianjic showcases its advantages with 10.5 times lower latency and 43.6% lower power consumption than the commonly used mobile robot processor Jetson Xavier NX.

Recent advances in artificial intelligence have enhanced the abilities of mobile robots in dealing with complex and dynamic scenarios. However, to enable computationally intensive algorithms to be executed locally in multitask robots with low latency and high efficiency, innovations in computing hardware are required. Here, we report TianjicX, a neuromorphic computing hardware that can support true concurrent execution of multiple cross-computing-paradigm neural network (NN) models with various coordination manners for robotics. With spatiotemporal elasticity, TianjicX can support adaptive allocation of computing resources and scheduling of execution time for each task. Key to this approach is a high-level model, “Rivulet,” which bridges the gap between robotic-level requirements and hardware implementations. It abstracts the execution of NN tasks through distribution of static data and streaming of dynamic data to form the basic activity context, adopts time and space slices to achieve elastic resource allocation for each activity, and performs configurable hybrid synchronous-asynchronous grouping. Thereby, Rivulet is capable of supporting independent and interactive execution. Building on Rivulet with hardware design for realizing spatiotemporal elasticity, a 28-nanometer TianjicX neuromorphic chip with event-driven, high parallelism, low latency, and low power was developed. Using a single TianjicX chip and a specially developed compiler stack, we built a multi-intelligent-tasking mobile robot, Tianjicat, to perform a cat-and-mouse game. Multiple tasks, including sound recognition and tracking, object recognition, obstacle avoidance, and decision-making, can be concurrently executed. Compared with NVIDIA Jetson TX2, latency is substantially reduced by 79.09 times, and dynamic power is reduced by 50.66%.

Intelligent navigation is a fundamental technology that enables unmanned systems to achieve autonomy in the intelligent era. However, existing navigation schemes suffer from high computational complexity and power consumption, as well as low robustness in complex or unknown environments. To address these challenges, this article proposes a novel 3-D brain-inspired simultaneous localization and mapping (SLAM) method, called oriented FAST and rotated BRIEF (ORB)-NeuroSLAM, based on the ORB features. The proposed method takes inspiration from the robust and low-power navigation capabilities of humans and animals. The ORB-NeuroSLAM leverages the ORB features of camera images to compute robot self-motion and visual cues. Then, continuous attractor neural networks (CANNs) model multilayered head-direction cells and 3-D grid cells that exist in animal brains. These cells are utilized jointly to represent the robot poses. Efficient and robust methods for loop closure detection and experience map construction were also developed. The proposed method was verified on ten KITTI data sets, and experimental results demonstrate that it outperforms state-of-the-art brain-inspired SLAM methods in terms of accuracy and efficiency. Additionally, it is comparable to the state-of-the-art visual SLAM method ORB-SLAM3.

Misalignment between the camera frame and the operator frame is commonly seen in a teleoperated system and usually degrades the operation performance. The effects of such misalignment have not been fully investigated for eye-in-hand systems–systems that have the camera ( eye ) mounted to the end-effector ( hand ) to gain compactness in confined spaces such as in endoscopic surgery. This article provides a systematic study on the effects of the camera frame misalignment in a teleoperated eye-in-hand robot and proposes a simple correction method in the view display. A simulation is designed to compare the effects of the misalignment under different conditions. Users are asked to move a rigid body from its initial position to the specified target position via teleoperation, with different levels of misalignment simulated. It is found that misalignment between the input motion and the output view is much more difficult to compensate by the operators when it is in the orthogonal direction ( $\sim$ 40 s) compared with the opposite direction ( $\sim$ 20 s). An experiment on a real concentric tube robot with an eye-in-hand configuration is also conducted. Users are asked to telemanipulate the robot to complete a pick-and-place task. Results show that with the correction enabled, there is a significant improvement in the operation performance in terms of completion time (mean 40.6%, median 38.6%), trajectory length (mean 34.3%, median 28.1%), difficulty (50.5%), unsteadiness (49.4%), and mental stress (60.9%).

Vision-based ego-velocity estimation is a fundamental problem in robot state estimation. However, the constraints of frame-based cameras, including motion blur and insufficient frame rates in dynamic settings, readily lead to the failure of conventional velocity estimation techniques. Mammals exhibit a remarkable ability to accurately estimate their ego-velocity during aggressive movement. Hence, integrating this capability into robots shows great promise for addressing these challenges. In this letter, we propose a brain-inspired framework for linear-angular velocity estimation, dubbed NeuroVE. The NeuroVE framework employs an event camera to capture the motion information and implements spiking neural networks (SNNs) to simulate the brain's spatial cells' function for velocity estimation. We formulate the velocity estimation as a time-series forecasting problem. To this end, we design an Astrocyte Leaky Integrate-and-Fire (ALIF) neuron model to encode continuous values. Additionally, we have developed an Astrocyte Spiking Long Short-term Memory (ASLSTM) structure, which significantly improves the time-series forecasting capabilities, enabling an accurate estimate of ego-velocity. Results from both simulation and real-world experiments indicate that NeuroVE has achieved an approximate 60% increase in accuracy compared to other SNN-based approaches.

Digital terrestrial multimedia broadcasting (DTMB) signals have been proven as good signals of opportunity (SoP) for positioning due to their stronger transmission power, while micro-electro-mechanical systems (MEMS) sensors are feasible to couple with wireless signals to achieve good performance of positioning. Studies have found that neurons in the hippocampus and the entorhinal cortex are the “GPS” system in the brain, which can help animals navigate in large and complex environments. To achieve stable and intelligent positioning, a neuro-inspired positioning system named MM-NeuroPos is proposed to integrate MEMS sensors and DTMB signals. In specific, MEMS sensors in the smartphone are used to acquire self-motion cues, and a DTMB signals receiver is developed to perceive external ranging cues. Then, the self-motion cues and external ranging cues are integrated into the multi-type and multi-scale navigation cells model, including grid cells, head direction cells, speed cells, and distance cells. Finally, the optimal position is decoded from these different types of multi-scale cells by maximum probability optimization to realize the construction of the experience map. Field test results showed that the 68% positioning errors of the MM-NeuroPos system are 1.19 m, while errors of the self-motion cues only are 1.91 m, and the errors of the classical particle filter (PF) system are 1.65 m. This proves that the MM-NeuroPos system based on multi-type and multi-scale neuron model has superior positioning performance than the PF system.