The selection of the optimizer is critical for convergence in the field of on-chip training. As one second moment optimizer, adaptive moment estimation (ADAM) shows a significant advantage compared with non-moment optimizers such as stochastic gradient descent (SGD) and first-moment optimizers such as Momentum. However, ADAM is hard to implement on hardware due to the computationally intensive operations, including square, root extraction, and division. This work proposed Hardware-ADAM (HW-ADAM), an efficient fixed-point accelerator for ADAM highlighting hardware-oriented mathematical optimizations. HW-ADAM has two designs: Efficient-ADAM (E-ADAM) unit reduced the hardware resource consumption by around 90% compared with the related work. E-ADAM achieved a throughput of 2.89 MUOP/s (Million Updating Operation per Second), which is 2.8× of the original ADAM. Fast-ADAM (F-ADAM) unit reduced 91.5% flip-flops, 65.7% look-up tables, and 50% DSPs compared with the related work. The F-ADAM unit achieved a throughput of 16.7 MUOP/s, which is 16.4× of the original ADAM.

With the development of deep learning, a higher level of perception of the environment such as the semantic level can be achieved in the simultaneous localization and mapping (SLAM) domain. However, previous works did not achieve a natural-language level of perception. Therefore, LP-SLAM (Language-Perceptive RGB-D SLAM) is proposed that leverages large language models (LLMs). The texts in the scene can be detected by scene text recognition (STR) and mapped as landmarks with a task-driven selection. A text error correction chain (TECC) is designed with a similarity classification method, a two-stage memory strategy, and a text clustering method. The proposed architecture is designed to deal with the mis-detection and mis-recognition cases of STR and to provide accurate text information to the framework. The proposed framework takes input images and generates a 3D map with sparse point cloud and task-related texts. Finally, a natural user interface (NUI) is designed based on the constructed map and LLM, which gives position instructions based on users’ natural queries. The experimental results validated the proposed TECC design and the overall framework. We publish the virtual dataset with ground truth, as well as the source code for further research. https://github.com/GroupOfLPSLAM/LP_SLAM.

The hardware architecture of a deep-neural-network-based feature point extraction method is proposed for the simultaneous localization and mapping (SLAM) in robotic applications, which is named the Feature Point based SLAM Network (FPSNET). Some key techniques are deployed to improve the hardware and power efficiency. The data path is devised to reduce overall off-chip memory accesses. The intermediate data and partial sums resulting in the convolution process are stored in available on-chip memories, and optimized hardware is employed to compute the one-point activation function. Meanwhile, address generation units are used to avoid data overlapping in memories. The proposed FPSNET has been designed in 65 nm CMOS technology with a core area of 8.3 mm². This work reduces the memory overhead by 50% compared to traditional data storage for activation and overall by 35% for on-chip memories. The synthesis and simulation results show that it achieved a 2.0× higher performance compared with the previous design while achieving a power efficiency of 1.0 TOPS/W, which is 2.4× better than previous work. Compared to other ASIC designs with close peak throughput and power efficiency performance, the presented FPSNET has the smallest chip area (at least 42.4% reduction).

This article presents an energy-efficient hardware accelerator for optimized bundle adjustment (BA) for mobile high-frame-rate visual odometry (VO). BA uses graph optimization techniques to optimize poses and landmarks and the applications are robot navigation, virtual reality (VR), and augmented reality (AR). Existing software implementations of BA optimization involve complex computational flows, numerical calculations, Lie group, and Lie algebra conversions. This poses challenges of slow computational speeds and high power consumption. A two-level reuse hardware architecture is proposed and implemented that efficiently updates the Jacobian matrix while reducing the field-programmable gate array (FPGA) hardware resources by 25%. A set of methodologies is proposed to quantify the errors caused by fixed-point systems during optimization. A fully pipelined architecture is implemented to increase computational speed while reducing hardware resources by 29%. This design features a parallel equation solver that improves processing speed by $2\times $ compared to conventional approaches. This article employs a single-frame local BA VO on the KITTI dataset and EuRoC dataset, achieving an average translational error of 0.75% and a rotational error of $0.0028~^{\circ } $ /m. The proposed hardware achieves a performance ranging from 188 to 345 frames/s in optimizing two main feature extraction methods with a maximum of 512 extracted feature points. Compared to state-of-the-art implementations, the accelerator achieved a minimum energy efficiency ratio of 11.6 mJ and $191~\mu $ J on the FPGA platform and application-specific integrated circuits (ASICs) platform, respectively. These improvements underscore the potential of FPGAs to enhance VO systems’ adaptability and efficiency in complex environments.

With the rapid development of information in the new era, a concept is put forward. You can observe a scene that doesn't exist near you at home, outdoors and anywhere. It can be a swimming pool, a museum or even an Olympic Games. In this paper, we design a 3D scene roaming system based on augmented reality, which uses augmented reality technology to build a 3D virtual world presented by AR devices in the real world, and with the support of AR devices, users can walk around freely and autonomously in the 3D virtual world to observe the presented scene in an all-round way. The design of the roaming system is mainly developed in Windows environment. The unity3d platform is used for 3D scene design and creation. Mask technology is used to realize the Mask on the basis of 3D scene. A transparent invisible film is pasted on the surface of 3D scene to hide some scenes. It is required that the hidden scenes can be seen only after the scene roaming through the Mask film. Using the ARcore development technology of building the software platform of augmented reality application launched by Google to realize the presentation of 3D scene and human-computer interaction. Package all programs into SDKs and send them to Android platform for running and effect verification.

Simultaneous Localization and Mapping (SLAM) is the process by which a mobile robot can build a map of the surrounding environment and compute its own location. Feature point extraction is one of the key components of a SLAM system. The extraction accuracy and efficiency of corner detection directly affect the overall accuracy and throughput of the system. However, the complexity of corner detection algorithms makes it challenging to achieve real-time implementation and efficient, low-cost hardware design, especially for mobile robots. Harris corner detection class algorithms including Harris and GFTT (Good Feature to Track) have improved accuracy. However, those algorithms require high resource consumption and latency when implemented on hardware platforms. The GFTT achieves higher accuracy than Harris while requiring higher computational complexity. To address the throughput problem, SFTT (Simple Feature to Track), a new Harris class detection algorithm is proposed, and the corresponding hardware accelerator is designed. The proposed SFTT significantly reduced the computational complexity compared with the Harris algorithm and GFTT. Experiments have shown SFTT also achieved slightly higher accuracy compared with the two algorithms. Furthermore, the GFTT accelerator is designed which reaches up to 325 fps at the frequency of 100MHz. The proposed design has achieved an improvement in throughput by 1.3× times and power efficiency by 1.7× times as compared to state-of-the-art design.