When a mobile robot is asked to navigate intelligently in an environment, it needs to estimate its own and the environment's state. One of the popular methods for robot state and position estimation is particle filtering (PF). Visual Teach and Repeat (VT&R) is a type of navigation that uses a camera to navigate the robot along the previously traversed path. Particle filters are usually used in VT&R to fuse data from odometry and camera to estimate the distance traveled along the path. However, in VT&R, there are other valuable states that the robot can benefit from, especially when moving through changing environments. We propose a multidimensional particle filter to estimate the robot state in VT&R navigation. Apart from the traveled distance, our particle filter estimates lateral and heading deviation from the taught path as well as the current appearance of the environment. This appearance is estimated using maps created across various environmental conditions recorded during the previous traversals. The joint state estimation is based on contrastive neural network architecture, allowing self-supervised learning. This architecture can process multiple images in parallel, alleviating the potential overhead caused by computing the particle filter over the maps simultaneously. We conducted experiments to show that the joint robot/environment state estimation improves navigation accuracy and robustness in a continual mapping setup. Unlike the other frameworks, which treat the robot position and environment appearance separately, our PF represents them as one multidimensional state, resulting in a more general uncertainty model for VT&R.

Visual teach and repeat navigation (VT&R) is popular in robotics thanks to its simplicity and versatility. It enables mobile robots equipped with a camera to traverse learned paths without the need to create globally consistent metric maps. Although teach and repeat frameworks have been reported to be relatively robust to changing environments, they still struggle with day-to-night and seasonal changes. This paper aims to find the horizontal displacement between prerecorded and currently perceived images required to steer a robot towards the previously traversed path. We employ a fully convolutional neural network to obtain dense representations of the images that are robust to changes in the environment and variations in illumination. The proposed model achieves state-of-the-art performance on multiple datasets with seasonal and day/night variations. In addition, our experiments show that it is possible to use the model to generate additional training examples that can be used to further improve the original model’s robustness. We also conducted a real-world experiment on a mobile robot to demonstrate the suitability of our method for VT&R.

The performance of deep neural networks and the low costs of computational hardware has made computer vision a popular choice in many robotic systems. An attractive feature of deep-learned methods is their ability to cope with appearance changes caused by day–night cycles and seasonal variations. However, deep learning of neural networks typically relies on large numbers of hand-annotated images, which requires significant effort for data collection and annotation. We present a method that allows autonomous, self-supervised training of a neural network in visual teach-and-repeat (VT&R) tasks, where a mobile robot has to traverse a previously taught path repeatedly. Our method is based on a fusion of two image registration schemes: one based on a Siamese neural network and another on point-feature matching. As the robot traverses the taught paths, it uses the results of feature-based matching to train the neural network, which, in turn, provides coarse registration estimates to the feature matcher. We show that as the neural network gets trained, the accuracy and robustness of the navigation increases, making the robot capable of dealing with significant changes in the environment. This method can significantly reduce the data annotation efforts when designing new robotic systems or introducing robots into new environments. Moreover, the method provides annotated datasets that can be deployed in other navigation systems. To promote the reproducibility of the research presented herein, we provide our datasets, codes and trained models online.

Nowadays, robots can operate in environments which are not tailored for them. This allows their deployments in changing and human-populated environments, which recent advances in machine learning methods enabled. The efficiency of these methods is largely determined by the quality of their training data. An up-to-date and well-balanced training dataset is paramount for achieving robust robot operation. To achieve long-term operation, the robot has to deal with perpetual environmental changes, forcing it to keep its models up-to-date. We present an exploration method allowing a mobile robot to gather high-quality data to update its models both while performing its duties and when idle, maximizing effectivity. The robot evaluates the quality of the data gathered in the past and based on that, it creates preferences which influence how often these locations are visited. This exploration method was integrated with a self-supervised visual teach-and-repeat pipeline. We show the precision and robustness of visual-based navigation to improve when using machine-learned models trained by our exploration method. Our research resulted in a robotic navigation system that can not only annotate its training data but also ensure that its training dataset is balanced and up-to-date.

The advances in deep learning for image processing greatly impacted mobile robots’ ability to navigate using vision-based techniques. There are many applications in which the robot is required to follow a specific visual cue. One newly emerging method for autonomous navigation along a consistent visual cue is Visual Teach and Generalise (VTAG). It is a unified approach to extract suitable features for path following in repetitive and structured environments. However, a relatively large number of data samples is required for the time-consuming training of a detector tied to a specific visual cue. We introduce an improved version of VTAG that does not need any neural network training and significantly reduces the requirements for the number of data samples. The core idea is to use an average representation of images collected during short supervised traversals, capturing characteristics of repetitive paths. The average representation is created in a latent space crafted by contrastive learning with the linear matching scheme and can be computed in real time, even on a CPU. The capability of rapid deployment, precision, and robustness of the presented method is evaluated in multiple field experiments performed in different environments.