We present a workflow for seamless real-time navigation and 3D thermal mapping in combined indoor and outdoor environments in a global reference frame. The automated workflow and partly real-time capabilities are of special interest for inspection tasks and also for other time-critical applications. We use a hand-held integrated positioning system (IPS), which is a real-time capable visual-aided inertial navigation technology, and augment it with an additional passive thermal infrared camera and global referencing capabilities. The global reference is realized through surveyed optical markers (AprilTags). Due to the sensor data’s fusion of the stereo camera and the thermal images, the resulting georeferenced 3D point cloud is enriched with thermal intensity values. A challenging calibration approach is used to geometrically calibrate and pixel-co-register the trifocal camera system. By fusing the terrestrial dataset with additional geographic information from an unmanned aerial vehicle, we gain a complete building hull point cloud and automatically reconstruct a semantic 3D model. A single-family house with surroundings in the village of Morschenich near the city of Jülich (German federal state North Rhine-Westphalia) was used as a test site to demonstrate our workflow. The presented work is a step towards automated building information modeling.

In professional use cases like police or fire brigade missions, coordinated and systematic force management is crucial for achieving operational success during intervention by the emergency personnel. A real-time situation picture enhances the coordination of the team. This situation picture includes not only an overview of the environment but also the positions, i.e., localization, of the emergency forces. The overview of the environment can be obtained either from known situation pictures like floorplans or by scanning the environment with the aid of visual sensors. The self-localization problem can be solved outdoors using the Global Navigation Satellite System (GNSS), but it is not fully solved indoors, where the GNSS signal might not be received or might be degraded. In this paper, we propose a novel combination of an inertial localization technique based on simultaneous localization and mapping (SLAM) with 3D building scans, which are used as prior information, for geo-referencing the positions, obtaining a situation picture, and finally visualizing the results with an appropriate visualization tool. We developed a new method for converting point clouds into a hexagonal prism map specifically designed for our SLAM algorithm. With this combination, we could keep the equipment for first responders as lightweight as required. We showed that the positioning led to an average accuracy of less than 1m indoors, and the final visualization including the building layout obtained by the 3D building reconstruction will be advantageous for coordinating first responder operations.

For professional use cases like police or fire brigade operations, a reliable self-localization is advantageous for the coordination of first responders (FRs). Self-localization is especially difficult in indoor scenarios where neither global navigation satellite systems’ signals may be received nor additional signals of opportunity exist for enabling reliable positioning. In this article, we propose an overall system that combines self-localization, communication of the FRs’ locations, 3-D building reconstruction or floor plans (if available), and visualization. The indoor navigation technique is based solely on inertial sensors and builds on a simultaneous localization and mapping technique. It is capable of using any information about the building layout as prior information for enhancing indoor positioning, georeferencing the positions, and finally, visualizing the results in a suitable visualization tool.

The ability to test methods based on simulated data is an important component in the development of intelligent localization systems at the present time. This includes the replication of realistic environments and sensors on the one hand. On the other hand, an implementation of a correct movement profile of the target mobile platform is required. This is complicated by the large variety of possible platform variants. We propose a method to transfer movement profiles that have been recorded in real world into a simulation environment. The associated modular simulation framework was designed especially, but not limiting, for the development of visual-inertial odometry of a handheld navigation system. We demonstrate the application of our pipeline through an experiment to determine the influence of motion blur on the visual odometry component.

Currently, self-localization of vehicles is primarily carried out by GPS receivers of different price and performance classes, which are permanently installed in vehicles. Known limitations of GPS-based localization include signal disturbances due to multipath propagation, as it may occur in narrow street canyons or tree alleys, as well as signal losses in tunnels or entrances to buildings, for example. It may happen that horizontal GPS channels still support a useful 2D positioning with low uncertainties while the GPS altitude channel has very large deviations or error uncertainties. Additional sensors (IMU or cameras) help to detect sections of trajectories where the GPS signal becomes unusable. Based on a traditional navigation strategy (i.e. integration of IMU and GPS), we also study the possible integration of visual odometry obtained from a stereo camera system. The paper reports about a large-scale project studying multi-sensor integration for very accurate and robust self-localization of vehicles.