Indoor fire accidents have become increasingly common in recent years. More and more firefighting robots have been designed to solve the problem of fires. However, the indoor environment is very complex, with high temperatures, thick smoke, more turns, and various burning substances. In this study, a firefighting robot with autonomous inspection and automatic fire-extinguishing functions intended for use in unknown indoor environments was designed. Considering water’s poor efficiency and its inability to extinguish some combustion materials, other fire extinguishers were applied to design the extinguishing system. The robot can install four different extinguishers as required and select the appropriate fire extinguisher to spray it automatically. Based on the Cartographer SLAM (simultaneous localization and mapping) theory, a robot map-building system was built using Lidar scanners, IMU (inertial measurement unit) sensors, encoders, and other sensors. The accurate identification and location of the fire source were achieved using an infrared thermal imager and the YOLOv4 deep learning algorithm. Finally, the performance of the firefighting robot was evaluated by creating a simulated-fire experimental environment. In an autonomous inspection process of the on-fire environment, the firefighting robot could identify the flame in real-time, trigger the fire-extinguishing system to carry out automatic fire extinguishing, and contain the fire in its embryonic stage.
Thermal infrared imaging is less affected by lighting conditions and smoke compared to visible light imaging. However, thermal infrared images often have lower resolution and lack rich texture details, making them unsuitable for stereo matching and 3D reconstruction. To enhance the quality of infrared stereo imaging, we propose an advanced stereo matching algorithm. Firstly, the images undergo preprocessing using a non-local mean noise reduction algorithm to remove thermal noise and achieve a smoother result. Subsequently, we perform camera calibration using a custom-made chessboard calibration board and Zhang’s camera calibration method to obtain accurate camera parameters. Finally, the disparity map is generated using the SGBM (semi-global block matching) algorithm based on the weighted least squares method, enabling the 3D point cloud reconstruction of the object. The experimental results demonstrate that the proposed algorithm performs well in objects with sufficient thermal contrast and relatively simple scenes. The proposed algorithm reduces the average error value by 10.9 mm and the absolute value of the average error by 1.07% when compared with the traditional SGBM algorithm, resulting in improved stereo matching accuracy for thermal infrared imaging. While ensuring accuracy, our proposed algorithm achieves the stereo reconstruction of the object with a good visual effect, thereby holding high practical value.
Due to hot toxic smoke and unknown risks under fire conditions, detection and relevant reconnaissance are significant in avoiding casualties. A fire reconnaissance robot was therefore developed to assist in the problem by offering important fire information to fire fighters. The robot consists of three main systems, a display operating system, video surveillance, and mapping and positioning navigation. Augmented reality (AR) goggle technology with a display operating system was also developed to free fire fighters’ hands, which enables them to focus on rescuing processes and not system operation. Considering smoke disturbance, a thermal imaging video surveillance system was included to extract information from the complicated fire conditions. Meanwhile, a simultaneous localization and mapping (SLAM) technology was adopted to build the map, together with the help of a mapping and positioning navigation system. This can provide a real-time map under the rapidly changing fire conditions to guide the fire fighters to the fire sources or the trapped occupants. Based on our experiments, it was found that all the tested system components work quite well under the fire conditions, while the video surveillance system produces clear images under dense smoke and a high-temperature environment; SLAM shows a high accuracy with an average error of less than 3.43%; the positioning accuracy error is 0.31 m; and the maximum error for the navigation system is 3.48%. The developed fire reconnaissance robot can provide a practically important platform to improve fire rescue efficiency to reduce the fire casualties of fire fighters.
Fire robots are an effective way to save lives from fire, but their limited detection accuracy has greatly hampered their practical applications in complicated fire conditions. This study therefore proposes an advanced thermal imaging flame detection model of YOLOv4-F based on YOLOv4-tiny. We replaced the Leaky ReLU activation function with the Mish activation function in the YOLOV4-tiny feature extraction network. A Spatial Pyramid Pooling (SPP) was also added to increase the receiving range of the feature extraction network. To improve the feature fusion efficiency between multi-scale feature layers, a Path Aggregation Network (PANet) was adopted to replace the YOLOv4-tiny Feature Pyramid Network (FPN) with full use of feature information; a high-quality dataset containing 14,757 thermal imaging flame images was built according to the PASCAL VOC 2007 dataset standard. The results show that, when compared to the YOLOv4-tiny, YOLOv5-s, and YOLOv7-tiny models, the average detection accuracy of the proposed YOLOv4-F model is 5.75% higher, the average mAP of the five IOU cases rises by 7.02%, and the average detection confidence of three scaled flames shows a 18.09% gain. The proposed YOLOV4-F meets the requirements of fire robots on real-time responses and accurate flame detection, offering an important tool to improve the performance of the current fire robots.
Obtaining thermal infrared images with prominent details, high contrast, and minimal background noise has always been a focal point of infrared technology research. To address issues such as the blurriness of details and low contrast in thermal infrared images, an enhancement algorithm for thermal infrared images based on multi-scale guided filtering is proposed. This algorithm fully leverages the excellent edge-preserving characteristics of guided filtering and the multi-scale nature of the edge details in thermal infrared images. It uses multi-scale guided filtering to decompose each thermal infrared image into multiple scales of detail layers and a base layer. Then, CLAHE is employed to compress the grayscale and enhance the contrast of the base layer image. Then, detail-enhancement processing of the multi-scale detail layers is performed. Finally, the base layer and the multi-scale detail layers are linearly fused to obtain an enhanced thermal infrared image. Our experimental results indicate that, compared to other methods, the proposed method can effectively enhance image contrast and enrich image details, and has higher image quality and stronger scene adaptability.
Low visibility and hot smoke environment under fire conditions can largely hamper the related fire rescue processes. Ultrasound obstacle avoidance technology is then useful for guidance. However, the biggest challenge of adopting ultrasound technology comes from accurate distance measurements under the disturbances of high temperature and soot particle concentration. It is critical to measure the propagation speed under the complicated fire conditions. Therefore, in this study, a baffle calibration method was proposed to improve the accuracy of distance measurement of an obstacle. The method is based on two ultrasound measurement systems, while one is used to calibrate the propagation speed of ultrasound based on the fixed distanced baffle and the other is for the dynamic measurement of obstacle distance based on the calibrated speed. The viability of this method on the guided vehicle was confirmed based on the experiments. From its comparison to those existing methods, such as constant speed and temperature compensation methods, it was known from that the proposed baffle calibration method provides the best prediction. It was obtained that the maximum errors based on the baffle calibration method are 2.75% and 2.62% under the two representative fire scenarios, respectively, which are much lower than those of constant speed (7.81% and 8.4%) and temperature compensation methods (10.4% and 5.12%).