This paper proposes a method that automatically calibrates four cameras of an around view monitor (AVM) system in a natural driving situation. The proposed method estimates orientation angles of four cameras composing the AVM system, and assumes that their locations and intrinsic parameters are known in advance. This method utilizes lane markings because they exist in almost all on-road situations and appear across images of adjacent cameras. It starts by detecting lane markings from images captured by four cameras of the AVM system in a cost-effective manner. False lane markings are rejected by analyzing the statistical properties of the detected lane markings. Once the correct lane markings are sufficiently gathered, this method first calibrates the front and rear cameras, and then calibrates the left and right cameras with the help of the calibration results of the front and rear cameras. This two-step approach is essential because side cameras cannot be fully calibrated by themselves, due to insufficient lane marking information. After this initial calibration, this method collects corresponding lane markings appearing across images of adjacent cameras and simultaneously refines the initial calibration results of four cameras to obtain seamless AVM images. In the case of a long image sequence, this method conducts the camera calibration multiple times, and then selects the medoid as the final result to reduce computational resources and dependency on a specific place. In the experiment, the proposed method was quantitatively and qualitatively evaluated in various real driving situations and showed promising results.

This paper presents a method for simplifying and quantizing a deep neural network (DNN)-based object detector to embed it into a real-time edge device. For network simplification, this paper compares five methods for applying channel pruning to a residual block because special care must be taken regarding the number of channels when summing two feature maps. Based on the comparison in terms of detection performance, parameter number, computational complexity, and processing time, this paper discovers the most satisfying method on the edge device. For network quantization, this paper compares post-training quantization (PTQ) and quantization-aware training (QAT) using two datasets with different detection difficulties. This comparison shows that both approaches are recommended in the case of the easy-to-detect dataset, but QAT is preferable in the case of the difficult-to-detect dataset. Through experiments, this paper shows that the proposed method can effectively embed the DNN-based object detector into an edge device equipped with Qualcomm’s QCS605 System-on-Chip (SoC), while achieving a real-time operation with more than 10 frames per second.

This paper proposes a camera system designed for local dynamic map (LDM) generation, capable of simultaneously performing object detection, tracking, and 3D position estimation. This paper focuses on improving existing approaches to better suit our application, rather than proposing novel methods. We modified the detection head of YOLOv4 to enhance the detection performance for small objects and to predict fiducial points for 3D position estimation. The modified detector, compared to YOLOv4, shows an improvement of approximately 5% mAP on the Visdrone2019 dataset and around 3% mAP on our database. We also proposed a tracker based on DeepSORT. Unlike DeepSORT, which applies a feature extraction network for each detected object, the proposed tracker applies a feature extraction network once for the entire image. To increase the resolution of feature maps, the tracker integrates the feature aggregation network (FAN) structure into the DeepSORT network. The difference in multiple objects tracking accuracy (MOTA) between the proposed tracker and DeepSORT is minimal at 0.3%. However, the proposed tracker has a consistent computational load, regardless of the number of detected objects, because it extracts a feature map once for the entire image. This characteristic makes it suitable for embedded edge devices. The proposed methods have been implemented on a system on chip (SoC), Qualcomm QCS605, using network pruning and quantization. This enables the entire process to be executed at 10 Hz on this edge device.

In order to overcome the limitations of GNSS/INS and to keep the cost affordable for mass-produced vehicles, a precise localization system fusing the estimated vehicle positions from low-cost GNSS/INS and low-cost perception sensors is being developed. For vehicle position estimation, a perception sensor detects a road facility and uses it as a landmark. For this localization system, this paper proposes a method to detect a road sign as a landmark using a monocular camera whose cost is relatively low compared to other perception sensors. Since the inside pattern and aspect ratio of a road sign are various, the proposed method is based on the part-based approach that detects corners and combines them to detect a road sign. While the recall, precision, and processing time of the state of the art detector based on a convolutional neural network are 99.63%, 98.16%, and 4802 ms respectively, the recall, precision, and processing time of the proposed method are 97.48%, 98.78%, and 66.7 ms, respectively. The detection performance of the proposed method is as good as that of the state of the art detector and its processing time is drastically reduced to be applicable for an embedded system.

The low-cost global navigation satellite systems combined with an inertial navigation system (GNSS/INS) used most frequently for vehicle localization show errors up to 10 m, approximately, even in open-sky environments such as highways. To reduce this error on highways, this paper proposes a localization method based on lane endpoints. Since a lane endpoint frequently appears on a road and can be detected in close proximity even by a low-cost monocular camera, it is a very useful landmark for precise localization. However, the lane width is generally less than 3.5 m, and the localization error from the GNSS is about 10 m. Therefore, if an ego-lane is not identified, the lane endpoints detected in an ego-lane can be falsely corresponded to the lane endpoints in the other lane of a map. This paper proposes an in-lane localization method that uses lane endpoints, the relation between a camera and a road, and the estimated vehicle’s orientation from a map. In addition, this paper proposes an ego-lane identification method that generates a hypothesis about an ego vehicle position per lane by using the proposed in-lane localization method and verifies each hypothesis by the projection of lane endpoints and an additional landmark such as a road sign. The average error of the proposed in-lane localization method is 0.248 m on highways. The success rate of the proposed ego-lane identification method is 99.28% by one trial and reaches 100% by fusing the results.