Accurate traffic sign recognition is one of the core technologies of intelligent driving systems, which face multiple challenges such as insufficient light and shadow interference at night. In this paper, we improve the YOLOv5 model for small, fuzzy, and partially occluded traffic sign targets at night and propose a high-precision nighttime traffic sign recognition method, “NTS-YOLO”. The method firstly preprocessed the traffic sign dataset by adopting an unsupervised nighttime image enhancement method to improve the image quality under low-light conditions; secondly, it introduced the Convolutional Block Attention Module (CBAM) attentional mechanism, which focuses on the shape of the traffic sign by weighting the channel and spatial features inside the model and color to improve the perception under complex background and uneven illumination conditions; and finally, the Optimal Transport Assignment (OTA) loss function was adopted to optimize the accuracy of predicting the bounding box and thus improve the performance of the model by comparing the difference between two probability distributions, i.e., minimizing the difference. In order to evaluate the effectiveness of the method, 154 samples of typical traffic signs containing small targets and fuzzy and partially occluded traffic signs with different lighting conditions at nighttime were collected, and the data samples were subjected to the CBAM, OTA, and a combination of the two methods, respectively, and comparative experiments were conducted with the traditional YOLOv5 algorithm. The experimental results showed that “NTS-YOLO” achieved a significant performance improvement in nighttime traffic sign recognition, with a mean average accuracy improvement of 0.95% for the target detection of traffic signs and 0.17% for instance segmentation.