To resolve the issues of a deep backbone network, a large model, slow reasoning speed on a mobile terminal, low detection accuracy for small targets and difficulties detecting and recognizing traffic lights in real time and accurately with YOLOv4, a traffic lights recognition method based on improved YOLOv4 is proposed. The lightweight ShuffleNetv2 network is utilized to replace the CSPDarkNet53 network of YOLOv4 to satisfy the requirements of a mobile terminal. The reformed k-means clustering algorithm is applied to generate anchor boxes for avoiding the sensitivity issue of outliers and initial values. A novel attention mechanism named CS²A is added to enhance the extraction capability of effective features. Multiple data augmentation methods are combined to improve the generalization ability of the model. Ultimately, the detection and recognition of traffic lights can be realized. The S²TLD dataset is selected for training and testing, and it can be proved that the recognition accuracy and model size are greatly optimized. Meanwhile, a self-made dataset is selected for training and testing. Compared with the conventional YOLOv4, the recognition accuracy of the proposed algorithm for traffic lights’ state information increases by 1.79%, and the model size decreases by 81.97%. Appropriate scenes are selected for real-vehicle testing and the results demonstrate that the detection speed of the presented algorithm increases by 16%, and the recognition effect for small targets increases by 37% in comparison with conventional YOLOv4.