With the proposed goal of “Carbon Neutrality”, photovoltaic energy is gradually gaining the leading role in energy transformation. At present, crystalline silicon cells are still the mainstream technology in the photovoltaic industry, but due to the similarity of defect characteristics and the small scale of the defects, automatic defect detection of photovoltaic cells (PV) by electroluminescence (EL) imaging is a challenging task. In order to better meet the growing demand for high-quality photovoltaic cell products in intelligent manufacturing and use, and ensure the safe and efficient operation of photovoltaic power stations, this paper proposes an improved abnormal detection method based on Faster R-CNN for the surface defect EL imaging of photovoltaic cells, which integrates a lightweight channel and spatial convolution attention module. It can analyze the crack defects in complex scenes more efficiently. The clustering algorithm was used to obtain a more targeted anchor frame for photovoltaic cells, which made the model converge faster and enhanced the detection ability. The normalized distance between the prediction box and the target box is minimized by considering the DIoU loss function for the overlapping area of the boundary box and the distance between the center points. The experiment shows that the average accuracy of surface defect detection for EL images of photovoltaic cells is improved by 14.87% compared with the original algorithm, which significantly improves the accuracy of defect detection. The model can better detect small target defects, meet the requirements of surface defect detection of photovoltaic cells, and proves that it has good application prospects in the field of photovoltaic cell defect detection.