Synthetic aperture radar (SAR) ship detection has been the focus of many previous studies. Traditional SAR ship detectors face challenges in complex environments due to the limitations of manual feature extraction. With the rise of deep learning (DL) techniques, SAR ship detection based on convolutional neural networks (CNNs) has achieved significant achievements. However, research on CNN-based SAR ship detection has mainly focused on improving detection accuracy, and relatively little research has been conducted on reducing computational complexity. Therefore, this paper proposes a lightweight detector, LssDet, for SAR ship detection. LssDet uses Shufflenet v2, YOLOX PAFPN and YOLOX Decopuled Head as the baseline networks, improving based on the cross sidelobe attention (CSAT) module, the lightweight path aggregation feature pyramid network (L-PAFPN) module and the Focus module. Specifically, the CSAT module is an attention mechanism that enhances the model’s attention to the cross sidelobe region and models the long-range dependence between the channel and spatial information. The L-PAFPN module is a lightweight feature fusion network that achieves excellent performance with little computational effort and a low parametric count. The Focus module is a low-loss feature extraction structure. Experiments showed that on the Sar ship detection dataset(SSDD), LssDet’s computational cost was 2.60 GFlops, the model’s volume was 2.25 M and $AP@[0.5:0.95]$ was 68.1%. On the Large-scale SAR ship detection dataset-v1.0 (LS-SSDD-v1.0), LssDet’s computational cost was 4.49 GFlops, the model’s volume was 2.25 M and $AP@[0.5:0.95]$ was 27.8%. Compared to the baseline network, LssDet had a 3.6% improvement in $AP@[0.5:0.95]$ on the SSDD, and LssDet had a 1.5% improvement in $AP@[0.5:0.95]$ on the LS-SSDD-v1.0. At the same time, LssDet reduced Floating-point operations per second (Flops) by 7.1% and Paraments (Params) by 23.2%. Extensive experiments showed that LssDet achieves excellent detection results with minimal computational complexity. Furthermore, we investigated the effectiveness of the proposed module through ablation experiments.