Due to the special properties of synthetic aperture radar (SAR) images, they are widely used in maritime applications, such as detecting ships at sea. To perform ship detection in SAR images, existing algorithms commonly utilize convolutional neural network (CNN). However, the challenges in acquiring SAR images and the imaging noise hinder CNN in performing SAR ship-detection tasks. In this paper, we revisit the relationship between SAR expert features and network abstract features, and propose an expert-feature-based convolutional neural network (EFCNet). Specifically, we exploit the inherent physical properties of SAR images by manually extracting a range of expert features, including electromagnetic scattering, geometric structure, and grayscale statistics. These expert features are then adaptively integrated with abstract CNN features through a newly designed multi-source features association module, which improves the common CNN’s capability to recognize ship targets. Experiment results on the SSDD demonstrate that EFCNet outperforms general CNN approaches. Furthermore, EFCNet achieves comparable detection performance to baseline methods while utilizing only 70% of the data capacity, highlighting its efficiency. This work aims to reignite interest in leveraging expert features in remote sensing tasks and offers promising avenues for improved SAR image interpretation.