X-ray-based non-destructive testing technology plays a crucial role in the quality monitoring of aluminum alloy wheel hubs. Due to the characteristics of the casting process, wheel hub images often exhibit low contrast and a discrete distribution of defect edges. Existing methods often face problems such as poor feature extraction capability, low efficiency of cross-scale information fusion, and susceptibility to interference from complex backgrounds when detecting such defects. Therefore, this study proposes an innovative detection framework for defects in aluminum alloy wheel hubs. The model employs data preprocessing to enhance the quality of original images; integrates an asymmetric pinwheel-shaped convolution (PConv) with an efficient receptive field, enabling efficient focus on the edge feature information of discrete defects; innovatively constructs a Mamba-based two-stage feature pyramid network (MFDPN), which improves the network’s defect localization capability in complex scenarios via a secondary focusing-diffusion mechanism; and incorporates a channel and spatial attention block (CASAB), strengthening the model’s ability to resist interference from complex backgrounds. On our self-built wheel hub defect dataset, the proposed model outperforms the baseline by 7.2% in mAP50 and 5% in Recall at 39 FPS inference speed, thus validating its high practical utility for automated aluminum alloy wheel hub defect detection.