Panoptic segmentation of ships plays a crucial role in intelligent navigation and maritime safety, providing essential references for route planning and collision avoidance. However, the complexity of the maritime environment, including issues such as water surface reflections, weather disturbances, and the challenge of detecting small ship targets, significantly increases the difficulty of the segmentation task. To address these challenges, this paper proposes a novel panoptic ship segmentation framework, FA PEM, based on the PEM algorithm. First, we propose the Dynamic Correlation-Aware Upsampling (DCAU) module, which adopts a content-adaptive sampling point selection and grouping upsampling strategy, significantly improving boundary alignment and fine-grained feature extraction. Second, we propose the Spatial-Frequency Attention Module (SFAM). By modeling both spatial and frequency domain features, this module integrates multi-scale deep convolutions and Fourier transforms, enhancing the model’s ability to perceive both global structures and local textures. Furthermore, to address the lack of an appropriate dataset for ship panoptic segmentation, we construct and annotate a new dataset, the Ship Panoptic Segmentation Dataset (SPSD), consisting of 4360 ship images. Experimental results demonstrate that FA PEM significantly outperforms the baseline FEM on both the Cityscapes and SPSD datasets, achieving advanced performance and exhibiting strong generalization ability.