Automated segmentation and evaluation of carotid plaques ultrasound images is of great significance for the diagnosis and early intervention of high-risk groups of cardiovascular and cerebrovascular diseases. However, it remains challenging to develop such solutions due to the relatively low quality of ultrasound images and heterogenous characteristics of carotid plaques. To address those problems, in this paper, we propose a novel deep convolutional neural network, FRDD-Net, with an encoder–decoder architecture to automatically segment carotid plaques. We propose the feature remapping modules (FRMs) and incorporate them into the encoding and decoding blocks to ameliorate the reliability of acquired features. We also propose a new dense decoding mechanism as part of the decoder, thus promoting the utilization efficiency of encoded features. Additionally, we construct a compound loss function to train our network to further enhance its robustness in the face of numerous cases. We train and test our network in multiple carotid plaque ultrasound datasets and our method yields the best performance compared to other state-of-the-art methods. Further ablation studies consistently show the advancement of our proposed architecture.