The pigments of cultural color paintings have faded with the passage of time. Color segmentations are essential for digital color reconstruction, but the complexity of color paintings makes it challenging to achieve high-precision segmentation using previous methods. To address the challenges of color painting segmentation, an optimized strategy based on U-Net is proposed in this paper. The residual blocks of a residual network (ResNet) are added to the original U-Net architecture, and a UR-Net is constructed for the semantic segmentation of color paintings. The following steps are taken. First, datasets of color paintings are obtained as training and test samples and are labeled with the two following pixel colors: earth red and ultramarine blue. Second, residual blocks are improved and added to fit the U-Net architecture. Then, a UR-Net is constructed and trained using the samples to obtain the semantic segmentation model. Finally, the effectiveness of the trained UR-Net model for segmenting the test samples is evaluated, and it is compared with the K-means clustering algorithm, ResNet, and U-Net. Data from several studies suggest that the segmentation accuracy of the UR-Net model is higher than that of other methods for the color segmentation of painted images, and the IoUs of the segmented earth red and ultramarine blue pixels are 0.9346 and 0.9259, respectively, achieving the desired results. The proposed UR-Net model provides theoretical and methodological support for further in-depth research on color recognition and segmentation of cultural color paintings.