Color vision deficiency represents an inability to perceive differences between certain colors that can be distinguished in the case of regular color vision. This article proposes a new daltonization method for re-coloring image segments perceived as confusingly colored by color deficient observers and, thus, to improve their perceived image quality. The idea behind this approach is that only one image color center should be located on one confusion line. If colors of two or more segments lie on the same confusion line, then they should be remapped in a direction perpendicular to the confusion line taking into account the image content—the color distribution of other segments. The method conserves the image naturalness by restricting, for each segment center, an area of admissible remapping. This achieves re-coloring balance where colors are made sufficiently distinguishable from each other, but they do not deviate too much from the original image colors. The proposed re-coloring concept is applicable to all types of dichromacy and anomalous trichromacy with adjustment for their set of confusion lines. The simulation results show that the re-coloring converts an original image to a version with improved color distinction, confirmed by evaluations of eight color deficient users. An intentionally chosen example demonstrates when and why the proposed method performs better against content-independent methods. Furthermore, results and subjective evaluations indicate that the method provides more natural re-coloring results for anomalous trichromats than in the case of content-dependent methods optimized for dichromats.