Color texture classification has been an area of intensive research activity. From the very onset, approaches to combining color and texture have been the subject of much discussion, and in particular, whether they should be considered joint or separately. We present a comprehensive comparison of the most prominent approaches both from a theoretical and experimental standpoint. The main contributions of our work are: (i) the establishment of a generic and extensible framework to classify methods for color texture classification on a mathematical basis, and (ii) a theoretical and experimental comparison of the most salient existing methods. Starting from an extensive set of experiments based on the Outex dataset, we highlight those texture descriptors that provide good accuracy along with low dimensionality. The results suggest that separate color and texture processing is the best practice when one seeks for optimal compromise between accuracy and limited number of features. We believe that our work may serve as a guide for those who need to choose the appropriate method for a specific application, as well as a basis for the development of new methods.