Fermentation is the key process to produce the special color of congou black tea. The machine vision technology is applied to detect the color space changes of black tea’s color in RGB, Lab and HSV, and to find out its relevance to black tea’s fermentation quality. And then the color feature parameter is used as input to establish physicochemical indexes (TFs, TRs, and TBs) and sensory features’ linear and non-linear quantitative evaluation model. Results reveal that color features are significantly correlated to quality indices. Compared with the other two color models (RGB and HSV), CIE Lab model can better reflect the dynamic variation features of quality indices and foliage color information of black tea. The predictability of non-linear models (RF and SVM) is superior to PLS linear model, while RF model presents a slight advantage over the classic SVM model since RF model can better represent the quantitative analytical relationship between image information and quality indices. This research has proved that computer image color features and non-linear method can be used to quantitatively evaluate the changes of quality indices (e.g. sensory quality) and the pigment during black tea’s fermentation. Besides, the test is simple, fast, and nondestructive.