In this study, a multi-starters fermentation system involved lactic acid bacteria and yeasts was applied to obtain a novel acidified goat milk (AGM). Significant differences were found in the volatile flavor profile among goat milk, goat yogurt, and AGM reflected by principal component analysis of electronic nose (E-nose) data. Gas chromatography–mass spectrometry (GC-MS) results indicated that the relative content of free octanoic acid decreased, and more aromas were formed in AGM, which were considered to mask the goaty smell and give AGM a pleasant flavor. Rheological analysis indicated that AGM had higher apparent viscosity and G’ and G’’ moduli than goat yogurt and goat milk. Therefore, the goat yogurt fermented by lactic acid bacteria and K. marxianus exhibits a new method to alleviate the goaty flavor in goat milk and provides a novel option for those who were allergic to milk protein and dislike goaty flavor in goat milk.