High-quality milling of carbon fiber reinforced polymer (CFRP) composites is of great importance for the high-performance manufacturing of structures made of this hard-to-machine material. In this paper, a multiscale finite element (FE) model, considering the thermal–mechanical coupling effect, was developed to simulate the milling process and reveal its material removal mechanism. The corresponding milling experiments were conducted to validate the simulated cutting forces and temperature, which were in good agreement with the experiment results. In the macroscale model, the Hashin failure criteria were used to estimate the failure of the composites. In the microscale model, the fibers, matrix, and the fiber–matrix interface were modeled separately, to investigate the mechanisms of material removal behavior during milling, among fiber breakage, matrix cracking, and fiber–matrix debonding. Based on the macroscale numerical and experimental results, the higher cutting speed was demonstrated to improve the surface quality of CFRP milling. According to the results from the microscale model, the material removal mechanism varies depending on the orientation of the fibers and can be divided into four stages. The outcome of this work provides guidelines to further investigate optimal manufacturing parameters for the milling of CFRP composites and their cutting mechanisms.