The flotation chemistry between pyrite and isomeric xanthates (butyl xanthate and isobutyl xanthate) was investigated by means of adsorption experiments, surface tension tests, and molecular dynamic simulations in this work. The flotation chemical results were confirmed and further interpreted by quantum chemical calculations. The experiment results demonstrated that the isobutyl xanthate exhibited superior adsorption capacity and surface activity than those of butyl xanthate in flotation chemistry. In addition, molecular dynamic simulations were simultaneously performed in constant number, constant volume and temperature (NVT), and constant number, constant volume, and pressure (NPT) ensemble, indicating that the NPT ensemble was more suitable to the flotation system and the isobutyl xanthate was easier to be adsorbed on pyrite surface compared with butyl xanthate during an appropriate range of concentrations. Furthermore, the quantum chemical calculations elucidated that the isobutyl xanthate presented higher reactivity than that of the corresponding butyl xanthate based on the frontier molecular orbital theory of chemical reactivity, which was consistent with experimental and simulation results obtained. This work can provide theoretical guidance for an in-depth study of the flotation chemistry of pyrite with isomeric xanthates.