Compound faults diagnosis is a challenge for rotating machinery fault diagnosis. The vibration signals measured from gearboxes are usually complex, non-stationary, and nonlinear. When compound faults occur in a gearbox, weak fault characteristic signals are always submerged by the strong ones. Therefore, it is difficult to detect a weak fault by using the demodulating analysis of vibration signals of gearboxes directly. The key to compound faults diagnosis of gearboxes is to separate different fault characteristic signals from the collected vibration signals. Aiming at that problem, a new method for the compound faults diagnosis of gearboxes is proposed based on the energy operator demodulating of optimal resonance components. In this method, the genetic algorithm is first used to obtain the optimal decomposition parameters. Then the compound faults vibration signals of a gearbox are subject to resonance-based signal sparse decomposition (RSSD) to separate the fault characteristic signals of the gear and the bearing by using the optimal decomposition parameters. Finally, the separated fault characteristic signals are analyzed by energy operator demodulating, and each one’s instantaneous amplitude can be calculated. According to the spectra of instantaneous amplitudes of fault characteristic signals, the faults of the gear and the bearing can be diagnosed, respectively. The performance of the proposed method is validated by using the simulation data and the experiment vibration signals from a gearbox with compound faults.