Nonlinear friction, uncertainties and external disturbances always exist in two-axis opto-electronic tracking and measuring system, which reduces the control accuracy of the system. In this paper, a robust adaptive integral sliding mode control (RAISMC) strategy based on novel nonlinear extended state observer (NNESO) is proposed for high precision motion control of this nonlinear servo system. The proposed controller comprises improved nonlinear ESO, integral sliding mode control and robust adaptive control methods. By modifying the fal function to eliminate the discontinuity of ESO, a novel nonlinear observer is obtained to measure and estimate the disturbances, uncertainties and states of the system. The integral sliding mode control strategy is combined with the novel nonlinear observer and robust adaptive control to compensate (cancel) the effects of nonlinear friction, uncertainties and external disturbances. The stability of the opto-electronic tracking system is proved by Lyapunov theory. Comparative simulation and actual experiments are implemented on the opto-electronic tracking system, and the results verify the superiority and effectiveness of RAISM control scheme with NNESO, which is superior to the existing state-of-the-art method.