This paper proposes a visual servo scheme for hoisting positioning under disturbance conditions. In actual hoisting work, disturbances such as equipment and load vibration are inevitable, which brings challenges to the development of a visual servo for hoisting positioning. The main problems are as follows: (1) the correlation between visual error and disturbance is not considered or well resolved; (2) the disturbance has a great influence on the control stability, but it is difficult to model. At present, there is no detailed research on the above problems. In this paper, the visual error is defined by the image error of the feedback signal based on dynamic equations containing disturbances. An adaptive sliding mode control algorithm is employed to decrease the influence of external disturbance, and the coefficient of the slide surface is established based on the adaptive gain. In view of the belief that it is difficult to model disturbance terms, a nonlinear disturbance observer is introduced to obtain equivalent disturbance. On this basis, an adaptive control algorithm with disturbance compensation is proposed to improve the robustness of the visual servo system. We use Lyapunov’s method to analyze the stability conditions of the system. Compared with the other state-of-the-art methods, the simulation results show that our method has superior performance in convergence, accuracy, and restraining disturbance. Finally, the proposed algorithm is applied to the hoisting platform for experimental research, which proves the effectiveness of the controller.