Fault diagnosis of rope tension is significantly important for hoisting safety, especially in mine hoists. Conventional diagnosis methods based on force sensors face some challenges regarding sensor installation, data transmission, safety, and reliability in harsh mine environments. In this paper, a novel fault diagnosis method for rope tension based on the vibration signals of head sheaves is proposed. First, the vibration signal is decomposed into some intrinsic mode functions (IMFs) by the ensemble empirical mode decomposition (EEMD) method. Second, a sensitivity index is proposed to extract the main IMFs, then the de-noised signal is obtained by the sum of the main IMFs. Third, the energy and the proposed improved permutation entropy (IPE) values of the main IMFs and the de-noised signal are calculated to create the feature vectors. The IPE is proposed to improve the PE by adding the amplitude information, and it proved to be more sensitive in simulations of impulse detecting and signal segmentation. Fourth, vibration samples in different tension states are used to train a particle swarm optimization–support vector machine (PSO-SVM) model. Lastly, the trained model is implemented to detect tension faults in practice. Two experimental results validated the effectiveness of the proposed method to detect tension faults, such as overload, underload, and imbalance, in both single-rope and multi-rope hoists. This study provides a new perspective for detecting tension faults in hoisting systems.