This paper proposes a novel fault diagnosis method for rolling bearing based on hierarchical refined composite multiscale fluctuation-based dispersion entropy (HRCMFDE) and particle swarm optimization-based extreme learning machine (PSO-ELM). First, HRCMFDE is used to extract fault features in the vibration signal at different time scales. By introducing the hierarchical theory algorithm into the vibration signal decomposition process, the problem of missing high-frequency signals in the coarse-grained process is solved. Fluctuation-based dispersion entropy (FDE) has the characteristics of insensitivity to noise interference and high computational efficiency based on the consideration of nonlinear time series fluctuations, which makes the extracted feature vectors more effective in describing the fault information embedded in each frequency band of the vibration signal. Then, PSO is used to optimize the input weights and hidden layer neuron thresholds of the ELM model to improve the fault identification capability of the ELM classifier. Finally, the performance of the proposed rolling bearing fault diagnosis method is verified and analyzed by using the CWRU dataset and MFPT dataset as experimental cases, respectively. The results show that the proposed method has high identification accuracy for the fault diagnosis of rolling bearings with varying loads and has a good load migration effect.