Rating prediction is an important technology in the personalized recommendation field. Prediction results are influenced by many factors, such as time, and their accuracy directly affects the quality of the recommendation. Current time-based collaborative filtering (CF) algorithms have improved the technology of prediction accuracy to a certain extent, but they fail to differentiate the time-sensitivity of different users, which further affects prediction accuracy. To address this issue, we have proposed a rating prediction algorithm based on user time-sensitivity differences. First, we analyzed and modeled the time sensitivities of users, utilized cosine distance and relative entropy to build a judgment function, and then judged the time sensitivities of users based on a voting strategy. Next, we applied the time-sensitivity difference to improve the traditional CF algorithm and optimized the combination of parameters. Finally, we tested our algorithm on standard datasets. The experimental results showed that there are many users who have different sensitivities to time. According to these experimental results, our proposed algorithm has achieved a higher prediction accuracy than other state-of-the-art algorithms.