The existing fault diagnosis methods of rotating machinery constructed with both shallow learning and deep learning models are mostly based on vibration analysis under steady rotating speed. However, the rotating speed frequently changes to meet practical engineering needs. The shallow learning models largely depend on domain experience of feature extraction, and training a deep learning model requires large samples and a long time. In addition, vibration monitoring has the shortcomings of contact measurement, small coverage, and noise interference. To address these problems, this article proposes a new fault diagnosis method with the least square interactive support matrix machine (LSISMM) and infrared thermal images. In this method, a novel matrix-form classifier called LSISMM is constructed under the concept of nonparallel interactive hyperplanes to fully leverage the structure information of infrared thermal images. To improve the computation efficiency, a new least square loss constraint is designed for LSISMM. Besides, we derive an effective solution framework based on the alternating direction method of the multiplier (ADMM) framework. The constructed LSISMM is directly used to analyze the collected thermal images of rotating machinery under time-varying speeds. Experiment results demonstrate that the proposed method is superior to state-of-the-art methods in terms of diagnosis accuracy and efficiency, especially under small thermal image samples.