The rail fastening system is an important part of a high-speed railway track. It is always critical to the operational safety and comfort of railway vehicles. Therefore, the condition detection of the rail fastening system, looseness or absence, is an important task in railway maintenance. However, the vision-based method cannot identify the severity of rail fastener looseness. In this paper, the condition of rail fastening system is monitored based on an automatic and remote-sensing measurement system. Meanwhile, wavelet packet analysis is used to analyze the acceleration signals, based on which two damage indices are developed to locate the damage position and evaluate the severity of rail fasteners looseness, respectively. To verify the effectiveness of the proposed method, an experiment is performed on a high-speed railway experimental platform. The experimental results show that the proposed method is effective to assess the condition of the rail fastening system. The monitoring system significantly reduces the inspection time and increases the efficiency of maintenance management.