Railway inspection is an important task in railway maintenance to ensure safety. The fastener is a major part of the railway which fastens the tracks to the ground. The current article presents an efficient method to detect fasteners on the basis of image processing and pattern recognition techniques, which can be used to detect the absence of fasteners on the corresponding track in high-speed(up to 400 km/h). The Direction Field is extracted as the feature descriptor for recognition. In addition, the appropriate weight coefficient matrix is presented for robust and rapid matching in a complex environment. Experimental results are presented to show that the proposed method is computation efficient and robust for the detection of fasteners in a complex environment. Through the practical device fixed on the track inspection train, enough fastener samples are obtained, and the feasibility of the method is verified at 400 km/h.