The impurity rate is one of the core indicators for evaluating the quality of rapeseed combine harvesters. It directly affects the economic value of rapeseed. At present, the impurity rate of rapeseed combine harvesters mainly relies on manual detection during shutdown, which cannot be monitored in real time. Due to the lack of accurate real-time impurity rate data, the operation parameters of rapeseed harvesters mainly depend on the driver’s experience, which results in large fluctuations in field harvest quality. In this research, a machine vision-based method of impurity detection for rapeseed harvesters, including an image acquisition device and impurity detection algorithm, was developed. The image acquisition device is equipped with a direct-current light source, a conveyor belt, and an industrial camera for taking real-time images of rapeseed samples. Based on the color and shape characteristics of impurity and rapeseed, the detection of rapeseed and impurity was achieved. A quantitative model for the rapeseed impurity rate was constructed to calculate the real-time impurity rate of machine-harvested rapeseed accurately. The field experiment showed that the average accuracy of the detection system for the impurity rate in rapeseed was 86.36% compared with the manual detection data. The impurity detection system proposed in this paper can swiftly and effectively identify rapeseed and impurity and accurately calculate the impurity rate, which can be applied to rapeseed harvesters to provide data support for the adjustment of operating parameters.