The cleaning device is an important part of combine harvesters, as its superior or inferior performance directly affects the performance of the combine harvester greatly. With an increasing rice yield, the current single-duct cleaning performance declines greatly, and causes a large grain sieve loss level and a direct grain loss for the farms. To optimize the existing single-duct cleaning device to meet the large feeding rate requirement, firstly, the terminal velocity of rice grain and MOG (material other than grain) for different varieties was experimentally measured by the custom-made device. The effects of the moisture content of rice grains and the length of short straws on terminal velocity were studied in detail. Then, the gas–solid two-phase flow theory was comprehensively applied by utilizing the discrete element method (DEM) and computational fluid dynamics (CFD) to study the working mechanism of the existing single-duct cleaning unit, and the cleaning performance was evaluated from the view of the motion law of the threshing output within the cleaning shoe. At last, a multi-duct cleaning device was put forward, and a field experiment was performed to assess the performance of the newly developed cleaning device. The results showed that the grain sieve losses ratio and grain impurity ratio improved dramatically, proving that structure optimization of the cleaning device was feasible.