Granite sawdust is a by-product in the process of stone processing, which is usually piled up, thus easily causing environmental pollution. To achieve resource utilization, granite sawdust was used as a partial substitution of cement in this work. The effects of different sawdust contents (10–50%) were systematically studied on the pore structure and the mechanical properties of its dry powder mortar. Combined with the grey correlation theory, the correlation between pore size distribution and compressive strength was analyzed. The results showed that the consistency and mechanical properties of the mortar gradually decreased along with the increasing sawdust content, while its critical pore-diameter decreased. The mortar performance was the best when its sawdust content is 10%, which meets the M25 technical requirements. When content reaches up to 30%, the mortar still met the strength standard of M20. Compared to fly ash, the mortar with 30% sawdust as the substitution has a higher water retention rate but lower mechanical strength. The grey correlation analysis indicated that the pores with diameters less than 10 nm and greater than 1000 nm had the most significant impact on the compressive strength.