The idea of sustainable development assumes that natural resources must be treated as limited goods and that waste must be managed rationally. This idea and the constant striving to reduce production costs make the use of waste materials as substitutes for traditionally used raw materials from non-renewable sources increasingly popular. In cement concrete technology, there are many possibilities to use waste as components of mortars and concretes. The subject of this paper is a fine-grained material, obtained as a by-product during the preparation of aggregate for mineral-asphalt mixtures. The aim of the research was to test the suitability of the selected type of powder, namely limestone powder, as a component of cement composites. The paper presents an evaluation of the potential of using the limestone powder as a substitute for the fine aggregate, focusing on the impact of such a modification on aspects of durability. The sulfate degradation and chloride ion diffusion in concrete were investigated. The overall desirability function has been determined. It was demonstrated that the satisfactory value of the general desirability can be attributed to most of the investigated concretes. Positive test results support the potential of replacing part of natural fine aggregate with the tested waste limestone dust without a negative impact on the durability of concrete.