Textile sludge has complex components and certain toxicity, which is in urgent need of resource treatment. The effect of textile sludge replacing cement and aggregates on the properties of polypropylene fiber concrete has been investigated by testing the compressive strength, drying shrinkage, heavy metal leaching concentration, micro morphology, and nanomechanical properties. The results show that the utilization of 10% textile sludge replacing cement increases the later strengths of concrete and decreases the drying shrinkage due to its denser microstructure. With the further content increase of textile sludge replacing cement, the strengths of concrete are reduced and the drying shrinkage is increased. The utilization of textile sludge replacing aggregates increases the compressive strengths of concrete and the drying shrinkage at every age, and among them, the concrete with 15% textile sludge replacing aggregates shows the highest compressive strengths, and the drying shrinkage of concrete increases with the content increase of textile sludge replacing aggregates. The concrete with textile sludge is a good solidification with heavy metal ions. The utilization of 10% textile sludge replacing cement improves the microstructure of concrete and helps to produce more high-density calcium silicate hydrate and reduces the thickness of the interfacial transition zone.