With the rapid development of urbanization, many new buildings are erected, and old ones are demolished and/or recycled. Thus, the reuse of building materials and improvements in reuse efficiency have become hot research topics. In recent years, scholars around the world have worked on improving recycle aggregates in concrete and broadening the scope of applications of recycled concrete. This paper reviews the findings of research on the effects of recycled fine aggregates (RFAs) on the permeability, drying shrinkage, carbonation, chloride ion penetration, acid resistance, and freeze–thaw resistance of concrete. The results show that the content of old mortar and the quality of recycled concrete are closely related to the durability of prepared RFA concrete. For example, the drying shrinkage value with a 100% RFA replacement rate is twice that of normal concrete, and the depth of carbonation increases by approximately 110%. Moreover, the durability of RFA concrete decreases as the RFA replacement rate and the water–cement ratio improve. Fortunately, the use of zeolite materials such as fly ash, silica fume, and meta kaolin as surface coatings for RFAs or as external admixtures for RFA concrete had a positive effect on durability. Furthermore, the proper mixing methods and/or recycled aggregates with optimized moisture content can further improve the durability of RFA concrete.