Calcium carbonate is wildly used in cementitious composites at different scales and can affect the properties of cementitious composites through physical effects (such as the filler effect, dilution effect and nucleation effect) and chemical effects. The effects of macro (>1 mm)-, micro (1 μm–1 mm)- and nano (<1 μm)-sizes of calcium carbonate on the hydration process, workability, mechanical properties and durability are reviewed. Macro-calcium carbonate mainly acts as an inert filler and can be involved in building the skeletons of hardened cementitious composites to provide part of the strength. Micro-calcium carbonate not only fills the voids between cement grains, but also accelerates the hydration process and affects the workability, mechanical properties and durability through the dilution, nucleation and even chemical effects. Nano-calcium carbonate also has both physical and chemical effects on the properties of cementitious composites, and these effects behave even more effectively than those of micro-calcium carbonate. However, agglomeration of nano-calcium carbonate reduces its enhancement effects remarkably.