Waste management is a vital environmental issue in the world today. Municipal solid wastes (MSWs) are discarded in huge quantities on a daily basis and need to be well controlled. Incineration is a common method for reducing the volume of these wastes, yet it produces ashes that require further assessment. Municipal solid waste incineration bottom ash (MSWI-BA) is the bulk byproduct of the incineration process and has the potential to be used in the construction sector. This paper offers a review of the use of MSWI-BA as aggregates in cementitious materials. With the growing demand of aggregates in cementitious materials, MSWI-BA is considered for use as a partial or full alternative. Although the physical and chemical properties of MSWI-BA are different than those of natural aggregates (NA) in terms of water absorption, density, and fineness, they can be treated by various methods to ensure suitable quality for construction purposes. These treatment methods are classified into thermal treatment, solidification and stabilization, and separation processes, where this review focuses on the techniques that reduce deficiencies limiting the use of MSWI-BA as aggregates in different ways. When replacing NA in cementitious materials, MSWI-BA causes a decrease in workability, density, and strength. Moreover, they cause an increase in water absorption, air porosity, and drying shrinkage. In general, the practicality of using MSWI-BA in cementitious materials is mainly influenced by its treatment method and the replacement level, and it is concluded that further research, especially on durability, is required before MSWI-BA can be efficiently used in the production of sustainable cementitious materials.