The concrete industry increasingly seeks sustainable alternatives to conventional materials to reduce the environmental impact while maintaining structural performance. This study evaluates the use of locally sourced rhyolite as a coarse aggregate combined with recycled supplementary cementitious materials (SCMs) to address the sustainability and durability. Due to its high silica content, rhyolite is prone to the alkali–silica reaction (ASR), which may affect concrete durability. Concrete mixtures incorporating rhyolite with silica fume (SF), Class F fly ash (FA), and slag cement (SC) were tested for compressive strength, porosity, density, absorption, mortar bar expansion, electrical resistivity, and rapid chloride permeability. All rhyolite-based mixtures—regardless of SCM incorporation—achieved higher 90-day compressive strengths than the conventional control mixture, with 10% SF reaching the highest value. Additionally, each recycled SCM effectively reduced ASR-induced expansion, with 20% FA showing the most significant reduction and superior durability, including the greatest decrease in chloride permeability and the highest electrical resistivity, indicating enhanced corrosion resistance. These results confirm that rhyolite aggregates, when combined with SCMs, can improve durability and reduce ASR. Therefore, rhyolite shows potential for use in structural concrete under standard exposure conditions. This strategy supports circular economy goals by incorporating regional and recycled materials to develop concrete with improved durability characteristics.