The concrete industry is a core element of the building sector, but it has to deal with the increasing attention on the environmental issues related to the production process: increasing energy efficiency and the adoption of alternative fuels or raw materials represent the most relevant solutions. The present work analyses physical, mechanical, and environmental performances of concrete incorporating residues derived from four main sources (construction and demolition waste, residues from waste treatment, metallurgical industry by-products, and others), as substitutes of either fine or coarse aggregates. Fine aggregates showed the highest number of alternatives and replacement level, with the relevant impact on concrete properties; coarse aggregates, however, always reach a complete replacement, with the exclusion of glass that highly affects the mechanical performance. Construction and metallurgical industry categories are the main sources of alternative materials for both the components, with ceramic and lead slag reaching a full replacement for fine and coarse aggregates.