Due to its superior drought tolerance, high biomass yield, and stress resistance, sweet sorghum (Sorghum bicolor L.) has emerged as an ideal candidate for sustainable forage production in arid, semi-arid, and mildly saline–alkaline regions. This study aimed to evaluate the effects of replacing corn silage (CS) with either forage sorghum silage (FSS) or sugar sweet sorghum silage (SSS) on goat meat quality, the rumen microbial community, and meat composition. Thirty 3-month-old Boer goats (average body weight: 13.44 ± 1.67 kg) were randomly assigned to five dietary treatments; the control group contained 50% corn silage (CON), while the experimental groups contained 50% FSS (group I), 70% FSS (group II), 50% SSS (group III), or 70% SSS (group V), with each group receiving the same concentrate diet but different roughage sources. The results showed that compared to the CON group (50% CS), the experimental groups had a significantly increased average daily weight gain (ADG) (p < 0.05) and slaughter rate (SR). It is noteworthy that group III (50% SSS) showed a significant increase of 12.4% in SR (p < 0.01). Analysis of the silage characteristics and changes in the rumen microbial community revealed that feeding SSS and FSS increased the relative abundance of Ruminococcus in the rumen, further enhanced the degradation and conversion of silage neutral detergent fiber (NDF), and promoted the synthesis of fatty acids and amino acids. Specifically, FSS significantly increased the amino acid content in the meat, while SSS effectively improved the crude protein (CP) and crude fat (CF) contents. In conclusion, replacing 50% CS with FSS or SSS can effectively improve the meat quality and growth performance of Boer goats.