Developing prefabricated buildings (PBs) and optimizing the construction supply chain represent effective strategies for reducing carbon emissions in the construction industry. Prefabricated building supply chain (PBSC) carbon reduction suffers from synergistic difficulties, limited rationality, and environmental complexity. Therefore, investigating carbon emission reduction in PBSC is essential. In this study, PBSC participants are divided into four categories according to the operation process. Gaussian white noise is introduced to simulate the random perturbation factors, and a four-way stochastic evolutionary game model is constructed and numerically simulated. The study found the following: Stochastic perturbation factors play a prominent role in the evolution speed of the agent; the emission reduction benefit and cost of the participant significantly affect the strategy selection; the operation status of the PBSC is the key to strategy selection, and it is important to pay attention to the synergy of the participants at the first and the last end of the PBSC; the influence of the external environment on strategies is mainly manifested in the loss caused and the assistance provided; and the information on emission reduction is an important factor influencing strategies. Finally, we provide suggestions for promoting carbon emission reduction by participants in the PBSC from the perspective of resisting stochastic perturbation, enhancing participants’ ability, and strengthening PBSC management; external punishment and establishing a cross-industry information sharing platform is more important than the reward.