RIS-aided CF massive MIMO requires joint AP precoding and RIS phase control. MADDPG is scalable but its critic becomes numerically fragile at large γ. We propose reward-centering MADDPG (RC-MADDPG), which removes the dominant state-independent offset in the discounted value. In the present quasi-static block-fading model, γ is treated as an algorithmic horizon parameter for repeated block-wise control. RC-MADDPG improves spectral efficiency by 10.7% over MADDPG at γ = 0.99, remains stable at γ = 0.999, and outperforms both reward-scaled and PopArt-MADDPG baselines in the high-γ regime.