There is unprecedented increase in low-temperature stress (LTS) during post-heading stages in rice as a consequence of the recent climate changes. Quantifying the effect of LTS on yields is key to unraveling the impact of climatic changes on crop production, and therefore developing corresponding mitigation strategies. The present research was conducted to analyze and quantify the effect of post-heading LTS on rice yields as well as yield and grain filling related parameters. A two-year experiment was conducted during rice growing season of 2018 and 2019 using two Japonica cultivars (Huaidao 5 and Nanjing 46) with different low-temperature sensitivities, at four daily minimum/maximum temperature regimes of 21/27 °C (T1), 17/23 °C (T2), 13/19 °C (T3) and 9/15 °C (T4). These temperature treatments were performed for 3 (D1), 6 (D2) or 9 days (D3), at both flowering and grain filling stages. We found LTS for 3 days had no significant effect on grain yield, even when the daily mean temperature was as low as 12 °C. However, LTS of between 6 and 9 days at flowering but not at filling stage significantly reduced grain yield of both cultivars. Comparatively, Huaidao 5 was more cold tolerant than Nanjing 46. LTS at flowering and grain filling stages significantly reduced both maximum and mean grain filling rates. Moreover, LTS prolonged the grain filling duration of both cultivars. Additionally, there was a strong correlation between yield loss and spikelet fertility, spikelet weight at maturity, grain filling duration as well as mean and maximum grain filling rates under post-heading LTS (p < 0.001). Moreover, the effect of post-heading LTS on rice yield can be well quantified by integrating the canopy temperature (CT) based accumulated cold degree days (ACDD_CT) with the response surface model. The findings of this research are useful in modeling rice productivity under LTS and for predicting rice productivity under future climates.