Rice grain yield and quality are predicted to be highly vulnerable to global warming. Five genotypes including heat-tolerant and susceptible checks, a heat-tolerant near-isogenic line and two hybrids were exposed to control (31 °C/23 °C, day/night), high night-time temperature (HNT; 31 °C/30 °C), high day-time temperature (HDT; 38 °C/23 °C) and high day- and night-time temperature (HNDT; 38 °C/30 °C) treatments for 20 consecutive days during the grain-filling stage. Grain-filling dynamics, starch metabolism enzymes, temporal starch accumulation patterns and the process of chalk formation were quantified. Compensation between the rate and duration of grain filling minimized the impact of HNT, but irreversible impacts on seed-set, grain filling and ultimately grain weight were recorded with HDT and HNDT. Scanning electron microscopy demonstrated irregular and smaller starch granule formation affecting amyloplast build-up with HDT and HNDT, while a quicker but normal amylopast build-up was recorded with HNT. Our findings revealed temporal variation in the starch metabolism enzymes in all three stress treatments. Changes in the enzymatic activity did not derail starch accumulation under HNT when assimilates were sufficiently available, while both sucrose supply and the conversion of sucrose into starch were affected by HDT and HNDT. The findings indicate differential mechanisms leading to high day and high night temperature stress-induced loss in yield and quality. Additional genetic improvement is needed to sustain rice productivity and quality under future climates.