High-performance extruded aluminum alloys with complex textures suffer significant dimension variation under environmental temperature fluctuations, dramatically decreasing the precision of navigation systems. This research mainly focuses on the effect of the texture of extruded pure aluminum on its dimensional stability after various annealing processes. The result reveals that a significant increment in the area fraction of recrystallized grains with orientation and a decrement in the area fraction of grains with orientation were found with increasing annealing temperature. Moreover, with the annealing temperature increasing from 150 °C to 400 °C, the residual plastic strain after twelve thermal cycles with a temperature range of 120 °C was changed from −1.6 × 10⁻⁵ to −4.5 × 10⁻⁵. The large amount of equiaxed grains with orientation was formed in the microstructure of the extruded pure aluminum and the average grain size was decreased during thermal cycling. The area fraction of grain with crystallographic orientation of the sample annealed at 400 °C after thermal cycling was 30.9% higher than annealed at 350 °C (23.7%) or at 150 °C (18.7%). It is attributed to the increase in the proportion of recrystallization grains with direction as the annealing temperature increases, provided more nucleation sites for the formation of fine equiaxed grains with orientation. The main orientation of the texture was rotated from parallel to to parallel to after thermal cycling. The change in the orientation of grains contributed to a change in interplanar spacing, which explains the change in the dimension along the extrusion direction during thermal cycling.