In this study, the effect of finish rolling temperature on the critical crack tip opening displacement (CTOD) of typical 500 MPa grade weathering steel was elucidated. The microstructures were observed via optical microscope (OM), scanning electron microscope (SEM), transmission electron microscope (TEM), and electron back-scattered diffraction (EBSD). The cryogenic fracture toughness and microstructures of steels were analyzed at different finish rolling temperatures (780–840 °C). The results show that a mixed microstructure, i.e., granular bainitic ferrite (GBF), polygonal ferrite (PF), and martensite/austenite (M/A), constituent was formed in each sample. With the decrease of the finish rolling temperature, the GBF content decreased, PF content increased, and the high angle grain boundary (HAGB) number fraction of the matrix increased. Furthermore, the fraction of M/A constituents was increased with reduced average size. The value of CTOD increased significantly from 0.28 to 1.12 mm as the finish rolling temperature decreased from 840 to 780 °C. Both the decrease of M/A constituents and the increase of HAGB increased the cryogenic (−40 °C) fracture toughness of the typical 500 MPa grade weathering steel.