Irradiation embrittlement occurs in the cladding materials of fusion reactors during irradiation. Determining the ductile–brittle transition temperature via Charpy impact testing is the primary method for evaluating irradiation embrittlement. Standard-sized V-shaped Charpy impact specimens (CVN) are too large in size and have high induced radioactivity. Small-sized specimens (KLST) can solve these problems, but the performance data measured from small-sized specimens are different from those of standard specimens. In other words, there is a size effect in impact performance. The notch size and hammer impact speed of KLST specimens are different from those of CVN specimens. The influence of these factors on impact performance requires further study. In response to these issues, on the basis of the previous experiments conducted by the research group, GTN damage models of CVN specimens and KLST specimens are constructed using the inverse operation method. Numerical simulation of the impact on the upper platform area is carried out for KLST specimens and variable-sized KLST specimens. Compared with the test results, the numerical simulation results are in good agreement, verifying the accuracy and reliability of the model. The results show that the notch angle and radius have little influence on the plastic zone. The cross-sectional area of the notch has a significant impact on the plastic zone. The impact velocity within the range of 3.8 m/s to 5.24 m/s affects the impact response process, but does not affect the load–displacement curve, the length of the non-plastic deformation zone, or the volume of the plastic zone.