To improve the shape memory effect (SME) of 304 austenitic steel effectively and efficiently, thermomechanical cycling, comprising deformation at room temperature and annealing, was applied. The influences of cycle number and annealing temperature on the SME and microstructures in 304 austenitic steel were investigated by light microscope (LM), X-ray diffraction (XRD), and transmission electron microscope (TEM). The shape recovery ratio was remarkably improved from 16% to 40% after two thermomechanical cycles. The optimum annealing temperature was 833 K in the process of thermomechanical cycling. The improved SME by thermomechanical cycling was mainly related to stress-induced ε martensite rather than stress-induced α’ martensite. The reason is that thermomechanical cycling can not only promote the occurrence of the stress-induced γ→ε martensitic transformation, but also suppress the subsequently stress-induced ε→α′ transformation.