In this paper, a series of shear specimens with or without groove were manufactured to mainly analyze the effects of grooves (or shear section height) and steel fibers on the shear properties of concrete with recycled coarse aggregate through double-side direct shear test. In addition, the relationship between the shear strength and the compressive strength and splitting tensile strength of steel fiber reinforced concrete with recycled coarse aggregate (SFRCAC) was also discussed. The experimental results showed that the peak load, deformation corresponding to the peak load and calculated shear strength of the specimens with grooves were lower than those of the specimens without grooves. The steel fiber and recycled coarse aggregate (RCA) had a significant effect on the shear properties of SFRCAC. As the volume content of steel fibers increased, the shear strength of SFRCAC and the corresponding deformation increased gradually. With the replacement ratio of RCA increasing, the shear strength of SFRCAC decreased but the corresponding deformation increased gradually. Finally, the formula for calculating the shear strength of SFRCAC was proposed by analyzing and fitting the test results and the data of related literature.