Experiments were performed on four specimens of steel frames with infilled recycled aggregate concrete shear walls (SFIRACSWs), one specimen of infilled ordinary concrete wall, and one pure-steel frame were conducted under horizontal low cyclic loading. The influence of the composite forms of steel frames and RACSWs (namely, infilled cast-in-place and infilled prefabricated) on the failure modes, transfer mechanisms of lateral force, bearing capacity, and ductility of SFIRACSWs is discussed, and the concrete type and connecting stiffness of beam–column joints (BCJs) are also considered. Test results showed that infilled RACSWs can increase the bearing capacity and lateral stiffness of SFIRACSWs. The connecting stiffness of BCJs slightly influences the seismic behavior of SFIRACSWs. In the infilled cast-in-place RACSWs, the wall cracks mainly extended along the diagonal direction. The bearing capacity was 2.4 times higher than in the pure steel frame, the initial stiffness was 4.3 times higher, and the displacement ductility factors were 2.44–2.69 times higher. In the infilled prefabricated RACSWs, the wall cracks mainly extended along the connection between the embedded T-shape connectors and walls before finally connecting along the horizontal direction. Moreover, shear failure occurred in the specimens. The bearing capacity was 1.44 times higher than that of the pure steel frame, the initial stiffness was 2.8 times higher, and the displacement ductility factors were 3.32–3.40 times higher. The degradation coefficients of the bearing capacity were more than 0.85, indicating that the specimens demonstrated a high safety reserve.