The interface between soil and concrete in cold climates has a significant effect on the structural integrity of embedded structures, including piles, liners, and others. In this study, a novel temperature control system was employed to conduct direct shear tests on this interface. The test conditions included normal stress (25 to 100 kPa), temperature (ranging from 20 to −6 °C), water content (from 10 to 19%), and shear rates (0.1 to 1.2 mm/min). Simultaneously, the deformation process of the interface was continuously photographed using a modified visual shear box, and the non-uniform deformation mechanism of the interface was analyzed by combining digital image correlation (DIC) technology with the photographic data. The findings revealed that the shear stress–shear displacement curves did not exhibit a discernible peak strength at elevated temperatures, indicating deformation behavior characterized by strain hardening. In the frozen state, however, the deformation softened, and the interfacial ice bonding strength exhibited a positive correlation with decreasing temperature. When the initial water content was 16% and the normal stress was 100 kPa, the peak shear strength increased significantly from 99.9 kPa to 182.9 kPa as the test temperature dropped from 20 °C to −6 °C. Both shear rate and temperature were found to have a marked effect on the peak shear strength, with interface cohesion being the principal factor contributing to this phenomenon. At a shear rate of 0.1 mm/min, the curve showed hardening characteristics, but at other shear rates, the curves exhibited strain-softening behavior, with the softening becoming more pronounced as shear rates increased and temperatures decreased. Due to the refreezing of interfacial ice, the residual shear strength increased in proportion to the reduction in shear rate. On a mesoscopic level, it was evident that the displacement of soil particles near the interface exhibited more pronounced changes. At lower shear rates, the phenomenon of interfacial refreezing became apparent, as evidenced by the periodic changes in interfacial granular displacement at the interface.