To address the challenge of recognizing sheep faces of the same type, which exhibit significant similarities and varying performance of RGB images under different lighting conditions and angles, this paper proposes a dual-branch multi-modal sheep face recognition model based on the ResNet18 architecture. This model effectively learns geometric features from depth data and texture features from RGB data, thereby enhancing recognition accuracy. Initially, the model employs two InceptionV2 layers, one for the RGB channel and another for the depth channel, to extract specific features from both modalities. Subsequently, the losses from the two modalities are computed. In the mid-stage, the two modalities are fused using the Convolutional Block Attention Module (CBAM), and in the final stage, a residual network is utilized to learn the complementary features between the modalities. Experimental results demonstrate that this model benefits from effective multi-modal fusion, achieving high accuracy in sheep face recognition under complex lighting conditions and various angles.