The current image registration models have problems such as low feature point matching accuracy, high memory consumption, and significant computational complexity in heterogeneous image registration, especially in complex environments. In this context, significant differences in lighting and leaf occlusion in orchards can result in inaccurate feature extraction during heterogeneous image registration. To address these issues, this study proposes an AD-ResSug model for heterogeneous image registration. First, a VGG16 network was included as the encoder in the feature point encoder system, and the positional encoding was embedded into the network. This enabled us to better understand the spatial relationships between feature points. The addition of residual structures to the feature point encoder aimed to solve the gradient diffusion problem and enhance the flexibility and scalability of the architecture. Then, we used the Sinkhorn AutoDiff algorithm to iteratively optimize and solve the optimal transmission problem, achieving optimal matching between feature points. Finally, we carried out network pruning and compression operations to minimize parameters and computation cost while maintaining the model’s performance. This new AD-ResSug model uses evaluation indicators such as peak signal-to-noise ratio and root mean square error as well as registration efficiency. The proposed method achieved robust and efficient registration performance, verified through experimental results and quantitative comparisons of processing color with ToF images captured using heterogeneous cameras in natural apple orchards.