Tree species in relatives refer to species belonging to the same genus with high morphological similarity and small botanical differences, making it difficult to perform classification and usually requiring manual identification by experts. To reduce labor costs and achieve accurate species identification, we conducted research on the image classification of tree species in relatives based on deep learning and proposed a dual-branch feature fusion Vision Transformer model. This model is designed with a dual-branch architecture and two effective blocks, a Residual Cross-Attention Transformer Block and a Multi-level Feature Fusion method, to enhance the influence of shallow network features on the final classification and enable the model to capture both overall image information and detailed features. Finally, we conducted ablation studies and comparative experiments to validate the effectiveness of the model, achieving an accuracy of 90% on the tree relatives dataset.