To solve the problem that traditional seed recognition methods are not completely suitable for needle-shaped seeds, such as Bidens L., in agricultural production, this paper proposes a model construction idea that combines the advantages of deep residual models in extracting high-level abstract features with multiscale feature extraction fusion, taking into account the depth and width of the network. Based on this, a multiscale feature fusion deep residual network (MSFF-ResNet) is proposed, and image segmentation is performed before classification. The image segmentation is performed by a popular semantic segmentation method, U²Net, which accurately separates seeds from the background. The multiscale feature fusion network is a deep residual model based on a residual network of 34 layers (ResNet34), and it contains a multiscale feature fusion module and an attention mechanism. The multiscale feature fusion module is designed to extract features of different scales of needle-shaped seeds, while the attention mechanism is used to improve the ability to select features of our model so that the model can pay more attention to the key features. The results show that the average accuracy and average F1-score of the multiscale feature fusion deep residual network on the test set are 93.81% and 94.44%, respectively, and the numbers of floating-point operations per second (FLOPs) and parameters are 5.95 G and 6.15 M, respectively. Compared to other deep residual networks, the multiscale feature fusion deep residual network achieves the highest classification accuracy. Therefore, the network proposed in this paper can classify needle-shaped seeds efficiently and provide a reference for seed recognition in agriculture.