In this paper, we propose a tree trunk and obstacle detection method in a semistructured apple orchard environment based on improved YOLOv5s, with an aim to improve the real-time detection performance. The improvement includes using the K-means clustering algorithm to calculate anchor frame and adding the Squeeze-and-Excitation module and 10% pruning operation to ensure both detection accuracy and speed. Images of apple orchards in different seasons and under different light conditions are collected to better simulate the actual operating environment. The Gradient-weighted Class Activation Map technology is used to visualize the performance of YOLOv5s network with and without improvement to increase interpretability of improved network on detection accuracy. The detected tree trunk can then be used to calculate the traveling route of an orchard carrier platform, where the centroid coordinates of the identified trunk anchor are fitted by the least square method to obtain the endpoint of the next time traveling rout. The mean average precision values of the proposed model in spring, summer, autumn, and winter were 95.61%, 98.37%, 96.53%, and 89.61%, respectively. The model size of the improved model is reduced by 13.6 MB, and the accuracy and average accuracy on the test set are increased by 5.60% and 1.30%, respectively. The average detection time is 33 ms, which meets the requirements of real-time detection of an orchard carrier platform.