Guava fruit is readily concealed by branches, making it difficult for picking robots to rapidly grip. For the robots to plan collision-free paths, it is crucial to segment branches and fruits. This study investigates a fast and accurate obstacle segmentation network for guava-harvesting robots. At first, to extract feature maps of different levels quickly, Mobilenetv2 is used as a backbone. Afterwards, a feature enhancement module is proposed to fuse multi-level features and recalibrate their channels. On the basis of this, a decoder module is developed, which strengthens the connection between each position in the feature maps using a self-attention network, and outputs a dense segmentation map. Experimental results show that in terms of the mean intersection over union, mean pixel accuracy, and frequency weighted intersection over union, the developed network is 1.83%, 1.60% and 0.43% higher than Mobilenetv2-deeplabv3+, and 3.77%, 2.43% and 1.70% higher than Mobilenetv2-PSPnet; our network achieved an inference speed of 45 frames per second and 35.7 billion floating-point operations per second. To sum up, this network can realize fast and accurate semantic segmentation of obstacles, and provide strong technical and theoretical support for picking robots to avoid obstacles.