In the field of semantic segmentation, the limited receptive field of convolutional neural networks leads to insufficient extraction of global features, thereby affecting the accuracy of network segmentation. To address this issue, a Hierarchical Hybrid Encoder Network (HHEnet) based on Transformers is proposed for semantic segmentation. Firstly, to solve the problem of limited global feature information caused by the network’s limited receptive field, a Hierarchical Hybrid Encoder (HHE) is introduced, which consists of a Hierarchical Convolutional Encoder (HCE) and a Hierarchical Transformer Encoder (HTE). The encoder combines the advantages of convolution and transformers, allowing for effective extraction of both shallow and deep features. In order to further enhance spatial and global semantic information, the Feature Enhancement Module (FEM) was introduced, which consisted of two feature enhancement modules: spatial feature enhancement module (SEM) and global feature enhancement module (GEM), which enhanced spatial detail information and global semantic information respectively. Thus the accuracy of semantic segmentation can be improved. Finally, to alleviate the discrepancy between the features of the convolutional encoder and the transformer encoder, a Feature Guidance Module (FGM) is introduced. Experimental results conducted on Cityscapes, ADE20K and PASCAL VOC2012 datasets achieved mIoU scores of up to 81.9%, 49.4% and 79.1%, respectively. Compared to state-of-the-art networks, the research results confirm the higher segmentation accuracy of the proposed HHEnet in this study.

Recently, owing to the requirements of inference speed, most real-time semantic segmentation networks often have shallow network depth, which limits the receptive field size of the model, leading to the limited acquisition of semantic information and resulting in intraclass inconsistency and ultimately a decrease in segmentation accuracy. Additionally, the shallow network depth also restricts the feature extraction capability of the network, reducing its robustness and ability to adapt to complex scenes. To address these issues, a bilateral network with a rich semantic extractor (RSE) for real-time semantic segmentation (BRSeNet) is presented to perform real-time semantic segmentation. First, to solve the problem of insufficient semantic feature information extraction, an RSE is proposed, which includes a multiscale global semantic extraction module (MGSEM) and a semantic fusion module (SFM). The MGSEM can extract rich global semantics and expand the effective receptive field. Simultaneously, the SFM efficiently integrates multiscale local semantics with multiscale global semantics, resulting in more comprehensive semantic information for the network. Finally, based on the characteristics of detail and semantic branches, a bilateral reconstruction aggregation module is designed to reconstruct the contextual information of detail features, model the interdependencies on semantic feature channels, and enhance feature representation. Comprehensive experiments on the challenging Cityscapes and ADE20K datasets are conducted. The experimental results show that the proposed BRSeNet achieves mean intersection over union of 74.9% and 35.7% at inference speeds of 74 and 65 frames per second, respectively, and ensures a favorable balance between segmentation accuracy and inference speed.