For semantic segmentation tasks, there are problems in using multi-branch structures to enrich feature maps and aggregate different branches of feature maps in a certain network depth, such as the insufficient richness of feature maps and the incomplete aggregation of feature maps. Given multi-branch feature maps and branch aggregation, this study proposes a lightweight method, called multi-branch aggregation atrous spatial pyramidal pooling, by introducing an attention mechanism and CARAFE to enrich the features of the feature maps, giving the feature maps adaptive parameters, periodically adjusting the adaptive parameters, and aggregating the feature maps in both the vertical and horizontal directions. First, the atrous pyramid is retained and the attention mechanism and CARAFE are used to handle the pooling features to obtain 10 different feature maps. Secondly, giving each feature map a cascaded adaptive parameter and periodically adjusting the adaptive parameter to promote or suppress certain feature maps prevents the model from being at a local minimum or saddle point for long periods due to aleatory uncertainty. Finally, the feature maps are vertically aggregated, horizontally aggregated, and weighted for summation. This work demonstrates a competitive performance on the benchmark dataset, with an improvement of 1.88% in MPA, 0.66% in FWIoU, and 1.29% in MIoU compared to atrous spatial pyramid pooling. The benchmark datasets include PASCAL VOC 2012 and CIFAR-100.