Weakly supervised semantic segmentation using only image-level labels is critical since it alleviates the need for expensive pixel-level labels. Most cuttingedge methods adopt two-step solutions that learn to produce pseudo-ground-truth using only image-level labels and then train off-the-shelf fully supervised semantic segmentation network with these pseudo labels. Although these methods have made significant progress, they also increase the complexity of the model and training. In this paper, we propose a one-step approach for weakly supervised image semantic segmentation-attention guided enhancement network (AGEN), which produces pseudo-pixel-level labels under the supervision of image-level labels and trains the network to generate segmentation masks in an end-to-end manner. Particularly, we employ class activation maps (CAM) produced by different layers of the classification branch to guide the segmentation branch to learn spatial and semantic information. However, the CAM produced by the lower layer can capture the complete object region but with many noises. Thus, the self-attention module is proposed to enhance object regions adaptively and suppress irrelevant object regions, further boosting the segmentation performance. Experiments on the Pascal VOC 2012 dataset demonstrate that AGEN outperforms alternative state-of-the-art weakly supervised semantic segmentation methods exclusively relying on image-level labels.
With the development of artificial intelligence and deep learning technologies, image captioning has become an important research direction at the intersection of computer vision and natural language processing. The purpose of image captioning is to generate corresponding natural language descriptions by understanding the content of images. This technology has broad application prospects in fields such as image retrieval, autonomous driving, and visual question answering. Currently, many researchers have proposed region-based image captioning methods. These methods generate captions by extracting features from different regions of an image. However, they often rely on local features of the image and overlook the understanding of the overall scene, leading to captions that lack coherence and accuracy when dealing with complex scenes. Additionally, image captioning methods are unable to extract complete semantic information from visual data, which may lead to captions with biases and deficiencies. Due to these reasons, existing methods struggle to generate comprehensive and accurate captions. To fill this gap, we propose the Semantic Scenes Encoder (SSE) for image captioning. It first extracts a scene graph from the image and integrates it into the encoding of the image information. Then, it extracts a semantic graph from the captions and preserves semantic information through a learnable attention mechanism, which we refer to as the dictionary. During the generation of captions, it combines the encoded information of the image and the learned semantic information to generate complete and accurate captions. To verify the effectiveness of the SSE, we tested the model on the MSCOCO dataset. The experimental results show that the SSE improves the overall quality of the captions. The improvement in scores across multiple evaluation metrics further demonstrates that the SSE possesses significant advantages when processing identical images.
Beam search is a commonly used algorithm in image captioning to improve the accuracy and robustness of generated captions by finding the optimal word sequence. However, it mainly focuses on the highest-scoring sequence at each step, often overlooking the broader image context, which can lead to suboptimal results. Additionally, beam search tends to select similar words across sequences, causing repetitive and less diverse output. These limitations suggest that, while effective, beam search can be further improved to better capture the richness and variety needed for high-quality captions. To address these issues, this paper presents meshed context-aware beam search (MCBS). In MCBS for image captioning, the generated caption context is dynamically used to influence the image attention mechanism at each decoding step, ensuring that the model focuses on different regions of the image to produce more coherent and contextually appropriate captions. Furthermore, a penalty coefficient is introduced to discourage the generation of repeated words. Through extensive testing and ablation studies across various models, our results show that MCBS significantly enhances overall model performance.