Transformer-based semantic segmentation methods have achieved excellent performance in recent years. Mask2Former is one of the well-known transformer-based methods which unifies common image segmentation into a universal model. However, it performs relatively poorly in obtaining local features and segmenting small objects due to relying heavily on transformers. To this end, we propose a simple yet effective architecture that introduces auxiliary branches to Mask2Former during training to capture dense local features on the encoder side. The obtained features help improve the performance of learning local information and segmenting small objects. Since the proposed auxiliary convolution layers are required only for training and can be removed during inference, the performance gain can be obtained without additional computation at inference. Experimental results show that our model can achieve state-of-the-art performance (57.6% mIoU) on the ADE20K and (84.8% mIoU) on the Cityscapes datasets.

Transformer-based semantic segmentation has been applied to various visual recognition applications and achieved outstanding performance in recent years. Since most of these approaches adopt a pretrained backbone and finetune it for semantic segmentation, they are not efficient in capturing semantic contextual information during the encoding stage, leading to sub-optimal segmentation performance. To address this problem, SeMask proposes a semantic attention operation to incorporate the semantic contextual information of an image during the encoding stage and improves the segmentation performance. However, the architecture of SeMask is entirely based on the attention mechanisms of Transformers and has some limitations to fully exploit the local details, which are important for more accurate segmentation. In this paper, we introduce a novel semantic layer into the encoder side of a Transformer-based segmentation model. The proposed semantic layer consists of depthwise convolutions with different kernel sizes to capture multi-scale local details. It is integrated at different stages of a hierarchical Transformer backbone to acquire multi-scale semantic contextual information on the encoder side to improve the overall segmentation performance, especially for more accurate segmentation of small objects. Our proposed method can be integrated with common segmentation models such as Semantic-FPN and Mask Transformers. Experimental results show that our proposed method can achieve state-of-the-art performance on the ADE20K dataset with 58.24% mIoU and the Cityscapes dataset with 84.97% mIoU.

Transformer-based models have achieved impressive performance in semantic segmentation in recent years. However, the multi-head self-attention mechanism in Transformers incurs significant computational overhead and becomes impractical for real-time applications due to its high complexity and large latency. Numerous attention variants have been proposed to address this issue, yet the overall performance and inference speed still have limitations. In this paper, we propose an efficient multi-scale Transformer (EMSFormer) that employs learnable keys and values based on the single-head attention mechanism and a dual-resolution structure for real-time semantic segmentation. Specifically, we propose multi-scale single-head attention (MS-SHA) to effectively learn multi-scale attention and improve feature representation capability. In addition, we introduce cross-resolution single-head attention (CR-SHA) to efficiently combine the global context-rich features in the low-resolution branch to the features in the high-resolution branch. Experimental results show that our proposed method can achieve state-of-the-art performance with real-time inference speed on ADE20K, Cityscapes, and CamVid datasets.

Deploying deep neural networks on edge devices such as embedded GPUs and FPGAs remains challenging due to high computational cost, memory footprint, and sensitivity to class imbalance. Existing compression techniques, including knowledge distillation, often overlook hardware constraints and minority-class performance, leading to suboptimal real-world deployment. This article introduces CB-DistillGrad, a unified hardware-aware distillation framework that jointly optimizes accuracy, model compactness, and deployment efficiency. The proposed system integrates: (1) a lightweight student architecture built with depthwise separable convolutions, squeeze-and-excitation attention, progressive dropout, and hardware-friendly Hard Swish activation; (2) a CB-Distill Loss that couples class-balanced focal loss with reweighted Kullback–Leibler divergence to mitigate teacher bias and improve minority-class accuracy; and (3) gradient conflict resolution via PCGrad to stabilize optimization under multi-objective supervision. The resulting student network achieves 87.11% overall accuracy on the RSCD road-surface dataset using only 130K parameters (0.52 MB at FP32), a 96.8% reduction compared to the 4.1M-parameter teacher. On the NVIDIA GPU, inference occurs at 1.05 ms/image (949 FPS) with 72Wpower, 2.15× faster than the teacher. On the NVIDIA Jetson AGX Orin, inference occurs at 11.70 ms/image (85 FPS) with 7.36 W power, achieving 2.8× speedup and 4.5× energy reduction. On the Xilinx ZCU104 FPGA, inference occurs at 62.19 ms/image (16 FPS) with 4.42Wpower, demonstrating a 27× energy reduction compared to GPU execution. These results demonstrate a practical framework for training and deploying compact, class-balanced, and hardware-optimized deep learning models on resource-constrained platforms.

Common video-based object detectors exploit temporal contextual information to improve the performance of object detection. However, detecting objects under challenging conditions has not been thoroughly studied yet. In this paper, we focus on improving the detection performance for challenging events such as aspect ratio change, occlusion, or large motion. To this end, we propose a video object detection network using event-aware ConvLSTM and object relation networks. Our proposed event-aware ConvLSTM is able to highlight the area where those challenging events take place. Compared with traditional ConvLSTM, with the proposed method it is easier to exploit temporal contextual information to support video-based object detectors under challenging events. To further improve the detection performance, an object relation module using supporting frame selection is applied to enhance the pooled features for target ROI. It effectively selects the features of the same object from one of the reference frames rather than all of them. Experimental results on ImageNet VID dataset show that the proposed method achieves mAP of 81.0% without any post processing and can handle challenging events efficiently in video object detection.

With the rapid development of deep learning techniques, the performance of object detection has increased significantly. Recently, several approaches on joint learning of object detection and semantic segmentation have been proposed to exploit the complementary benefits of the two highly correlated tasks. In this work, we propose a weakly-annotated auxiliary multi-label segmentation network that boosts object detection performance without additional computational cost at inference. The proposed auxiliary segmentation network is trained using weakly-annotated dataset and therefore does not require expensive pixel-level annotations for training. Different from the previous approaches, we use multi-label segmentation to jointly supervise auxiliary segmentation and object detection for better occlusion handling. The proposed method can be integrated with any one-stage object detector such as RetinaNet, YOLOv3, YOLOv4, or SSD. Our experimental results on the MS COCO dataset show that the proposed method can improve the performance of popular one-stage object detectors without slowing down the inference speed regardless of the sub-optimal training sample selection schemes.