11 August 2021 in IEEE Transactions on Multimedia
Existing multi-task models for object recognition and segmentation have verified the effectiveness of joint optimization of two semantic tasks. However, learning discriminative representations with insufficient training data and redundant contextual information from the background remains challenging. Semantic attributes are designed as powerful and informative mid-level features that 1) share information across categories to model the interclass correlation and that 2) can be localized in the object region to benefit foreground extraction. This paper introduces a novel attribute-aware feature encoding (AFE) module to a multi-task network for object recognition and segmentation with the aim of improving both semantic tasks by regularizing feature encoding with auxiliary attribute learning. Intuitively, attribute learning in our method not only provides extra supervision signals to capture interclass correlation in object classification but also refines the output of object segmentation via weakly supervised attribute localization. The experimental results on two public benchmarks show that our method yields remarkable improvement in both semantic tasks and auxiliary attribute estimation over existing methods.
28 May 2019 in IEEE Transactions on Multimedia
Intuitively, we can think of object recognition and attribute prediction as correlated tasks. However, they appeared to conflict in a simple two-branch multi-task framework (a category branch and an attribute branch) with a shared backbone part (convolutional layers and pooling layers). The performance dropped along with the iterative training steps. This result might have been caused by the noncoherent feature distribution between the object recognition features and the attribute prediction features. Recognition features are discriminative for different categories and are not sensitive to intracategory variations, while attribute prediction features are discriminative for different attributes, although these attributes can exist in objects from the same category. Thus, a conflict occurs when we force the network to learn the two kinds of distinct features simultaneously. To address this problem, we propose the CAP-net (Category and Attribute Prediction network), in which a category-constrained attribute prediction structure is introduced to transfer the object recognition knowledge and avoid the conflict between two features. The CAP-net parameters can be learned easily with a regularization method. Extensive experimental results show that the CAP-net outperforms the state-of-the-art methods on object recognition and attribute prediction tasks.
1 January 2021 in Wireless Communications and Mobile Computing
Existing attribute learning methods rely on predefined attributes, which require manual annotations. Due to the limitation of human experience, the predefined attributes are not capable enough of providing enough description. This paper proposes a self‐supervised attribute learning (SAL) method, which automatically generates attribute descriptions by differentially occluding the object region to deal with the above problems. The relationship between attributes is formulated with triplet loss functions and is utilized to supervise the CNN. Attribute learning is used as an auxiliary task of a multitask image classification and segmentation network, in which self‐supervision of attributes motivates the CNN to learn more discriminative features for the main semantic tasks. Experimental results on public benchmarks CUB‐2011 and Pascal VOC show that the proposed SAL‐Net can obtain more accurate classification and segmentation results without additional annotations. Moreover, the SAL‐Net is embedded into a multiobject recognition and segmentation system, which realizes instance‐aware semantic segmentation with the help of a region proposal algorithm and a fusion nonmaximum suppression algorithm.
6 December 2024 in Neurocomputing