Transformer is a powerful model widely used in artificial intelligence applications. It contains complex structures and has extremely high computational requirements that are not suitable for embedded intelligent sensors with limited computational resources. The binary quantization technology takes up less memory space and has a faster calculation speed; however, it is seldom studied for the lightweight transformer. Compared with full-precision networks, the key bottleneck lies in the distribution shift problem caused by the existing binary quantization methods. To tackle this problem, the feature distribution alignment operation in binarization is investigated. The median shift and mean restore is designed to ensure consistency between the binary feature distribution and the full-precision transformer. Then, a knowledge distillation architecture for distribution correction is developed, which has a teacher–student structure comprising a full-precision and binary transformer, to further rectify the feature distribution of the binary student network to ensure the completeness and accuracy of the data. Experimental results on the CIFAR10, CIFAR100, ImageNet-1k, and TinyImageNet datasets show the effectiveness of the proposed binary optimization model, which outperforms the previous state-of-the-art binarization mechanisms while maintaining the same computational complexity.
15 July 2025 in CAAI Transactions on Intelligence Technology
In recent years, the transformer model has demonstrated excellent performance in computer vision (CV) applications. The key lies in its guided representation attention mechanism, which uses dot‐product to depict complex feature relationships, and comprehensively understands the context semantics to obtain feature weights. Then feature enhancement is implemented by guiding the target matrix through feature weights. However, the uncertainty and inconsistency of features are widespread that prone to confusion in the description of relationships within dot‐product attention mechanisms. To solve this problem, this paper proposed a novel approximate‐guided representation learning methodology for vision transformer. The kernelised matroids fuzzy rough set is defined, wherein the closed sets inside kernelised fuzzy information granules of matroids structures can constitute the subspace of lower approximation in rough sets. Thus, the kernel relation is employed to characterise image feature granules that will be reconstructed according to the independent set in matroids theory. Then, according to the characteristics of the closed set within matroids, the feature attention weight is formed by using the lower approximation to realise the approximate guidance of features. The approximate‐guided representation mechanism can be flexibly deployed as a plug‐and‐play component in a wide range of CV tasks. Extensive empirical results demonstrate that the proposed method outperforms the majority of advanced prevalent models, especially in terms of robustness.