The fusion of multi-polarized petrographic images of rock thin sections involves the fusion of feature information from microscopic images of rock thin sections illuminated under both plane-polarized and orthogonal-polarized light. During the fusion process of rock thin section images, the inherent high resolution and abundant feature information of the images pose substantial challenges in terms of computational complexity when dealing with massive datasets. In engineering applications, to ensure the quality of image fusion while meeting the practical requirements for high-speed processing, this paper proposes a novel fast fusion Transformer. The model leverages a soft matching algorithm based on intuitionistic fuzzy sets to merge redundant tokens, effectively mitigating the negative effects of asymmetric dependencies between tokens. The newly generated artificial tokens serve as brokers for the Query (Q), forming a novel lightweight fusion strategy. Both subjective visual observations and quantitative analyses demonstrate that the Transformer proposed in this paper is comparable to existing fusion methods in terms of performance while achieving a notable enhancement in its inference efficiency. This is made possible by the attention paradigm, which is equivalent to a generalized form of linear attention, and the newly designed loss function. The model has been experimented on with multiple datasets of different rock types and has exhibited robust generalization capabilities. It provides potential for future research in diverse geological conditions and broader application scenarios.