With the flourish of deep learning, transformer models have achieved remarkable performance in dealing with many computer vision tasks. However, their applications in infrared small target detection is limited due to two factors: (1) the high computational complexity of the conventional transformer models reduces the efficiency of detection; (2) the small target is easily left out in the visual environment with complex backgrounds. To deal with the issues, we propose a lightweight infrared small target detection method based on a linear transformer named IstdVit, which achieves high accuracy and low delay in infrared small target detection. The model consists of two parts: a multi-scale linear transformer and a lightweight dual feature pyramid network. It combines the strengths of a lightweight feature extraction module and the multi-head attention mechanism, effectively representing the small targets in the complex background at an economical computational cost. Additionally, it incorporates rotational position encoding to improve understanding of spatial context. The experiments conducted on the NUDT-SIRST and IRSTD-1K datasets indicate that IstdVit achieves a good balance between speed and accuracy, outperforming other state-of-the-art methods while maintaining a low number of parameters.