Recent deep learning methods have achieved promising performance in infrared small target detection (IRSTD) but with high computational cost, limiting deployment or operation on resource-limited scenarios. There is an urgent need to develop both lightweight and high-precision model compression methods. In this paper, we propose a Multi-Directional Contrast Spatial Attention-driven Feature Distillation (DSAD) method for achieving quick and high-performance IRSTD. Specifically, we first extract feature maps from teacher and student networks. Then, a standard Gaussian transformation is adopted to eliminate magnitude effects. After that, a Multi-Directional Contrast Spatial Attention (DSA) is designed to capture multi-directional spatial information from teacher features, which can make student networks pay more attention to small target areas while suppressing background. Finally, we propose a Perceptual Weighted Mean Square Error (PWMSE) distillation loss by combining the DSA with feature discrepancies, guiding student networks to learn more effective information from small target features. Experimental results on the two benchmark datasets (e.g., NUDT-SIRST and NUAA-SIRST) demonstrate that our distillation method can achieve remarkable detection performance compared with the teacher counterparts on several benchmark IRSTD networks (e.g., DNANet, AMFU-Net, and DMFNet) and introduce consistent gains in inference speed (i.e., 2× more) on edge devices (NVIDIA AGX and HUAWEI Ascend-310B).