The existing small target detection methods may suffer serious false alarm rate and low probability of detection in the situation of intricate background clutter. To cope with this problem, a novel small target detection method is proposed in this paper. Initially, the infrared image is transformed to the infrared gradient vector field (IGVF), where some new distinctive characters of the target and background clutter can be exploited. The small targets show as sink points, while the heavy clutter illustrates high direction coherence in IGVF. Then, the multiscale flux density (MFD) is proposed to quantify the extent of sink point character. In the MFD map, the small targets can be well enhanced and background clutters can be suppressed simultaneously. After that, by analyzing the coherence of heavy clutter shown in the IGVF, the gradient direction diversity (GDD) is presented. The residual noise caused by the heavy clutter in IGVF can be further suppressed by GDD. Finally, an adaptive threshold is adopted to separate the targets. Extensive experiments, including both real data and synthesized data, show that the proposed method outperforms other state-of-the-art methods, especially for infrared images with complex background clutter. Moreover, the experiments prove that the proposed method can work stably for different small target quantities, distances between adjacent targets, target shapes, and noise types with reasonable computational cost.