The rapid popularity of UAVs has encouraged the development of Anti-UAV technology. Infrared-detector-based visual tracking for UAVs provides an encouraging solution for Anti-UAVs. However, it still faces the problem of tracking instability caused by environmental thermal crossover and similar distractors. To address these issues, we propose a spatio-temporal-focused Siamese network for infrared UAV tracking, called STFTrack. This method employs a two-level target focusing strategy from global to local. First, a feature pyramid-based Siamese backbone is constructed to enhance the feature expression of infrared UAVs through cross-scale feature fusion. By combining template and motion features, we guide prior anchor boxes towards the suspicious region to enable adaptive search region selection, thus effectively suppressing background interference and generating high-quality candidates. Furthermore, we propose an instance-discriminative RCNN based on metric learning to focus on the target UAV among candidates. By measuring calculating the feature distance between the candidates and the template, it assists in discriminating the optimal target from the candidates, thus improving the discrimination of the proposed method to infrared UAV. Extensive experiments on the Anti-UAV dataset demonstrate that the proposed method achieves outstanding performance for infrared tracking, with 91.2% precision, 66.6% success rate, and 67.7% average overlap accuracy, and it exceeded the baseline algorithm by 2.3%, 2.7%, and 3.5%, respectively. The attribute-based evaluation demonstrates that the proposed method achieves robust tracking effects on challenging scenes such as fast motion, thermal crossover, and similar distractors. Evaluation on the LSOTB-TIR dataset shows that the proposed method reaches a precision of 77.2% and a success rate of 63.4%, outperforming other advanced trackers.