Single-object tracking in complex scenes faces challenges such as drastic target scale variation and strong background interference. To address these issues, an object tracking algorithm based on multi-scale attention and adaptive fusion is proposed. The method integrates a multi-scale attention module and an adaptive gated fusion module, enabling the adaptive mining of key features and dynamic adjustment of fusion weights across multi-level features. This effectively highlights target regions, suppresses redundant information, and enhances the model’s discriminative capability and robustness under complex backgrounds and occlusion. Experiments are conducted on the OTB100 and UAV123 datasets. Results show that, compared with the baseline model, the proposed algorithm improves the success rate and precision by 1.9% and 3.3%, respectively, on OTB100, and by 2.9% and 3.5%, respectively, on UAV123. Moreover, it achieves superior performance when facing typical challenging attributes such as occlusion, scale variation, and background clutter. In summary, the proposed algorithm enhances both tracking accuracy and robustness, offering a viable approach for object tracking under complex conditions.