Air-to-ground visual target persistent tracking technology for swarm drones, as a crucial interdisciplinary research area integrating computer vision, autonomous systems, and swarm collaboration, has gained increasing prominence in anti-terrorism operations, disaster relief, and other emergency response applications. While recent advancements have predominantly concentrated on improving long-term visual tracking through image algorithmic optimizations, insufficient exploration has been conducted on developing system-level persistent tracking architectures, leading to a high target loss rate and limited tracking endurance in complex scenarios. This paper designs an asynchronous multi-task parallel architecture for drone-based long-term tracking in air-to-ground scenarios, and improves the persistent tracking capability from three levels. At the image algorithm level, a long-term tracking system is constructed by integrating existing object detection YOLOv10, multi-object tracking DeepSort, and single-object tracking ECO algorithms. By leveraging their complementary strengths, the system enhances the performance of the detection and multi-object tracking while mitigating model drift in single-object tracking. At the drone system level, ground target absolute localization and geolocation-based drone spiral tracking strategies are conducted to improve target reacquisition rates after tracking loss. At the swarm collaboration level, an autonomous task allocation algorithm and relay tracking handover protocol are proposed, further enhancing the long-term tracking capability of swarm drones while boosting their autonomy. Finally, a practical swarm drone system for persistent air-to-ground visual tracking is developed and validated through extensive flight experiments under diverse scenarios. Results demonstrate the feasibility and robustness of the proposed persistent tracking framework and its adaptability to wild real-world applications.