Aerial image sequence mosaicking is one of the challenging research fields in computer vision. To obtain large-scale orthophoto maps with object detection information, we propose a vision-based image mosaicking algorithm without any extra location data. According to object detection results, we define a complexity factor to describe the importance of each input Image and dynamically optimize the feature extraction process. The feature points extraction and matching processes are mainly guided by the speeded-up robust features (SURF) and the grid motion statistic (GMS) algorithm respectively. A robust reference frame selection method is proposed to eliminate the transformation distortion by searching for the center area based on overlaps. Besides, the sparse Levenberg-Marquardt (LM) algorithm and the heavy occluded frames removal method are Applied to reduce accumulated errors and further improve the mosaicking performance. The proposed algorithm is performed by using multithreading and graphics processing unit (GPU) acceleration on several aerial image datasets. Extensive experiment results demonstrate that our algorithm outperforms most of the existing aerial image mosaicking methods in visual quality while guaranteeing a high calculation speed.