Previous studies have shown that the use of appropriate ground control points (GCPs) and camera calibration models can optimize photogrammetry. However, the synergistic effects of GCPs and camera calibration models on UAV-SfM photogrammetry are still unknown. This study used camera models with varying complexities under different GCP conditions (in terms of number and quality) for UAV-SfM photogrammetry. The correlation matrix and root mean squared error (RMSE) were used to analyze the synergistic effects of GCPs and camera models. The results show that (1) without GCPs, complex camera models reduce distortion parameter correlation and improve terrain modeling accuracy by about 70%, with Model C (with F, Cx, Cy, K1–K4, and P1–P4) being the most widely applicable. (2) Increasing the number of GCPs enhances the terrain modeling accuracy more effectively than increasing the camera model complexity, reducing the RMSE by 45–70%, while the model complexity does not affect the required GCP number. (3) A strong interaction exists between the GCP quality and camera models: High-quality GCPs enhance camera model performance, while complex camera models reduce the requirement of GCP quality. This study provides both theoretical insights and practical guidance for efficient and low-cost UAV-SfM photogrammetry in different scenarios.