This study addresses the shortcomings of conventional orbital dynamics methods in order to determine initial orbits for short-arc segments of space objects. By integrating the temporal characteristics of observational data, we innovate a multi-scale Informer temporal modeling approach, proposing a high-precision algorithm for short-arc-segment initial orbit determination. The study analyses why Informer models yield differing results across various time windows. First, a radar observation target model accounting for multiple perturbations and a training data generator were established to produce training data for the Informer. Subsequently, an Informer network framework was designed, encompassing data preprocessing, network architecture, and training algorithms. Realistic scenarios and evaluation metrics were then configured for digital simulation. The model’s feasibility for low-Earth-orbit satellites was validated through digital simulation for different scenarios. The results in Scenario 1 demonstrate that compared to DNN methods, this approach achieves improvements in Root Mean Square Error (RMSE) across six dimensions in ECI—x, y, z, vx, vy, and vz—of 84.04%, 80.56%, 41.38%, 60.00%, 89.03%, and 64.17% respectively; compared to the best results of the Gibbs method across different windows, this approach improves the RMSE by 25%, 23%, and 46% in the three velocity dimensions (vx, vy, and vz) in the ECI frame, respectively. The results in Scenario 2 demonstrate the universality of this method. Furthermore, the reasons for differing outcomes across Informer models with varying time windows were analyzed, alongside the rationale for the integrated Informer model outperforming individual Informer models.