The rapid proliferation of high-resolution remote sensing video (RSV) has made object segmentation an increasingly critical yet challenging task. Existing methods face two fundamental limitations in this domain: (1) the intrinsic complexity of RSVs arising from diminutive object sizes, dense distributions, and drastic scale variations; (2) prohibitive computational overhead and latency caused by the massive spatial resolution of remote sensing imagery. To address these challenges, we propose Trajectory-SAM, a novel object trajectory-guided segmentation framework tailored for rapid and accurate object segmentation in RSVs. Specifically, we first employ a Graph Neural Network to extract representative keyframes, improving data utilization efficiency while suppressing attention drift and interpretation bias induced by redundant frames. We then introduce Temporal-LoRA, a parameter-efficient fine-tuning strategy for the Segment Anything Model (SAM), substantially enhancing its spatiotemporal contextual understanding in remote sensing domains. Finally, we devise a sample-augmentation-based knowledge distillation strategy to accelerate inference without sacrificing segmentation quality. Extensive experiments on the SAT-MTB, iSAID, and VHR-10 datasets demonstrate that our proposed Trajectory-SAM achieves superior performance, establishing a compelling trade-off between accuracy and speed.