Satellite-terrestrial integrated networks (STINs) require a robust handover mechanism to ensure reliable mobility management and load balancing. However, many studies still focus on measurement-triggered or learning-based single-user handover rules. In multi-user STIN scenarios, such schemes often trigger inefficient handovers that degrade users’ quality of service (QoS). Therefore, this letter proposes a handover-oriented learning scheme for multi-UE STINs and designs a reward function based on a service-differentiated utility function that combines QoS metrics with costs and temporal penalties for ping-pong and too-late handovers. Handover decisions are learned via multi-agent proximal policy optimization (MAPPO) under a centralized training with decentralized execution (CTDE) paradigm. When decisions become coupled due to shared resource constraints, the learning process is stabilized by a centralized critic. Simulations demonstrate that the proposed method achieves higher average utility with fewer handovers and failures compared with threshold-based and single-agent reinforcement learning baselines.