In geostationary Earth orbit high-throughput satellite beam hopping systems, dynamic non-uniform traffic causes inter-beam service imbalance. Existing scheduling methods primarily rely on instantaneous demand, causing long-term deprived beams to be repeatedly skipped and thereby accumulating delay and queuing risk. We propose a service urgency-aware framework, termed the beam-hopping urgency-aware resource scheduling architecture (BH-URSA). It constructs a unified service-urgency metric to characterize long-term service pressure, integrates link-state information into an interpretable base priority, and employs bounded-residual actor-critic learning to adaptively adjust the base priority. A greedy priority scheduler then generates feasible allocations under hard constraints. Simulations show that under medium load, the average completion delay drops to 0.251 slots, an 87.3% reduction over the baseline. Under high load, the 10th percentile tail service rate reaches 27.75%, improving by 21.11 and 22.90 percentage points over the Demand-Greedy and Probabilistic-BQP baselines, respectively, while maintaining a competitive demand satisfaction ratio and a drop rate comparable to the Demand-Greedy baseline. These results demonstrate that explicit long-term service-state modeling significantly improves delay and tail-service performance.