Congestion propagates in urban road networks following latent patterns, constrained by the physical topology of the road network and influenced by human mobility behavior. Exploring and leveraging these patterns can provide decision-making support for traffic management authorities. However, existing methods typically assume independent congestion effects, neglecting the sequential impacts of congestion propagation and failing to capture long-term dependencies in the spreading process. To address these limitations, we propose a novel reinforcement learning-based congestion propagation pattern mining method (named GRL_CPP). This approach formalizes congestion propagation as a Markov Decision Process (MDP) and employs reinforcement learning algorithms to continuously optimize decisions, automatically learning long-term dependencies within congestion propagation sequences. With the exploration mechanism of reinforcement learning, GRL_CPP can actively identify the most discriminative propagation patterns in complex traffic environments while avoiding unrealistic assumptions. Extensive experiments on both real-world and synthetic datasets demonstrate that our method significantly outperforms state-of-the-art baselines, achieving over 41% average improvements in real-world bidirectional congestion analysis across various data sparsity scenarios, and 74% improvement in the F1-score for true propagation graph reconstruction.