Humans often express empathy toward others in daily conversations, and exploring ways to respond empathetically is a crucial step in building human-like dialogue systems. Psychological theories hold that empathy mainly consists of two aspects: affection and cognition. However, previous approaches fail to adequately integrate these two aspects, which could lead to inappropriate responses. To address this issue, we propose a reinforcement learning method that integrates empathy level alignment and semantic relevance (RLES) for empathetic response generation. RLES is primarily divided into two stages: fine-tuning the pre-trained T5 model during the supervised learning stage, and then further training the fine-tuned T5 model during the reinforcement learning stage. Specifically, during the supervised learning stage, we fine-tune the pre-trained T5 model to improve its adaptability in empathetic dialogue scenarios. During the reinforcement learning stage, we first initialize the policy with the fine-tuned T5 model. Then, a reward model was designed by incorporating empathy level alignment and semantic relevance from the perspectives of affective and cognitive empathy, respectively. This design reflects a balanced symmetry between the affective and cognitive aspects of empathy in their contributions to the final reward. Finally, the PPO algorithm was used to further train the fine-tuned T5 model, enabling it to generate empathetic responses by maximizing the expected reward. Extensive experimental results on a benchmark dataset demonstrate that RLES outperforms the baselines in both automatic and human evaluations and can generate more empathetic and semantically relevant responses.