Ship trajectory tracking controllers based on deep reinforcement learning (DRL) are widely applied in various fields such as autonomous driving and robotics due to their strong adaptive learning capabilities and optimization decision-making ability. However, ship trajectory control faces challenges such as long training cycles and poor convergence performance. These issues are primarily caused by the unreasonable design of algorithm models and reward functions, which limit the performance optimization and energy efficiency improvements in real-world navigation. In this paper, we propose a ship trajectory tracking control algorithm based on deep reinforcement learning. The proposed algorithm introduces maximum entropy theory and experience replay techniques. Additionally, it enhances the reward function module by adding reward terms and fitting weight designs. A three-dimensional simulation environment is constructed to validate the proposed method. The results demonstrate that the controller designed in this study outperforms traditional DRL controllers in terms of fast convergence, convergence stability, and final reward values. The controller meets the requirements for tracking conventional trajectories and shows stable and efficient performance in both wide-area water search experiments and river channel traversal experiments. These experimental results provide valuable insights for future research directions.