Unmanned aerial vehicles (UAVs) face significant challenges in maintaining stability when subjected to external wind disturbances and internal noise. This paper addresses these issues by introducing a real-time wind speed fitting algorithm and a wind field model that accounts for varying wind conditions, such as wind shear and turbulence. To improve control in such conditions, a deep reinforcement learning (DRL) strategy is developed and tested through both simulations and real-world experiments. The results indicate a 65% reduction in trajectory tracking error with the DRL controller. Additionally, a UAV built for testing exhibited enhanced stability and reduced angular deviations in wind conditions up to level 5. These findings demonstrate the effectiveness of the proposed DRL-based control strategy in increasing UAV resilience to wind disturbances.