To address the inherent limitations of the Dhole Optimization Algorithm (DOA)—limited exploration range, insufficient population diversity, and slow convergence—this paper proposes a Modified Dhole Optimization Algorithm (MDOA) integrating a Beta distribution-based opposition learning strategy, a DE/rand-to-best/1 differential mutation mechanism, and nonlinear parameter control. MDOA is evaluated on 41 CEC2017 and CEC2022 benchmark functions, outperforming 11 state-of-the-art algorithms in convergence speed, accuracy, and robustness. It is then applied to five engineering optimization problems: compression spring design, speed reducer weight minimization, rolling bearing optimization, tubular column design, and moisture content prediction of Dendrobium huoshanense using near-infrared spectroscopy with a BP neural network. The MDOA-BP model reduces MAE, RMSE, MSE, and MAPE by 27.5%, 27.8%, 47.6%, and 31.0%, respectively, while increasing $R^2$ from 0.8339 to 0.9130, achieving the best results among all comparison models. These results demonstrate that MDOA is a highly effective and robust optimizer for complex constrained engineering and high-dimensional optimization tasks.