Estimating forage yield and nutrient composition using hyperspectral remote sensing is a major challenge. However, there is still a lack of comprehensive research on the optimal wavelength for the analysis of various nutrients in pasture. In this research, conducted in Hailar District, Hulunber City, Inner Mongolia Autonomous Region, China, 126 sets of hyperspectral data were collected, covering a spectral range of 350 to 1800 nanometers. The primary objective was to identify key spectral bands for estimating forage dry matter yield (DMY), nitrogen content (NC), neutral detergent fiber (NDF), and acid detergent fiber (ADF) using principal component analysis (PCA), random forests (RF), and SHapley Additive exPlanations (SHAP) analysis methods, and then the RF and Extra-Trees algorithm (ERT) model was used to predict aboveground biomass (AGB) and nutrient parameters using the optimized spectral bands and vegetation indices. Our approach effectively minimizes redundancy in hyperspectral data by selectively employing crucial spectral bands, thus improving the accuracy of forage nutrient estimation. PCA identified the most variable bands at 400 nm, 520–550 nm, 670–720 nm, and 930–950 nm, reflecting their general spectral significance rather than a link to specific forage nutrients. Further analysis using RF feature importance pinpointed influential bands, predominantly within 930–940 nm and 700–730 nm. SHAP analysis confirmed critical bands for DMY (965 nm, 712 nm, and 1652 nm), NC (1390 nm and 713 nm), ADF (1390 nm and 715–725 nm), and NDF (400 nm, 983 nm, 1350 nm, and 1800 nm). The fitting accuracy for ADF estimated using RF was lower (R² = 0.58), while the fitting accuracy for other indicators was higher (R² ≥ 0.59). The performance and prediction accuracy of ERT (R² = 0.63) were noticeably superior to those of RF. In conclusion, our method effectively identifies influential bands, optimizing forage yield and quality estimation.