Heavy metal transfer from agricultural soils to edible crops is an important concern for food safety and sustainable land management in tropical regions, where intensive weathering and heterogeneous parent materials can decouple total soil metal concentrations from plant uptake. In this study, 1120 soil samples and 1093 crop samples representing 55 crop species and five operational crop groups were collected across Hainan Island, China. Geological background and 38 soil physicochemical properties, including available micronutrients such as B, Cu, Fe, Mn, Mo, S, and Zn, were integrated into Random Forest models to evaluate their associations with the bioaccumulation factors (BCFs) of Pb, Cr, Cd, As, and Hg. Mean BCFs on a dry-weight basis were 1.144 for Cd, 0.111 for Hg, 0.057 for Cr, 0.046 for As, and 0.009 for Pb. Cd transfer was relatively higher in vegetables, whereas woody fruits generally showed lower transfer. Five-block spatial validation with a 5 km training buffer yielded BCF R2 values of 0.045 for Cd, 0.122 for Cr, 0.060 for Hg, −0.079 for Pb, and −0.061 for As, which were substantially lower than the corresponding OOB estimates. Removing soil Cr reduced the spatial R2 for Cr BCF from 0.122 to 0.002, whereas removal of soil Cd had little effect on Cd BCF performance. Excluding crop group also reduced predictive performance for Cd, Cr, and Hg. Geological background was associated with total soil metal distributions, whereas soil pH, Fe/Al-related variables, exchangeable cations, and crop group showed associations with variation in soil-to-crop transfer. Accumulated local effects analysis showed nonlinear fitted relationships between BCF and several key soil properties within the observed data range. Overall, total soil metal concentrations alone provided an incomplete indication of crop accumulation. These results highlight the value of considering soil acidity, exchangeable base-cation status, geological background, and crop type together when evaluating heavy-metal transfer in highly weathered tropical agricultural soils.