In arid regions, potentially toxic elements (PTEs) can accumulate in oasis farmland soils, posing risks to both ecosystems and human health and threatening the long-term sustainability of agricultural production. However, we still lack a clear understanding of how multiple elements co-accumulate and what non-linear processes drive their buildup. Here, we investigated typical agricultural soils in the Aksu area of Xinjiang. We measured 12 elements (As, Cr, Cu, Ni, Zn, Co, V, Se, F, Ba, Sn, Mn) in 28 surface samples. To assess the pollution levels, we used three indices: the single-factor index (Pi), the geo-accumulation index (Igeo), and the Nemerow composite index. Source apportionment was performed with positive matrix factorization (PMF). We then built an XGBoost model to predict the Nemerow index, and applied SHAP (Shapley additive explanations) to quantify the marginal contribution and non-linear response of each element. Our results show that the average concentrations of Se, F, and As are 1.47, 1.27, and 1.35 times the national background values, respectively. The exceedance rate (Pi > 1) for these elements ranges from 78.6% to 92.9%. Nevertheless, the overall pollution is mild: only one out of 28 sampling sites (3.6%) falls into the moderately polluted category. PMF resolved three major sources: (1) parent material plus evaporation enrichment (F, Mn, and Ba, ~45% of the total contribution); (2) agricultural and anthropogenic activities (As, Cr, V, Zn, ~40%); and (3) local industrial or waste inputs (Sn and Ba, ~15%). The XGBoost model shows good predictive performance on the test set (R2 = 0.864, RMSE = 0.104). SHAP analysis reveals that Se, F, and As are the main drivers of the composite pollution index. Se has a clear threshold: once its concentration goes above 0.3 mg·kg−1, its positive contribution jumps sharply. Overall, the farmland soils in Aksu show mild enrichment of several elements, with Se and F as the main indicators. Evaporation enrichment and farming practices are the dominant processes behind this enrichment. The integrated framework—pollution indices, PMF source apportionment, XGBoost prediction, and SHAP interpretation—provides a scientifically sound way to manage soil environments in arid regions.