Phosphorus is the primary contributor to eutrophication in water bodies, and identifying phosphorus sources in rivers is crucial for controlling phosphorus pollution and subsequent eutrophication. Although phosphate oxygen isotopes (δ¹⁸O_P) have the capacity to trace phosphorus sources and cycling in water and sediments, they have not been used in small- to medium-sized watersheds, such as the Xiaodongjiang River (XDJ), which is located in an agricultural watershed, source–complex region of southern China. This study employed phosphate oxygen isotope techniques in combination with a land-use-based mixed end-member model and the MixSIAR Bayesian mixing model to quantitatively determine potential phosphorus sources in surface water and sediments. The δ¹⁸O_P values of the surface water ranged from 5.72‰ to 15.02‰, while those of sediment ranged from 10.41‰ to 16.80‰. In the downstream section, the δ¹⁸O_P values of the surface water and sediment were similar, suggesting that phosphate in the downstream water was primarily influenced by endogenous sediment control. The results of the land-use–source mixing model and Bayesian model framework demonstrated that controlling phosphorus inputs from fertilizers is essential for reducing phosphorus emissions in the XDJ watershed. Furthermore, ongoing rural sewage treatment, manure management, and the resource utilization of aquaculture substrates contributed to reduced phosphorus pollution. This study showed that isotope techniques, combined with multi-model approaches, effectively assessed phosphorus sources in complex watersheds, offering a theoretical basis for phosphorus pollution management to prevent eutrophication.