The spatial heterogeneity of leaf traits within canopies is an important source of uncertainty in leaf parameter estimation from unmanned aerial vehicle (UAV) imagery, especially in structurally complex orchards. In this study, we combined three-dimensional (3D) radiative transfer simulations with field measurements from litchi orchards to quantify bidirectional reflectance factor (BRF) uncertainty under four leaf trait distribution patterns within the canopy. Whole-canopy leaf traits were represented using: (1) a homogeneous canopy (HC), (2) vertically divided canopy (VDC), (3) horizontally divided canopy (HDC), and (4) a canopy divided into nine sections (CD9s). Among the simplified schemes, HDC produced BRF values most consistent with the CD9s configuration, while the largest deviation between CD9s and HC was observed at 570 nm with a maximum BRF normalized difference of 65.29%. Relative contribution rate analysis based on the symmetric relative difference (SRD, %) showed that leaf trait distribution pattern dominated the variability of several VIs, including NDVI, NDRE, CCI, SIPI, LICI, and PVI. Meanwhile, other VIs (e.g., NIRv, SAVI, OSAVI and EVI) were more strongly influenced by illumination–viewing geometry. Using multiangle UAV multispectral data improved the estimation of proxy leaf chlorophyll content (LCC, max R²cv = 0.52), while nadir-only data yielded the best results for leaf nitrogen mass-based content (LNC, max R²cv = 0.41). These results emphasize that reliable UAV-based leaf trait retrieval is closely related to leaf trait distribution pattern within the canopy and its interaction with other factors (e.g., illumination–viewing geometry).