Despite recent achievements to reduce surface quenching in NaYF₄:Yb,Er nanocrystals, a complete understanding of how the nanocrystal size affects the brightness of upconversion luminescence is still incomplete. Here we investigated upconversion luminescence of Yb,Er-doped nanocrystals in a broad range of sizes from 6 nm to 45 nm (cubic or hexagonal phases), displaying an increasing red-to-green luminescence intensity ratio and reduced luminescence lifetimes with decreasing size. By analyzing the upconversion process with a set of rate equations, we found that their asymptotic analytic solutions explain lower decay rates of red compared to green upconversion luminescence. Furthermore, we quantified the effect of the surface on luminescence lifetime in a model where nanocrystal emitters are divided between the near-surface and inside regions of each nanocrystal. We clarify the influence of the four nonradiative recombination mechanisms (intrinsic phonon modes, vibration energy of surface ligands, solvent-mediated quenching, and surface defects) on the decay rates for different-size nanocrystals, and find that the defect density dominates decay rates for small (below 15 nm) nanocrystals. Our results indicate that a defect-reduction strategy is a key step in producing small upconversion nanocrystals with increased brightness for a variety of bioimaging and biosensing applications.