Although perennial fruit crops represent 1% of global agricultural land, they are of a great economic importance in world trade and in the economy of many regions. The perennial woody nature of fruit trees, their physiological stages of growth, the root distribution pattern, and the presence of herbaceous vegetation in alleys make orchard systems efficient in the use and recycling of nitrogen (N). The present paper intends to review the existing literature on N nutrition of young and mature deciduous and evergreen fruit trees with special emphasis to temperate and Mediterranean climates. There are two major sources of N contributing to vegetative tree growth and reproduction: root N uptake and internal N cycling. Optimisation of the use of external and internal N sources is important for a sustainable fruit production, as N use efficiency by young and mature fruit trees is generally lower than 55% and losses of fertilizer N may occur with the consequent economic and environmental concern. Organic alternatives to mineral N fertilizer like the application of manure, compost, mulching, and cover crops are scarcely used in perennial fruit trees, in spite of the fact that society’s expectations call for more sustainable production techniques and the demand for organic fruits is increasing.