Nitrogen (N) is an essential and limiting nutrient for crop production, as it is a structural part of plants and is involved in various processes. Worldwide, agricultural soils lack one or more essential nutrients, and nitrogen is one of them. Adding a sufficient amount of N will increase production. However, the overuse of N and loss of N from the soil-plant system is detrimental to the environment and results in economic losses. Nitrogen has reactive forms like ammonia, ammonium, nitrate, nitrite, nitric oxide, and nitrous oxide. Some reactive forms of N are harmful to humans, animals, plants, and microbial ecology. Nitrate can cause the eutrophication of surface water and contamination of groundwater. Drinking nitrate-contaminated water can cause methemoglobinemia and other health issues. Nitrous oxide emission depletes the ozone layer and contributes to climate change. Ammonia emissions contribute to acid rain and are also responsible for nitrous oxide emissions. This review addresses different factors/pathways/circumstances that contribute to the loss of N from the soil-plant system and reduce nitrogen use efficiency (NUE). Different factors influence NUE like ammonia volatilization, nitrification, denitrification, immobilization, leaching, runoff, temperature, soil pH, soil texture, rainfall and irrigation, soil salinity, tillage, weeds, pests, diseases, N loss from plants, fires, crop rotation, crop nutrition, crop varieties, and nitrogen management (right time, right source, right place, and right rate/amount).