As an everyday beverage, coffee is consumed worldwide, generating a high amount of waste after brewing, which needs attention for its disposal. These residues are referred to as spent coffee grounds (SCGs), which have been shown to have applications as polymers/composites precursors, biofuels, and biofertilizers. This review focuses on agricultural applications usually based on organic matter to fertilize the soil and consequently improve plant growth. To date, SCGs have been shown to exhibit outstanding performance when applied as soil amendment and composting because it is a nutrient-rich organic waste without heavy metals. Therefore, this review presents the different options to use SCGs in agriculture. First, SCG composition using different characterization techniques is presented to identify the main components. Then, a review is presented showing how SCG toxicity can be resolved when used alone in the soil, especially at high concentrations. In this case, SCG is shown to be effective not only to enhance plant growth, but also to enhance nutritional values without impacting the environment while substituting conventional fertilizers. Finally, a conclusion is presented with openings for future developments.