Agriculture is essential for global income, poverty reduction, and food security, with crop yield being a crucial measure in this field. Traditional crop yield prediction methods, reliant on subjective assessments such as farmers’ experiences, tend to be error-prone and lack precision across vast farming areas, especially in data-scarce regions. Recent advancements in data collection, notably through high-resolution sensors and the use of deep learning (DL), have significantly increased the accuracy and breadth of agricultural data, providing better support for policymakers and administrators. In our study, we conduct a systematic literature review to explore the application of DL in crop yield forecasting, underscoring its growing significance in enhancing yield predictions. Our approach enabled us to identify 92 relevant studies across four major scientific databases: the Directory of Open Access Journals (DOAJ), the Institute of Electrical and Electronics Engineers (IEEE), the Multidisciplinary Digital Publishing Institute (MDPI), and ScienceDirect. These studies, all empirical research published in the last eight years, met stringent selection criteria, including empirical validity, methodological clarity, and a minimum quality score, ensuring their rigorous research standards and relevance. Our in-depth analysis of these papers aimed to synthesize insights on the crops studied, DL models utilized, key input data types, and the specific challenges and prerequisites for accurate DL-based yield forecasting. Our findings reveal that convolutional neural networks and Long Short-Term Memory are the dominant deep learning architectures in crop yield prediction, with a focus on cereals like wheat (Triticum aestivum) and corn (Zea mays). Many studies leverage satellite imagery, but there is a growing trend towards using Unmanned Aerial Vehicles (UAVs) for data collection. Our review synthesizes global research, suggests future directions, and highlights key studies, acknowledging that results may vary across different databases and emphasizing the need for continual updates due to the evolving nature of the field.