Deep learning technologies have been increasingly applied to precision weed management, and You Only Look Once (YOLO) has become widely adopted owing to its real-time detection capability and high accuracy. This systematic review analyzes YOLO-based weed recognition models with emphasis on field challenges and architectural optimization strategies. A structured search of Web of Science and Scopus, supplemented by manual searching, was conducted in accordance with the PRISMA 2020 guidelines. Following screening and eligibility assessment, 45 studies were included in the systematic synthesis. The main categories of analysis included attention mechanisms that enhance discrimination among vegetation with similar shapes and colors, multi-scale feature-fusion architectures that integrate information across feature pyramid levels to improve the detection of weeds of varying sizes and viewpoints, and lightweight architectures designed to support real-time operation on resource-constrained hardware, along with other optimization strategies. Recent developments increasingly combine these approaches to address crop–weed similarity, occlusion, scale variation, and complex field backgrounds while improving computational efficiency. The review further examines how architectural enhancement strategies contribute to detection accuracy and computational efficiency and discusses their implications for practical applications such as selective spraying and robotic weeding. Overall, this review links major field-level weed-recognition challenges with corresponding architectural optimization strategies and provides a systematic framework for developing accurate and efficient YOLO-based weed recognition systems for site-specific weed management.