The rapid expansion of online learning platforms has necessitated advanced systems to address scalability, personalization, and assessment challenges. This paper presents a comprehensive review of artificial intelligence (AI)-based decision support systems (DSSs) designed for online learning and assessment, synthesizing advancements from 2020 to 2025. By integrating machine learning, natural language processing, knowledge-based systems, and deep learning, AI-DSSs enhance educational outcomes through predictive analytics, automated grading, and personalized learning paths. This study examines system architecture, data requirements, model selection, and user-centric design, emphasizing their roles in achieving scalability and inclusivity. Through case studies of a MOOC platform using NLP and an adaptive learning system employing reinforcement learning, this paper highlights significant improvements in grading efficiency (up to 70%) and student performance (12–20% grade increases). Performance metrics, including accuracy, response time, and user satisfaction, are analyzed alongside evaluation frameworks combining quantitative and qualitative approaches. Technical challenges, such as model interpretability and bias, ethical concerns like data privacy, and implementation barriers, including cost and adoption resistance, are critically assessed, with proposed mitigation strategies. Future directions explore generative AI, multimodal integration, and cross-cultural studies to enhance global accessibility. This review offers a robust framework for researchers and practitioners, providing actionable insights for designing equitable, efficient, and scalable AI-DSSs to transform online education.