The deep learning paradigm is progressively shifting from non-smooth activation functions, exemplified by ReLU, to smoother alternatives such as GELU and SiLU. This transition is motivated by the fact that non-differentiability introduces challenges for gradient-based optimization, while an expanding body of research demonstrates that smooth activations yield superior convergence, improved generalization, and enhanced training stability. A central challenge, however, is how to systematically transform widely used non-smooth functions into smooth counterparts that preserve their proven representational strengths while improving differentiability and computational efficiency. To address this, we propose a general activation smoothing framework grounded in mollification theory. Leveraging the Epanechnikov kernel, the framework achieves statistical optimality and computational tractability, thereby combining theoretical rigor with practical utility. Within this framework, we introduce Smoothed ReLU (S-ReLU), a novel second-order continuously differentiable (C²) activation derived from ReLU that inherits its favorable properties while mitigating inherent drawbacks. Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet-1K with Vision Transformers and ConvNeXt consistently demonstrate the superior performance of S-ReLU over existing ReLU variants. Beyond computer vision, large-scale fine-tuning experiments on language models further show that S-ReLU surpasses GELU, underscoring its broad applicability across both vision and language domains and its potential to enhance stability and scalability.

Amid the exponential growth of streaming data and rising demands for real-time decision-making, Online Convex Optimization (OCO) has emerged as a foundational framework for sequential data processing in dynamic environments. This survey presents a systematic review of recent evolutionary and adaptive OCO strategies, offering a detailed taxonomy that classifies algorithms according to their constraint-handling mechanisms and environmental feedback. The analysis first examines Constrained OCO, elucidating the trade-offs between computational efficiency and theoretical guarantees across projection-based methods, projection-free Frank–Wolfe variants, and general convex optimization approaches. It then explores the Unconstrained OCO landscape, emphasizing the shift from parameter-dependent methods to fully adaptive, parameter-free algorithms capable of handling unknown comparator norms and gradient scales. Furthermore, the study synthesizes state-of-the-art applications in power systems, network communication, and quantitative finance, bridging theoretical OCO models with robust engineering solutions. The paper concludes by outlining critical open challenges and future research directions, such as the integration of OCO with deep learning, non-convex optimization, and robustness against adversarial corruptions in data-intensive scenarios.