This paper offers a comprehensive examination of the process involved in developing and automating supervised end-to-end machine learning workflows for forecasting and classification purposes. It offers a complete overview of the components (i.e., feature engineering and model selection), principles (i.e., bias–variance decomposition, model complexity, overfitting, model sensitivity to feature assumptions and scaling, and output interpretability), models (i.e., neural networks and regression models), methods (i.e., cross-validation and data augmentation), metrics (i.e., Mean Squared Error and F1-score) and tools that rule most supervised learning applications with numerical and categorical data, as well as their integration, automation, and deployment. The end goal and contribution of this paper is the education and guidance of the non-AI expert academic community regarding complete and rigorous machine learning workflows and data science practices, from problem scoping to design and state-of-the-art automation tools, including basic principles and reasoning in the choice of methods. The paper delves into the critical stages of supervised machine learning workflow development, many of which are often omitted by researchers, and covers foundational concepts essential for understanding and optimizing a functional machine learning workflow, thereby offering a holistic view of task-specific application development for applied researchers who are non-AI experts. This paper may be of significant value to academic researchers developing and prototyping machine learning workflows for their own research or as customer-tailored solutions for government and industry partners.
A mathematical method for accurately determining the upper and lower diffuse-fraction (kd) limits that divide the sky into clear, intermediate, and overcast is developed. Fourteen sites around the world are selected for demonstrating the methodology. The upper and lower kd values for these sites are determined from scatter plots of direct-normal solar radiation vs. kd pairs over the typical meteorological year of each site. They vary between 0.73 and 0.80 for the upper and between 0.24 and 0.27 for the lower kd limits. Plots of sunshine duration (SSD) vs. kd are prepared for 12 of the 14 sites. These plots show a decreasing trend in SSD with increasing values of kd, as anticipated. According to local climatology, the number of the SSD values in each sky-condition classification varies from site-to-site.
2 August 2024 in IEEE Transactions on Systems, Man, and Cybernetics: Systems
In the pathway to 2030 electricity generation decarbonization and 2050 net-zero economies, scalable integration of distributed load can support environmental goals and also help alleviate smart grid operational issues through its electricity market participation. In this work, a novel Stackelberg game theoretic framework is proposed for trading the energy bidirectionally between the demand-response (DR) aggregator and the prosumers (distributed load). This formulation allows for flexible energy arbitrage and additional monetary rewards while ensuring that the prosumers’ desired daily energy demand is met. Then, a scalable (linear with the number of prosumers and the number of learning samples), the decentralized privacy-preserving algorithm is proposed to find approximate equilibria with online sampling and learning of the prosumers’ cumulative best response, which finds applications beyond this energy game. Moreover, cost bounds are provided on the quality of the approximate equilibrium solution. Finally, the real data from the California day-ahead market and the UC Davis campus building energy demands are utilized to demonstrate the efficacy of the proposed framework and the algorithm.
This work investigates the influence of ground albedo on the solar radiation obtained by surfaces mounted on fixed-tilt-to-south, one-axis, and two-axis systems. To do this, estimation of the solar radiation difference is performed by applying real albedo and zero albedo. This is achieved within Saudi Arabia at 82 selected sites. Annual, seasonal, and monthly mean solar energy differences are computed as a function of the site’s number, latitude, and local near-real ground albedo. The great variation in the ground-albedo values at the 82 sites (0.1–0.46) could be thought of as having a significant effect on the solar radiation levels received on the three tracking modes. This analysis shows quite the opposite; zero-albedo ground diminishes solar radiation levels by 1.43%, 3.50%, and 3.20%, respectively, for the three modes. Therefore, in most solar engineering applications, a ground albedo of 0.2 (considered a reference) can be used without losing accuracy. This is the main conclusion of the study, which must, however, be applied with caution in areas with snow cover, especially for mode-III tracking systems. In such situations, the increase in solar radiation levels may be up to 15% (but ≈3.5% for mode-I and -II systems instead).
The decarbonization of electricity generation by 2030 and the realization of a net-zero economy by 2050 are central to the United States’ climate strategy. However, large-scale renewable integration introduces operational challenges, including extreme ramping, unsafe dispatch, and price volatility. This review investigates how demand–response (DR) aggregators and distributed loads can support these climate goals while addressing critical operational challenges. We hypothesize that current DR aggregator frameworks fall short in the areas of distributed load operational flexibility, scalability with the number of distributed loads (prosumers), prosumer privacy preservation, DR aggregator and prosumer competition, and uncertainty management, limiting their potential to enable large-scale prosumer participation. Using a systematic review methodology, we evaluate existing DR aggregator and prosumer frameworks through the proposed FCUPS criteria—flexibility, competition, uncertainty quantification, privacy, and scalability. The main results highlight significant gaps in current frameworks: limited support for decentralized operations; inadequate privacy protections for prosumers; and insufficient capabilities for managing competition, uncertainty, and flexibility at scale. We conclude by identifying open research directions, including the need for game-theoretic and machine learning approaches that ensure privacy, scalability, and robust market participation. Addressing these gaps is essential to shape future research agendas and to enable DR aggregators to contribute meaningfully to US climate targets.
This work investigates the influence of ground albedo on the solar radiation obtained by surfaces mounted on fixed-tilt to south, one-axis, and two-axis systems. To do this, estimation of the solar radiation difference is performed by applying real albedo and zero albedo. This is done within Saudi Arabia, at 82 selected sites. Annual, seasonal, and monthly mean solar energy differences are computed as function of the site’s number, latitude, and local near-real ground albedo. The great variation in the ground-albedo values at the 82 sites (0.1 - 0.46) could be thought as having a significant effect on the solar radiation levels received on the 3 tracking modes. The analysis shows quite the opposite; a zero-albedo ground diminishes solar radiation levels by 1.43%, 3.50%, and 3.20%, respectively for the 3 modes. Therefore, in most solar engineering applications a ground albedo of 0.2 (considered as reference) can be used without losing accuracy. This is the main conclusion of the study, which must, however, be applied with caution in areas with snow cover, especially for mode-III tracking systems. In such situations the increase in solar radiation levels may be up to 15% (but ≈3.5% for mode-I and -II systems instead).