Photochemical grid models are addressing an increasing variety of air quality related issues, yet procedures and metrics used to evaluate their performance remain inconsistent. This impacts the ability to place results in quantitative context relative to other models and applications, and to inform the user and affected community of model uncertainties and weaknesses. More consistent evaluations can serve to drive improvements in the modeling process as major weaknesses are identified and addressed. The large number of North American photochemical modeling studies published in the peer-reviewed literature over the past decade affords a rich data set from which to update previously established quantitative performance “benchmarks” for ozone and particulate matter (PM) concentrations. Here we exploit this information to develop new ozone and PM benchmarks (goals and criteria) for three well-established statistical metrics over spatial scales ranging from urban to regional and over temporal scales ranging from episodic to seasonal. We also recommend additional evaluation procedures, statistical metrics, and graphical methods for good practice. While we primarily address modeling and regulatory settings in the United States, these recommendations are relevant to any such applications of state-of-the-science photochemical models. Our primary objective is to promote quantitatively consistent evaluations across different applications, scales, models, model inputs, and configurations. The purpose of benchmarks is to understand how good or poor the results are relative to historical model applications of similar nature and to guide model performance improvements prior to using results for policy assessments. To that end, it also remains critical to evaluate all aspects of the model via diagnostic and dynamic methods. A second objective is to establish a means to assess model performance changes in the future. Statistical metrics and benchmarks need to be revisited periodically as model performance and the characteristics of air quality change in the future. Implications: We address inconsistent procedures and metrics used to evaluate photochemical model performance, recommend a specific set of statistical metrics, and develop updated quantitative performance benchmarks for those metrics. We promote quantitatively consistent evaluations across different applications, scales, models, inputs, and configurations, thereby (1) improving the user’s ability to quantitatively place results in context and guide model improvements, and (2) better informing users, regulators, and stakeholders of model uncertainties and weaknesses prior to using results for policy assessments. While we primarily address U.S. modeling and regulatory settings, these recommendations are relevant to any such applications of state-of-the-science photochemical models.

Numerical air quality models (AQMs) have been applied more frequently over the past decade to address diverse scientific and regulatory issues associated with deteriorated air quality in China. Thorough evaluation of a model's ability to replicate monitored conditions (i.e., a model performance evaluation or MPE) helps to illuminate the robustness and reliability of the baseline modeling results and subsequent analyses. However, with numerous input data requirements, diverse model configurations, and the scientific evolution of the models themselves, no two AQM applications are the same and their performance results should be expected to differ. MPE procedures have been developed for Europe and North America, but there is currently no uniform set of MPE procedures and associated benchmarks for China. Here we present an extensive review of model performance for fine particulate matter (PM_2.5) AQM applications to China and, from this context, propose a set of statistical benchmarks that can be used to objectively evaluate model performance for PM_2.5 AQM applications in China. We compiled MPE results from 307 peer-reviewed articles published between 2006 and 2019, which applied five of the most frequently used AQMs in China. We analyze influences on the range of reported statistics from different model configurations, including modeling regions and seasons, spatial resolution of modeling grids, temporal resolution of the MPE, etc. Analysis using a random forest method shows that the choices of emission inventory, grid resolution, and aerosol- and gas-phase chemistry are the top three factors affecting model performance for PM_2.5. We propose benchmarks for six frequently used evaluation metrics for AQM applications in China, including two tiers – “goals” and “criteria” – where goals represent the best model performance that a model is currently expected to achieve and criteria represent the model performance that the majority of studies can meet. Our results formed a benchmark framework for the modeling performance of PM_2.5 and its chemical species in China. For instance, in order to meet the goal and criteria, the normalized mean bias (NMB) for total PM_2.5 should be within 10 % and 20 %, while the normalized mean error (NME) should be within 35 % and 45 %, respectively. The goal and criteria values of correlation coefficients for evaluating hourly and daily PM_2.5 are 0.70 and 0.60, respectively; corresponding values are higher when the index of agreement (IOA) is used (0.80 for goal and 0.70 for criteria). Results from this study will support the ever-growing modeling community in China by providing a more objective assessment and context for how well their results compare with previous studies and to better demonstrate the credibility and robustness of their AQM applications prior to subsequent regulatory assessments.

The Comprehensive Air Quality Model with extensions (CAMx) is an open-source, state-of-the-science photochemical grid model that addresses tropospheric air pollution (ozone, particulates, air toxics) over spatial scales ranging from neighborhoods to continents. CAMx has been in continuous development for over 25 years and has been used by numerous entities ranging from government to industry to academia to support regulatory actions and scientific research addressing a variety of air quality issues. Here, we describe the technical formulation of CAMx v7.20, publicly released in May 2022. To illustrate an example of regional and seasonal model performance for predicted ozone and fine particulate matter (PM_2.5), we summarize a model evaluation from a recent 2016 national-scale CAMx application over nine climate zones contained within the conterminous US. We show that statistical performance for warm season maximum 8 h ozone is consistently within benchmark statistical criteria for bias, gross error, and correlation over all climate zones, and often near statistical goals. Statistical performance for 24 h PM_2.5 and constituents fluctuate around statistical criteria with more seasonal and regional variability that can be attributed to different sources of uncertainty among PM_2.5 species (e.g., weather influences, chemical treatments and interactions, emissions uncertainty, and ammonia treatments). We close with a mention of new features and capabilities that are planned for the next public releases of the model in 2024 and beyond.

Ground-level ozone (O₃) has emerged as a significant air pollutant in China, attracting increasing attention from both the scientific community and policymakers. Chemical transport models (CTMs) serve as crucial tools in addressing O₃ pollution, with frequent applications in predicting O₃ concentrations, identifying source contributions, and formulating effective control strategies. The accuracy and reliability of the simulated O₃ concentrations are typically assessed through model performance evaluation (MPE). However, the wide array of CTMs available, variations in input data, model setups, and other factors result in a broad range of differences between simulated and observed O₃ concentrations, highlighting the necessity of standardized benchmarks in O₃ evaluation. Building upon our previous work, this study conducted a thorough literature review of CTM applications simulating O₃ in China from 2006 to 2021. A total of 216 relevant articles out of a total of 667 reviewed were identified to extract quantitative MPE results and key model configurations. From our analysis, two sets of benchmark values for six commonly used MPE metrics are proposed for CTM applications in China, categorized into “goal” benchmarks representing optimal model performance and “criteria” benchmarks representing achievable model performance across a majority of studies. It is recommended that the normalized mean bias (NMB) for hourly O₃ and daily 8 h maximum O₃ concentrations should ideally fall within ±15 % and ±10 %, respectively, to meet the goal benchmark. If the criteria benchmarks are to be met, the NMB should be within ±30 % and ±20 %, respectively. Moreover, uncertainties in O₃ predictions due to uncertainties in various model inputs were quantified using the decoupled direct method (DDM) in a commonly used CTM. For the simulation period of June 2021, the total uncertainty of simulated O₃ ranged from 4 to 25 µg m⁻³, with anthropogenic volatile organic compound (AVOC) emissions contributing most to the uncertainty regarding O₃ in coastal regions and with O₃ boundary conditions playing a dominant role in the northwestern region. The proposed benchmarks for assessing simulated O₃ concentrations, in conjunction with our previous studies on PM_2.5 and other criteria air pollutants, represent a comprehensive and systematic effort to establish a model performance framework for CTM applications in China. These benchmarks aim to support the growing modeling community in China by offering a robust set of evaluation metrics and establishing a consistent evaluation methodology relative to the body of prior research, thereby helping to establish the credibility and reliability of CTM applications. These statistical benchmarks need to be periodically updated as models advance and as better inputs become available in the future.

The Comprehensive Air quality Model with extensions (CAMx) is an open-source, state-of-the-science photochemical grid model that addresses tropospheric air pollution (ozone, particulates, air toxics) over spatial scales ranging from neighborhoods to continents. CAMx has been in continuous development for over 25 years and used by numerous entities ranging from government to industry to academia to support regulatory actions and scientific research addressing a variety of air quality issues. Here we describe the technical formulation of CAMx v7.20, the current publicly available model version. To illustrate an example of regional and seasonal model performance for predicted ozone and fine particulate matter (PM_2.5), we summarize a model evaluation from a recent 2016 national-scale CAMx application over nine climate zones contained within the conterminous US. From that evaluation, we find that statistical performance for warm season maximum 8-hour ozone is consistently within benchmark statistical criteria for bias, gross error, and correlation over all climate zones, and often near statistical goals. Statistical performance for 24-hour PM_2.5 and constituents fluctuate around statistical criteria with more seasonal and regional variability that can be attributed to different sources of uncertainty among PM_2.5 species (e.g., weather influences, chemical treatments and interactions, emissions uncertainty, and ammonia treatments). We close with a mention of new features and capabilities that will be included in upcoming public releases of the model.