2 January 2016 in Journal of Computational and Graphical Statistics
Any decision-making process that relies on a probabilistic forecast of future events necessarily requires a calibrated forecast. This article proposes new methods for empirically assessing forecast calibration in a multivariate setting where the probabilistic forecast is given by an ensemble of equally probable forecast scenarios. Multivariate properties are mapped to a single dimension through a prerank function and the calibration is subsequently assessed visually through a histogram of the ranks of the observation’s preranks. Average ranking assigns a prerank based on the average univariate rank while band depth ranking employs the concept of functional band depth where the centrality of the observation within the forecast ensemble is assessed. Several simulation examples and a case study of temperature forecast trajectories at Berlin Tegel Airport in Germany demonstrate that both multivariate ranking methods can successfully detect various sources of miscalibration and scale efficiently to high-dimensional settings. Supplemental material in form of computer code is available online.
1 December 2020 in Environmental Research Letters
Reliable projections of extremes by climate models are becoming increasingly important in the context of climate change and associated societal impacts. Extremes are by definition rare events, characterized by a small sample associated with large uncertainties. The evaluation of extreme events in model simulations thus requires performance measures that compare full distributions rather than simple summaries. This paper proposes the use of the integrated quadratic distance (IQD) for this purpose. The IQD is applied to evaluate CMIP5 and CMIP6 simulations of monthly maximum and minimum near-surface air temperature over Europe and North America against both observation-based data and reanalyses. Several climate models perform well to the extent that these models’ performance is competitive with the performance of another data product in simulating the evaluation set. While the model rankings vary with region, season and index, the model evaluation is robust against changes in the grid resolution considered in the analysis. When the model simulations are ranked based on their similarity with the ERA5 reanalysis, more CMIP6 than CMIP5 models appear at the top of the ranking. When evaluated against the HadEX2 data product, the overall performance of the two model ensembles is similar.
27 September 2018 in Water Resources Research
Regional flood frequency analysis is commonly applied in situations where there exists insufficient data at a location for a reliable estimation of flood quantiles. We develop a Bayesian hierarchical modeling framework for a regional analysis of data from 203 large catchments in Norway with the generalized extreme value distribution as the underlying model. Generalized linear models on the parameters of the generalized extreme value distribution are able to incorporate location‐specific geographic and meteorological information and thereby accommodate these effects on the flood quantiles. A Bayesian model averaging component additionally assesses model uncertainty in the effect of the proposed covariates. The resulting regional model is seen to give substantially better predictive performance than the regional model currently used in Norway.
25 September 2014 in Geoscientific Instrumentation, Methods and Data Systems
The International Surface Temperature Initiative (ISTI) is striving towards substantively improving our ability to robustly understand historical land surface air temperature change at all scales. A key recently completed first step has been collating all available records into a comprehensive open access, traceable and version-controlled databank. The crucial next step is to maximise the value of the collated data through a robust international framework of benchmarking and assessment for product intercomparison and uncertainty estimation. We focus on uncertainties arising from the presence of inhomogeneities in monthly mean land surface temperature data and the varied methodological choices made by various groups in building homogeneous temperature products. The central facet of the benchmarking process is the creation of global-scale synthetic analogues to the real-world database where both the "true" series and inhomogeneities are known (a luxury the real-world data do not afford us). Hence, algorithmic strengths and weaknesses can be meaningfully quantified and conditional inferences made about the real-world climate system. Here we discuss the necessary framework for developing an international homogenisation benchmarking system on the global scale for monthly mean temperatures. The value of this framework is critically dependent upon the number of groups taking part and so we strongly advocate involvement in the benchmarking exercise from as many data analyst groups as possible to make the best use of this substantial effort.
8 October 2017 in Water Resources Research
Sea level rise has serious consequences for harbor infrastructure, storm drains and sewer systems, and many other issues. Adapting to sea level rise requires comparing different possible adaptation strategies, comparing the cost of different actions (including no action), and assessing where and at what point in time the chosen strategy should be implemented. All these decisions must be made under considerable uncertainty—in the amount of sea level rise, in the cost and prioritization of adaptation actions, and in the implications of no action. Here we develop two illustrative examples: for Bergen on Norway's west coast and for Esbjerg on the west coast of Denmark, to highlight how technical efforts to understand and quantify uncertainties in hydrologic projections can be coupled with concrete decision‐problems framed by the needs of the end‐users using statistical formulations. Different components of uncertainty are visualized. We demonstrate the value of uncertainties and show for example that failing to take uncertainty into account can result in the median‐projected damage costs being an order of magnitude smaller.