Simon Mason

Dr. Simon Mason

Columbia University

Research Focus & Metrics

Academic Career Length

23Years (2004-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 July 2004 in Monthly Weather Review
Verified
Journal Article
Closed access
SMSimon Mason
Simon Mason
The Brier and ranked probability skill scores are widely used as skill metrics of probabilistic forecasts of weather and climate. As skill scores, they compare the extent to which a forecast strategy outperforms a (usually simpler) reference forecast strategy. The most widely used reference strategy is that of “climatology,” in which the climatological probability (or probabilities in the case of the ranked probability skill score) of the forecast variable is issued perpetually. The Brier and ranked probability skill scores are often considered harsh standards. It is shown that the scores are harsh because the expected value of these skill scores is less than 0 if nonclimatological forecast probabilities are issued. As a result, negative skill scores can often hide useful information content in the forecasts. An alternative formulation of the skill scores based on a reference strategy in which the outcome is independent of the forecast is equivalent to using randomly assigned probabilities but is... Abstract The Brier and ranked probability skill scores are widely used as skill metrics of probabilistic forecasts of weather and climate. As skill scores, they compare the extent to which a forecast strategy outperforms a (usually simpler) reference forecast strategy. The most widely used reference strategy is that of “climatology,” in which the climatological probability (or probabilities in the case of the ranked probability skill score) of the forecast variable is issued perpetually. The Brier and ranked probability skill scores are often considered harsh standards. It is shown that the scores are harsh because the expected value of these skill scores is less than 0 if nonclimatological forecast probabilities are issued. As a result, negative skill scores can often hide useful information content in the forecasts. An alternative formulation of the skill scores based on a reference strategy in which the outcome is independent of the forecast is equivalent to using randomly assigned probabilities but is...

20 March 2008 in Meteorological Applications
Verified
Journal Article
Open access
PDF file
SMSimon Mason
Simon Mason
Although there are numerous reasons for performing a verification analysis, there are usually two general questions that are of interest: are the forecasts good, and can we be confident that the estimate of forecast quality is not misleading? When calculating a verification score, it is not usually obvious how the score can answer either of these questions. Some procedures for attempting to answer the questions are reviewed, with particular focus onp‐values and confidence intervals.P‐values are shown to be rather unhelpful in answering either question, especially when applied to probabilistic verification scores, and confidence intervals are to be preferred. However, confidence intervals cannot reveal biases in the value of a score that arises from an inadequate experimental design for testing on truly out‐of‐sample observations. Some specific problems with cross validation are highlighted. Finally, in the interests of increasing the insight into forecast strengths and weaknesses and in pointing towards methods for improving forecast quality, a plea is made for a more discriminating selection of verification procedures than has been adopted to date. Copyright © 2008 Royal Meteorological Society

1 January 2009 in Monthly Weather Review
Verified
Journal Article
Closed access
SMSimon Mason
Simon Mason
AWAndreas P. Weigel
Andreas P. Weigel
There are numerous reasons for calculating forecast verification scores, and considerable attention has been given to designing and analyzing the properties of scores that can be used for scientific purposes. Much less attention has been given to scores that may be useful for administrative reasons, such as communicating changes in forecast quality to bureaucrats and providing indications of forecast quality to the general public. The two-alternative forced choice (2AFC) test is proposed as a scoring procedure that is sufficiently generic to be usable on forecasts ranging from simple yes–no forecasts of dichotomous outcomes to forecasts of continuous variables, and can be used with deterministic or probabilistic forecasts without seriously reducing the more complex information when available. Although, as with any single verification score, the proposed test has limitations, it does have broad intuitive appeal in that the expected score of an unskilled set of forecasts (random guessing or perpetually identical forecasts) is 50%, and is interpretable as an indication of how often the forecasts are correct, even when the forecasts are expressed probabilistically and/or the observations are not discrete.

1 January 2004 in Malaria Journal
Verified
Journal Article
Open access
PDF file
JDJoaquim DaSilva
Joaquim DaSilva
BGBrad Garanganga
Brad Garanganga
VTVonai Teveredzi
Vonai Teveredzi
SMSabine M Marx
Sabine M Marx
SMSimon Mason
Simon Mason
SCStephen J Connor
Stephen J Connor
Malaria is a major public health problem for countries in the Southern Africa Development Community (SADC). While the endemicity of malaria varies enormously across this region, many of the countries have districts that are prone to periodic epidemics, which can be regional in their extent, and to resurgent outbreaks that are much more localized. These epidemics are frequently triggered by climate anomalies and often follow periods of drought. Many parts of Southern Africa have suffered rainfall deficit over the past three years and countries expect to see increased levels of malaria when the rains return to more 'normal' levels. Problems with drug and insecticide resistance are documented widely and the region contains countries with the highest rates of HIV prevalence to be found anywhere in the world. Consequently, many communities are vulnerable to severe disease outcomes should epidemics occur. The SADC countries have adopted the Abuja targets for Roll Back Malaria in Africa, which include improved epidemic detection and response, i.e., that 60% of epidemics will be detected within two weeks of onset, and 60% of epidemics will be responded to within two weeks of detection. The SADC countries recognize that to achieve these targets they need improved information on where and when to look for epidemics. The WHO integrated framework for improved early warning and early detection of malaria epidemics has been recognized as a potentially useful tool for epidemic preparedness and response planning. Following evidence of successful adoption and implementation of this approach in Botswana, the SADC countries, the WHO Southern Africa Inter-Country Programme on Malaria Control, and the SADC Drought Monitoring Centre decided to organize a regional meeting where countries could gather to assess their current control status and community vulnerability, consider changes in epidemic risk, and develop a detailed plan of action for the forthcoming 2004–2005 season. The following is a report on the 1st Southern African Regional Epidemic Outlook Forum, which was held in Harare, Zimbabwe, 26th–29th September, 2004.

1 January 2004 in Climatic Change
Verified
Journal Article
Closed access
PDF file
SMSimon Mason
Simon Mason

1 February 2007 in Monthly Weather Review
Verified
Journal Article
Open access
SMSimon Mason
Simon Mason
JGJacqueline S. Galpin
Jacqueline S. Galpin
LGLisa Goddard
Lisa Goddard
NGNicholas E. Graham
Nicholas E. Graham
BRBalakanapathy Rajartnam
Balakanapathy Rajartnam
Probabilistic forecasts of variables measured on a categorical or ordinal scale, such as precipitation occurrence or temperatures exceeding a threshold, are typically verified by comparing the relative frequency with which the target event occurs given different levels of forecast confidence. The degree to which this conditional (on the forecast probability) relative frequency of an event corresponds with the actual forecast probabilities is known as reliability, or calibration. Forecast reliability for binary variables can be measured using the Murphy decomposition of the (half) Brier score, and can be presented graphically using reliability and attributes diagrams. For forecasts of variables on continuous scales, however, an alternative measure of reliability is required. The binned probability histogram and the reliability component of the continuous ranked probability score have been proposed as appropriate verification procedures in this context, but are subject to some limitations. A procedure is proposed that is applicable in the context of forecast ensembles and is an extension of the binned probability histogram. Individual ensemble members are treated as estimates of quantiles of the forecast distribution, and the conditional probability that the observed precipitation, for example, exceeds the amount forecast [the conditional exceedance probability (CEP)] is calculated. Generalized linear regression is used to estimate these conditional probabilities. A diagram showing the CEPs for ranked ensemble members is suggested as a useful method for indicating reliability when forecasts are on a continuous scale, and various statistical tests are suggested for quantifying the reliability.