In the literature, numerous papers report comparative analyses of drought indices. In these types of studies, the similarity between drought indices is usually evaluated using the Pearson correlation coefficient, r, calculated between corresponding severity time series. However, it is well known that the correlation does not describe the strength of agreement between two variables. Two drought indices can exhibit a high degree of correlation but can, at the same time, disagree substantially, for example, if one index is consistently higher than the other. From an operational point of view, two indices can be considered in agreement when they indicate the same severity category for a given period (e.g. moderate drought). In this work, we compared six meteorological drought indices based on both correlation analysis and Cohen's Kappa test. This test is typically used in medical or social sciences to obtain a quantitative assessment of the degree of agreement between different methods or analysts. The indices considered are five timescale-dependent indices, i.e. the Percent of Normal Index, the Deciles Index, the Percentile Index, the Rainfall Anomaly Index, and the Standardised Precipitation Index, computed at the 1-, 3-, and 6-month timescales, and the Effective Drought Index, a relatively new index, which has a self-defined timescale. The indices were calculated for 15 stations in the Abruzzo region (central Italy) during 1951–2018. We found that the strength of agreement depends on both the criteria of drought severity classification and the different indices' calculation method. The Cohen's Kappa test indicates a prevailing moderate or fair agreement among the indices considered, despite the generally very high correlation between the corresponding severity times series. The results demonstrate that the Cohen's Kappa test is more effective than the correlation analysis in discriminating the actual strength of agreement/disagreement between drought indices.