As a concept, social vulnerability describes combinations of social, cultural, economic, political, and institutional processes that shape socioeconomic differentials in the experience of and recovery from hazards. Quantitative measures of social vulnerability are widely used in research and practice. In this paper, we establish criteria for the evaluation of social vulnerability indicators and apply those criteria to the most widely used measure of social vulnerability, the Social Vulnerability Index (SoVI). SoVI is a single quantitative indicator that purports to measure a place’s social vulnerability. We show that SoVI has some critical shortcomings regarding theoretical and internal consistency. Specifically, multiple SoVI-based measurements of the vulnerability of the same place, using the same data, can yield strikingly different results. We also show that the SoVI is often misaligned with theory; increases in variables that contribute to vulnerability, like the unemployment rate, often decrease vulnerability as measured by the SoVI. We caution against the use of the index in policy making or other risk-reduction efforts, and we suggest ways to more reliably assess social vulnerability in practice.

Much of the evaluation and tuning of a search system relies on relevance labels---annotations that say whether a document is useful for a given search and searcher. Ideally these come from real searchers, but it is hard to collect this data at scale, so typical experiments rely on third-party labellers who may or may not produce accurate annotations. Label quality is managed with ongoing auditing, training, and monitoring. We discuss an alternative approach. We take careful feedback from real searchers and use this to select a large language model (LLM), and prompt, that agrees with this feedback; the LLM can then produce labels at scale. Our experiments show LLMs are as accurate as human labellers and as useful for finding the best systems and hardest queries. LLM performance varies with prompt features, but also varies unpredictably with simple paraphrases. This unpredictability reinforces the need for high-quality "gold" labels.

Social science research, public and private sector decisions, and allocations of federal resources often rely on data from the American Community Survey (ACS). However, this critical data source has high uncertainty in some of its most frequently used estimates. Using 2006–2010 ACS median household income estimates at the census tract scale as a test case, we explore spatial and nonspatial patterns in ACS estimate quality. We find that spatial patterns of uncertainty in the northern United States differ from those in the southern United States, and they are also different in suburbs than in urban cores. In both cases, uncertainty is lower in the former than the latter. In addition, uncertainty is higher in areas with lower incomes. We use a series of multivariate spatial regression models to describe the patterns of association between uncertainty in estimates and economic, demographic, and geographic factors, controlling for the number of responses. We find that these demographic and geographic patterns in estimate quality persist even after we account for the number of responses. Our results indicate that data quality varies across places, making cross-sectional analysis both within and across regions less reliable. Finally, we present advice for data users and potential solutions to the challenges identified.