Some researchers have been arguing that eyewitness identification data from lineups should be analyzed using Receiver Operating Characteristic (ROC) analysis because it purportedly measures underlying discriminability. But ROC analysis, which was designed for 2 × 2 tasks, does not fit the 3 × 2 structure of lineups. Accordingly, ROC proponents force lineup data into a 2 × 2 structure by treating false-positive identifications of lineup fillers as though they were rejections. Using data from lineups versus showups, we illustrate how this approach misfires as a measure of underlying discriminability. Moreover, treating false-positive identifications of fillers as if they were rejections hides one of the most important phenomena in eyewitness lineups, namely filler siphoning. Filler siphoning reduces the risk of mistaken identification by drawing false-positive identifications away from the innocent suspect and onto lineup fillers. We show that ROC analysis confuses filler siphoning with an improvement in underlying discriminability, thereby fostering misleading theoretical conclusions about how lineups work.
Receiver Operating Characteristic (ROC) analysis has recently come in vogue for assessing the underlying discriminability and the applied utility of lineup procedures. Two primary assumptions underlie recommendations that ROC analysis be used to assess the applied utility of lineup procedures: (a) ROC analysis of lineups measures underlying discriminability, and (b) the procedure that produces superior underlying discriminability produces superior applied utility. These same assumptions underlie a recently derived diagnostic-feature detection theory, a theory of discriminability, intended to explain recent patterns observed in ROC comparisons of lineups. We demonstrate, however, that these assumptions are incorrect when ROC analysis is applied to lineups. We also demonstrate that a structural phenomenon of lineups, differential filler siphoning, and not the psychological phenomenon of diagnostic-feature detection, explains why lineups are superior to showups and why fair lineups are superior to biased lineups. In the process of our proofs, we show that computational simulations have assumed, unrealistically, that all witnesses share exactly the same decision criteria. When criterial variance is included in computational models, differential filler siphoning emerges. The result proves dissociation between ROC curves and underlying discriminability: Higher ROC curves for lineups than for showups and for fair than for biased lineups despite no increase in underlying discriminability. (PsycINFO Database Record
Our previous article (Wells et al., 2015a. Journal of Applied Research in Memory and Cognition ) showed how ROC analysis of lineups does not measure underlying discriminability or control for response bias. Wixted and Mickes (2015. Journal of Applied Research in Memory and Cognition ) concede these points. Hence, in this article we focus more on how forcing the 3 × 2 lineup into the 2 × 2 structure required for ROC analysis obscures important underlying phenomena of theoretical value. Moreover, ROC analysis fails to account for the unique diagnostic properties of exonerating eyewitness behaviors (filler identifications and rejections). We describe how an examination of the full 3 × 2 structure helps reveal the critical underlying phenomena that ROC analysis hides. We also show how a Bayesian approach yields a family of diagnosticity functions that exposes the unique diagnosticity of all three eyewitness behaviors (suspect identifications, filler identifications, and rejections). Moreover, we show how Bayesian methods can examine diagnosticity as a function of witness confidence for all three eyewitness behaviors, which gives it a significant applied advantage over ROC analysis.
6 May 2020 in Perspectives on Psychological Science
The conceptual frameworks provided by both the lineups-as-experiments analogy and signal detection theory have proven important to understanding how eyewitness lineups work. The lineups-as-experiments analogy proposes that when investigators use a lineup procedure, they are acting as experimenters and should therefore follow the same tried-and-true procedures that experimenters follow when executing an experiment. Signal detection theory offers a framework for distinguishing between factors that improve the trade-off between culprit and innocent-suspect identifications and factors that affect the frequency of suspect identifications. We integrate these two conceptual frameworks. We argue that an eyewitness lineup procedure is characterized by two simultaneous signal detection tasks. On one hand, the witness is tasked with determining whether the culprit is present in the lineup and identifying that person. On the other hand, the investigator knows which lineup member is the suspect and which lineup members are known-innocent fillers and is therefore tasked only with determining whether the suspect is the culprit. The investigator uses the witness’s identification decision and associated level of confidence to decide whether the suspect is the culprit. We leverage this realization to demonstrate a method for creating full receiver operating characteristic curves for eyewitness lineup procedures.
17 September 2019 in Advances in Methods and Practices in Psychological Science
Scientific advances across a range of disciplines hinge on the ability to make inferences about unobservable theoretical entities on the basis of empirical data patterns. Accurate inferences rely on both discovering valid, replicable data patterns and accurately interpreting those patterns in terms of their implications for theoretical constructs. The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted to the validity of inferences based on those findings. Using an example from cognitive psychology, we demonstrate a blinded-inference paradigm for assessing the quality of theoretical inferences from data. Our results reveal substantial variability in experts’ judgments on the very same data, hinting at a possible inference crisis.