Significance: This study measures face identification accuracy for an international group of professional forensic facial examiners working under circumstances that apply in real world casework. Examiners and other human face “specialists,” including forensically trained facial reviewers and untrained superrecognizers, were more accurate than the control groups on a challenging test of face identification. Therefore, specialists are the best available human solution to the problem of face identification. We present data comparing state-of-the-art face recognition technology with the best human face identifiers. The best machine performed in the range of the best humans: professional facial examiners. However, optimal face identification was achieved only when humans and machines worked in collaboration.
Professional forensic face examiners surpass untrained individuals on challenging face-identity matching tasks. We investigated qualitative/strategic differences in how forensic face examiners versus untrained people perform identity-matching tasks by analysing item responses (ratings of the likelihood that two images show the same person). We developed a novel analysis for quantifying item difficulty for participant groups and establishing group “winners” for items in conditions of interest. “Wisdom-of-the-crowds” effects were explored by fusing responses from varying numbers of participants to amplify strategic differences across groups. Results indicated that examiners use the internal face more effectively than untrained participants, but failed to exploit identity information in the external face and body. We also show that accuracy measures for examiners and controls must include both same-identity verifications and different-identity rejections to understand the role of perceptual skill and response bias in performance differences across participant groups.
We evaluated the detailed, behavioral properties of face matching performance in two specialist groups: forensic facial examiners and super‐recognizers. Both groups compare faces to determine identity with high accuracy and outperform the general population. Typically, facial examiners are highly trained; super‐recognizers rely on natural ability. We found distinct behaviors between these two groups. Examiners used the full 7‐point identity judgment scale (−3: “different”; +3: “same”). Super‐recognizers' judgments clustered toward highly confident decisions. Examiners' judgments for same‐ and different‐identities were symmetric across the scale midpoint (0); super‐recognizers' judgments were not. Examiners showed higher identity judgment agreement than super‐recognizers. Despite these qualitative differences, both groups showed insight into their own accuracy: more confident people and those who rated the task to be easier tended to be more accurate. Altogether, we show to better understand and interpret judgments according to the nature of someone's facial expertise, evaluations should assess more than accuracy.
The interest in forensic techniques capable of detecting many different media manipulation types has been growing, and system development with machine learning technology has been evolving in recent years. There has been, however, a lack of diversity in the data collections and in the evaluation methodologies for advancing multimedia forensics technologies. For the forensics research community, a well-defined evaluation is necessary to rapidly measure the accuracy and robustness of systems over diverse datasets collected under various environments. In this paper, we propose an evaluation framework and associated performance metrics and apply them to the 2018 Multimedia Forensics Challenge (MFC18). This MFC18 evaluation consists of five tasks and two challenges. A large number of datasets were created to support each task and for conducting the experiments using a structured evaluation framework. A total of 25 teams participated in the MFC18 evaluation; we analyse their performance on the tasks and challenges, and provide performance rankings for each team’s best-performing system.
While face recognition algorithms perform under many different unconstrained conditions, predicting this performance is not possible when a new location is introduced. Analyzing the impostor distribution of the videos of the Point-and-Shoot Challenge (PaSC) as well as its relationship to the genuine match distribution, we present a method for predicting the performance of an algorithm using only unlabeled data for a new location.