People commonly assume that pain is universally displayed, yet there are inconsistencies in pain expression and recognition from facial expressions, with culture, sex, and perceptual context of importance. In this study, we adopted a data-driven approach to test whether pain shows perceptual similarity to other commonly studied emotion categories, and whether this varies across ethnicity, sex, and age of the observer. We also examined whether perceptual context (including internalizing problems, emotion regulation strategies, emotional states, self-esteem, and personality traits) were associated with identifying pain from facial animations. Our results show that with no contextual information provided, participants were more likely to identify anger or disgust rather than pain, with differences emerging regarding ethnicity and sex of both observers’ and animations. In addition, a variety of perceptual context variables were associated with identifying pain rather than any other emotion from the animated videos. For White male, Asian female, and Black female animations, ethnicity was significantly associated with identifying pain rather than any other emotion, whereas White participants were less likely in general to identify pain compared to participants in the other ethnicity groups. A greater variety of perceptual context variables predicted identifying pain from the Black male and female animations particularly. These findings suggest the importance of considering cultural and perceptual context factors associated with pain recognition from facial expressions.