Automated recognition of human facial expressions of pain and emotions is to a certain degree a solved problem, using approaches based on computer vision and machine learning. However, the application of such methods to horses has proven difficult. Major barriers are the lack of sufficiently large, annotated databases for horses and difficulties in obtaining correct classifications of pain because horses are non-verbal. This review describes our work to overcome these barriers, using two different approaches. One involves the use of a manual, but relatively objective, classification system for facial activity (Facial Action Coding System), where data are analyzed for pain expressions after coding using machine learning principles. We have devised tools that can aid manual labeling by identifying the faces and facial keypoints of horses. This approach provides promising results in the automated recognition of facial action units from images. The second approach, recurrent neural network end-to-end learning, requires less extraction of features and representations from the video but instead depends on large volumes of video data with ground truth. Our preliminary results suggest clearly that dynamics are important for pain recognition and show that combinations of recurrent neural networks can classify experimental pain in a small number of horses better than human raters.

Equine orthopedic pain scales are targeted towards horses with moderate to severe orthopedic pain. Improved assessment of pain behavior and pain-related facial expressions at rest may refine orthopedic pain detection for mild lameness grades. Therefore, this study explored pain-related behaviors and facial expressions and sought to identify frequently occurring combinations. Orthopedic pain was induced by intra-articular LPS in eight horses, and objective movement asymmetry analyses were performed before and after induction together with pain assessments at rest. Three observers independently assessed horses in their box stalls, using four equine pain scales simultaneously. Increase in movement asymmetry after induction was used as a proxy for pain. Behaviors and facial expressions commonly co-occurred and were strongly associated with movement asymmetry. Posture-related scale items were the strongest predictors of movement asymmetry. Display of facial expressions at rest varied between horses but, when present, were strongly associated with movement asymmetry. Reliability of facial expression items was lower than reliability of behavioral items. These findings suggest that five body behaviors (posture, head position, location in the box stall, focus, and interactive behavior) should be included in a scale for live assessment of mild orthopedic pain. We also recommend inclusion of facial expressions in pain assessment.

Objective: This study investigated the relationship between orthopedic pain experienced at rest, and degree of movement asymmetry during trot in horses with induced reversible acute arthritis. Orthopedic pain was assessed with the Horse Grimace Scale (HGS), the Equine Utrecht University Scale of Facial Assessment of Pain (EQUUS-FAP), the Equine Pain Scale (EPS), and the Composite Orthopedic Pain Scale (CPS). Reliability and diagnostic accuracy were evaluated with intraclass correlation coefficients (ICC) and area under the curve (AUC).Study design and animals: Eight healthy horses were included in this experimental study, with each horse acting as its own control.Methods: Orthopedic pain was induced by intra-articular lipopolysaccharide (LPS) administration. Serial pain assessments were performed before induction and during pain progression and regression, where three observers independently and simultaneously assessed pain at rest with the four scales. Movement asymmetry was measured once before induction and a minimum of four times after induction, using objective gait analysis.Results: On average 6.6 (standard deviation 1.2) objective gait analyses and 12.1 (2.4) pain assessments were performed per horse. The ICC for each scale was 0.75 (CPS), 0.65 (EPS), 0.52 (HGS), and 0.43 (EQUUS-FAP). Total pain scores of all scales were significantly associated with an increase in movement asymmetry (R²values ranging from −0.0649 to 0.493); with CPS pain scores being most closely associated with movement asymmetry. AUC varied between scales and observers, and CPS was the only scale where all observers had a good diagnostic accuracy (AUC > 0.72).Conclusions and clinical relevance: This study identified significant associations between pain experienced at rest and degree of movement asymmetry for all scales. Pain scores obtained using CPS were most closely associated with movement asymmetry. CPS was also the most accurate and reliable pain scale. All scales had varying linear and non-linear relations between total pain scores and movement asymmetry, illustrating challenges with orthopedic pain assessment during rest in subtly lame horses since movement asymmetry needs to be rather high before total pain score increase.

A number of facial expressions are associated with pain in horses, however, the entire display of facial activities during orthopedic pain have yet to be described. The aim of the present study was to exhaustively map changes in facial activities in eight resting horses during a progression from sound to mild and moderate degree of orthopedic pain, induced by lipopolysaccharides (LPS) administered in the tarsocrural joint. Lameness progression and regression was measured by objective gait analysis during movement, and facial activities were described by EquiFACS in video sequences (n = 348, total length 892.5 min) of the horses obtained when resting in their box stalls. Predictive modeling identified 16 action units and action descriptors, related to ears, eyes, and lower face. Lower lip depressor (AU16), lips part (AU25), half blink (AU47), single ear forward (SEAD101) and single ear rotator (SEAD104) were selected as co-occurring significantly more in horses with pain than in horses without pain. The major change in co-occurring facial activities occurred in the transition from no pain to mild pain. In conclusion, resting horses with induced orthopedic pain showed a dynamic upper and lower facial repertoire and the relationship between level of pain intensity and facial activity appears complex.

Orthopedic disorders are common among horses, often leading to euthanasia, which often could have been avoided with earlier detection. These conditions often create varying degrees of subtle long-term pain. It is challenging to train a visual pain recognition method with video data depicting such pain, since the resulting pain behavior also is subtle, sparsely appearing, and varying, making it challenging for even an expert human labeller to provide accurate ground-truth for the data. We show that a model trained solely on a dataset of horses with acute experimental pain (where labeling is less ambiguous) can aid recognition of the more subtle displays of orthopedic pain. Moreover, we present a human expert baseline for the problem, as well as an extensive empirical study of various domain transfer methods and of what is detected by the pain recognition method trained on clean experimental pain in the orthopedic dataset. Finally, this is accompanied with a discussion around the challenges posed by real-world animal behavior datasets and how best practices can be established for similar fine-grained action recognition tasks. Our code is available at https://github.com/sofiabroome/painface-recognition.

The prevalence of vertical asymmetries is high in “owner-sound” warmblood riding horses, however the origin of these asymmetries is unknown. This study investigated correlations between vertical asymmetries and motor laterality. Young warmblood riding horses (N = 65), perceived as free from lameness were evaluated on three visits, each comprising objective gait analysis (inertial measurement units system) and a rider questionnaire on perceived sidedness of the horse. A subgroup (N = 40) of horses were also subjected to a forelimb protraction preference test intended as an assessment of motor laterality. We hypothesized associations between vertical asymmetry and motor laterality as well as rider-perceived sidedness. Vertical asymmetry was quantified as trial means of the stride-by-stride difference between the vertical displacement minima and maxima of the head (HDmin, HDmax) and pelvis (PDmin, PDmax). Laterality indexes, based on counts of which limb was protracted, and binomial tests were used to draw conclusions from the preference tests. In the three visits, 60–70% of horses exhibited vertical asymmetries exceeding clinically used thresholds for ≥1 parameter, and 22% of horses exhibited a side preference in the preference test as judged by binomial tests. Linear mixed models identified a weak but statistically significant correlation between perceived hindlimb weakness and higher PDmin values attributable to either of the hindlimbs (p = 0.023). No other statistically significant correlations to vertical asymmetry were seen for any of the questionnaire answers tested. Tests of correlation between the absolute values of laterality index and asymmetry parameters (HDmin, HDmax, PDmin, PDmax) identified a weak correlation (p = 0.049) with PDmax, but when accounting for the direction of asymmetry and motor laterality, no correlations were seen for either of the asymmetry parameters. No convincing evidence of associations between vertical asymmetries and motor laterality were seen and further studies investigating motor laterality and the origin of vertical asymmetries are needed.