Humans physically express emotion by modulating parameters that register on mammalian skin mechanoreceptors, but are unavailable in current touch-sensing technology. Greater sensory richness combined with data on affect-expression composition is a prerequisite to estimating affect from touch, with applications including physical human-robot interaction. To examine shear alongside more easily captured normal stresses, we tailored recent capacitive technology to attain performance suitable for affective touch, creating a flexible, reconfigurable and soft 36-taxel array that detects multitouch normal and 2-dimensional shear at ranges of 1.5kPa-43kPa and ± 0.3-3.8kPa respectively, wirelessly at 43Hz (1548 taxels/s). In a deep-learning classification of 9 gestures (N=16), inclusion of shear data improved accuracy to 88%, compared to 80% with normal stress data alone, confirming shear stress’s expressive centrality. Using this rich data, we analyse the interplay of sensed-touch features, gesture attributes and individual differences, propose affective-touch sensing requirements, and share technical considerations for performance and practicality.
In-body lived emotional experiences can be complex, with time-varying and dissonant emotions evolving simultaneously; devices responding in real-time to estimate personal human emotion should evolve accordingly. Models assuming generalized emotions exist as discrete states fail to operationalize valuable information inherent in the dynamic and individualistic nature of human emotions. Our multi-resolution emotion self-reporting procedure allows the construction of emotion labels along the Stressed-Relaxed scale, differentiating not only what the emotions are, but how they are transitioning - e.g., "hopeful but getting stressed" vs. "hopeful and starting to relax". We trained participant-dependent hierarchical models of contextualized individual experience to compare emotion classification by modality (brain activity and keypress force from a physical keyboard), then benchmarked classification performance at F1-scores = [0.44, 0.82] (chance F1=0.22, sigma = 0.01) and examined high-performing features. Notably, when classifying emotion evolution in the context of an experience that realistically varies in stress, pressure-based features from keypress force proved to be the more informative modality, and more convenient when considering intrusiveness and ease of collection and processing. Finally, we present our FEEL (Force, EEG and Emotion-Labelled) dataset, a collection of brain activity and keypress force data, labelled with self-reported emotion collected during tense videogame play (N = 16) and open-sourced for community exploration.
Naturalistic human touch expression can be an emotionally potent modality, and promising as an informative but unobtrusive input into machine-learning models of affect. However, application-ready emotion-aware technologies must be trained on labeled samples of authentic (felt) emotion of a significant range of intensity and valence. These may come at significant, even traumatic, personal cost for any modality. We examined (a) performance of the novel touch modality, and (b) how a care-centered protocol might manage personal burden. Participants (N=5; 3 team members), shared autobiographical stories that elicited powerful emotional dynamics, in 1-3 sessions each (total 10). During storytelling, incidental touch was captured on a pillow-mounted custom flexible 10×10-taxel pressure sensor, alongside physiological signals; then labeled with multiple passes of rich, multimodal self-reports. Protocol, study and analysis design prioritized reflexivity and participant experience. Accuracy: Participant-specific, touch-only models predicted emotion direction (trajectory slope) with 65.2$\pm$16.3% accuracy (2s windows; chance 25%, physiology-only models 64.1$\pm$16.9%), confirming the value of this unobtrusivechannel. Personal cost: Qualitative analysis contributed an extensive picture of the emotional toll of generating such data, but also some benefits. We offer recommendations for sustainable ethical sourcing of affective data which balance personalization, performance, therapeutic insight and participant care.