Koichiro Niinuma

Koichiro Niinuma

Auto-generated profile, based on publicly available information

Koichiro Niinuma

Fujitsu Research of America, Pittsburgh, Pennsylvania, USA

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 October 2024
Verified
Book Chapter
Closed access
PDF file
RCRohan Choudhury
Rohan Choudhury
KNKoichiro Niinuma
Koichiro Niinuma
KKKris M. Kitani
Kris M. Kitani
LJLászló A. Jeni
László A. Jeni


27 April 2022 in Frontiers in Signal Processing
Verified
Journal Article
Open access
KNKoichiro Niinuma
Koichiro Niinuma
IEItir Onal Ertugrul
Itir Onal Ertugrul
JCJeffrey F. Cohn
Jeffrey F. Cohn
LJLászló A. Jeni
László A. Jeni
Limited sizes of annotated video databases of spontaneous facial expression, imbalanced action unit labels, and domain shift are three main obstacles in training models to detect facial actions and estimate their intensity. To address these problems, we propose an approach that incorporates facial expression generation for facial action unit intensity estimation. Our approach reconstructs the 3D shape of the face from each video frame, aligns the 3D mesh to a canonical view, and trains a GAN-based network to synthesize novel images with facial action units of interest. We leverage the synthetic images to achieve two goals: 1) generating AU-balanced databases, and 2) tackling domain shift with personalized networks. To generate a balanced database, we synthesize expressions with varying AU intensities and perform semantic resampling. Our experimental results on FERA17 show that networks trained on synthesized facial expressions outperform those trained on actual facial expressions and surpass current state-of-the-art approaches. To tackle domain shift, we propose personalizing pretrained networks. We generate synthetic expressions of each target subject with varying AU intensity labels and use the person-specific synthetic images to fine-tune pretrained networks. To evaluate performance of the personalized networks, we use DISFA and PAIN databases. Personalized networks, which require only a single image from each target subject to generate synthetic images, achieved significant improvement in generalizing to unseen domains.

1 January 2021
Verified
Conference Paper
Closed access
KNKoichiro Niinuma
Koichiro Niinuma
IEItir Onal Ertugrul
Itir Onal Ertugrul
JCJeffrey F Cohn
Jeffrey F Cohn
LJLaszlo A Jeni
Laszlo A Jeni
Critical obstacles in training classifiers to detect facial actions are the limited sizes of annotated video databases and the relatively low frequencies of occurrence of many actions. To address these problems, we propose an approach that makes use of facial expression generation. Our approach reconstructs the 3D shape of the face from each video frame, aligns the 3D mesh to a canonical view, and then trains a GAN-based network to synthesize novel images with facial action units of interest. To evaluate this approach, a deep neural network was trained on two separate datasets: One network was trained on video of synthesized facial expressions generated from FERA17; the other network was trained on unaltered video from the same database. Both networks used the same train and validation partitions and were tested on the test partition of actual video from FERA17. The network trained on synthesized facial expressions outperformed the one trained on actual facial expressions and surpassed current state-of-the-art approaches.

29 April 2021 in Frontiers in Computer Science
Verified
Journal Article
Open access
KNKoichiro Niinuma
Koichiro Niinuma
IEItir Onal Ertugrul
Itir Onal Ertugrul
JCJeffrey F. Cohn
Jeffrey F. Cohn
LJLászló A. Jeni
László A. Jeni
The performance of automated facial expression coding is improving steadily. Advances in deep learning techniques have been key to this success. While the advantage of modern deep learning techniques is clear, the contribution of critical design choices remains largely unknown, especially for facial action unit occurrence and intensity across pose. Using the The Facial Expression Recognition and Analysis 2017 (FERA 2017) database, which provides a common protocol to evaluate robustness to pose variation, we systematically evaluated design choices in pre-training, feature alignment, model size selection, and optimizer details. Informed by the findings, we developed an architecture that exceeds state-of-the-art on FERA 2017. The architecture achieved a 3.5% increase in F1 score for occurrence detection and a 5.8% increase in Intraclass Correlation (ICC) for intensity estimation. To evaluate the generalizability of the architecture to unseen poses and new dataset domains, we performed experiments across pose in FERA 2017 and across domains in Denver Intensity of Spontaneous Facial Action (DISFA) and the UNBC Pain Archive.