Silvan Mertes

Silvan Mertes

Auto-generated profile, based on publicly available information

Silvan Mertes

Technical University of Applied Sciences Augsburg, Augsburg, Germany

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
8 April 2022 in Frontiers in Artificial Intelligence
Verified
Journal Article
Open access
SMSilvan Mertes
Silvan Mertes
THTobias Huber
Tobias Huber
KWKatharina Weitz
Katharina Weitz
AHAlexander Heimerl
Alexander Heimerl
EAElisabeth André
Elisabeth André
With the ongoing rise of machine learning, the need for methods for explaining decisions made by artificial intelligence systems is becoming a more and more important topic. Especially for image classification tasks, many state-of-the-art tools to explain such classifiers rely on visual highlighting of important areas of the input data. Contrary, counterfactual explanation systems try to enable a counterfactual reasoning by modifying the input image in a way such that the classifier would have made a different prediction. By doing so, the users of counterfactual explanation systems are equipped with a completely different kind of explanatory information. However, methods for generating realistic counterfactual explanations for image classifiers are still rare. Especially in medical contexts, where relevant information often consists of textural and structural information, high-quality counterfactual images have the potential to give meaningful insights into decision processes. In this work, we present GANterfactual, an approach to generate such counterfactual image explanations based on adversarial image-to-image translation techniques. Additionally, we conduct a user study to evaluate our approach in an exemplary medical use case. Our results show that, in the chosen medical use-case, counterfactual explanations lead to significantly better results regarding mental models, explanation satisfaction, trust, emotions, and self-efficacy than two state-of-the art systems that work with saliency maps, namely LIME and LRP.

10 March 2023 in Proceedings of the IEEE
Verified
Journal Article
Closed access
ATAndreas Triantafyllopoulos
Andreas Triantafyllopoulos
BSBjörn W. Schuller
Björn W. Schuller
GIGökçe İymen
Gökçe İymen
MSMetin Sezgin
Metin Sezgin
XHXiangheng He
Xiangheng He
TJTao Jianhua
Tao Jianhua
Speech is the fundamental mode of human communication, and its synthesis has long been a core priority in human–computer interaction research. In recent years, machines have managed to master the art of generating speech that is understandable by humans. However, the linguistic content of an utterance encompasses only a part of its meaning. Affect, or expressivity, has the capacity to turn speech into a medium capable of conveying intimate thoughts, feelings, and emotions—aspects that are essential for engaging and naturalistic interpersonal communication. While the goal of imparting expressivity to synthesized utterances has so far remained elusive, following recent advances in text-to-speech synthesis, a paradigm shift is well under way in the fields of affective speech synthesis and conversion as well. Deep learning, as the technology that underlies most of the recent advances in artificial intelligence, is spearheading these efforts. In this overview, we outline ongoing trends and summarize state-of-the-art approaches in an attempt to provide a broad overview of this exciting field.

19 April 2023
Verified
Conference Paper
Open access
RSRuben Schlagowski
Ruben Schlagowski
DNDariia Nazarenko
Dariia Nazarenko
YCYekta Can
Yekta Can
KGKunal Gupta
Kunal Gupta
SMSilvan Mertes
Silvan Mertes
EAElisabeth André
Elisabeth André
With face-to-face music collaboration being severely limited during the recent pandemic, mixed reality technologies and their potential to provide musicians a feeling of "being there" with their musical partner can offer tremendous opportunities. In order to assess this potential, we conducted a laboratory study in which musicians made music together in real-time while simultaneously seeing their jamming partner’s mixed reality point cloud via a head-mounted display and compared mental effects such as flow, affect, and co-presence to an audio-only baseline. In addition, we tracked the musicians’ physiological signals and evaluated their features during times of self-reported flow. For users jamming in mixed reality, we observed a significant increase in co-presence. Regardless of the condition (mixed reality or audio-only), we observed an increase in positive affect after jamming remotely. Furthermore, we identified heart rate and HF/LF as promising features for classifying the flow state musicians experienced while making music together.

14 December 2024 in ACM Transactions on Multimedia Computing, Communications, and Applications
Verified
Journal Article
Open access
FHFabio Hellmann
Fabio Hellmann
SMSilvan Mertes
Silvan Mertes
BMBenouis Mohamed
Benouis Mohamed
AHAlexander Hustinx
Alexander Hustinx
THTzung-Chien Hsieh
Tzung-Chien Hsieh
EAElisabeth André
Elisabeth André
In recent years, the increasing availability of personal data has raised concerns regarding privacy and security. One of the critical processes to address these concerns is data anonymization, which aims to protect individual privacy and prevent the release of sensitive information. This research focuses on the importance of face anonymization. Therefore, we introduce GANonymization, a novel face anonymization framework with facial expression-preserving abilities. Our approach is based on a high-level representation of a face, which is synthesized into an anonymized version based on a generative adversarial network (GAN). The effectiveness of the approach was assessed by evaluating its performance in removing identifiable facial attributes to increase the anonymity of the given individual face. Additionally, the performance of preserving facial expressions was evaluated on several affect recognition datasets and outperformed the state-of-the-art methods in most categories. Finally, our approach was analyzed for its ability to remove various facial traits, such as jewelry, hair color, and multiple others. Here, it demonstrated reliable performance in removing these attributes. Our results suggest that GANonymization is a promising approach for anonymizing faces while preserving facial expressions.

18 November 2024 in IEEE Transactions on Affective Computing
Verified
Journal Article
Open access
AHAlexander Heimerl
Alexander Heimerl
PPPooja Prajod
Pooja Prajod
SMSilvan Mertes
Silvan Mertes
TBTobias Baur
Tobias Baur
MKMatthias Kraus
Matthias Kraus
LBLinda Becker
Linda Becker
We present a multi-modal stress dataset that uses digital job interviews to induce stress. The dataset provides multi-modal data of 40 participants including audio, video (motion capturing, facial landmarks, eye tracking) as well as physiological information (photoplethysmography, electrodermal activity). In addition to that, the dataset contains time-continuous annotations for stress and occurred emotions (e.g., shame, anger, anxiety, and surprise). In order to establish a baseline, five different machine learning classifiers (Support Vector Machine, K-Nearest Neighbors, Random Forest, Feed-forward Neural Network, and Long-Short-Term Memory Network) have been trained and evaluated on the presented dataset for a binary stress classification task. The best-performing classifier has been a Long-Short-Term Memory Network, which achieved an accuracy of 91.7% and an F1-score of 90.2%. The ForDigitStress dataset is freely available to other researchers.

11 May 2024
Verified
Conference Paper
Open access
PRPol van Rijn
Pol van Rijn
SMSilvan Mertes
Silvan Mertes
KJKathrin Janowski
Kathrin Janowski
KWKatharina Weitz
Katharina Weitz
NJNori Jacoby
Nori Jacoby
EAElisabeth André
Elisabeth André
Speech is a natural interface for humans to interact with robots. Yet, aligning a robot’s voice to its appearance is challenging due to the rich vocabulary of both modalities. Previous research has explored a few labels to describe robots and tested them on a limited number of robots and existing voices. Here, we develop a robot-voice creation tool followed by large-scale behavioral human experiments (N=2,505). First, participants collectively tune robotic voices to match 175 robot images using an adaptive human-in-the-loop pipeline. Then, participants describe their impression of the robot or their matched voice using another human-in-the-loop paradigm for open-ended labeling. The elicited taxonomy is then used to rate robot attributes and to predict the best voice for an unseen robot. We offer a web interface to aid engineers in customizing robot voices, demonstrating the synergy between cognitive science and machine learning for engineering tools.