Burcu Sayin

Burcu Sayin

Auto-generated profile, based on publicly available information

Burcu Sayin

University of Trento

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
30 May 2021 in Artificial Intelligence Review
Verified
Journal Article
Open access
PDF file
BSBurcu Sayin
Burcu Sayin
EKEvgeny Krivosheev
Evgeny Krivosheev
JYJie Yang
Jie Yang
APAndrea Passerini
Andrea Passerini
FCFabio Casati
Fabio Casati
Training data creation is increasingly a key bottleneck for developing machine learning, especially for deep learning systems. Active learning provides a cost-effective means for creating training data by selecting the most informative instances for labeling. Labels in real applications are often collected from crowdsourcing, which engages online crowds for data labeling at scale. Despite the importance of using crowdsourced data in the active learning process, an analysis of how the existing active learning approaches behave over crowdsourced data is currently missing. This paper aims to fill this gap by reviewing the existing active learning approaches and then testing a set of benchmarking ones on crowdsourced datasets. We provide a comprehensive and systematic survey of the recent research on active learning in the hybrid human–machine classification setting, where crowd workers contribute labels (often noisy) to either directly classify data instances or to train machine learning models. We identify three categories of state of the art active learning methods according to whether and how predefined queries employed for data sampling, namely fixed-strategy approaches, dynamic-strategy approaches, and strategy-free approaches. We then conduct an empirical study on their cost-effectiveness, showing that the performance of the existing active learning approaches is affected by many factors in hybrid classification contexts, such as the noise level of data, label fusion technique used, and the specific characteristics of the task. Finally, we discuss challenges and identify potential directions to design active learning strategies for hybrid classification problems.

4 April 2025
Verified
Book Chapter
Closed access
PDF file
ISIpek Baris Schlicht
Ipek Baris Schlicht
ZZZhixue Zhao
Zhixue Zhao
BSBurcu Sayin
Burcu Sayin
LFLucie Flek
Lucie Flek
PRPaolo Rosso
Paolo Rosso

9 May 2025 in IEEE Access
Verified
Journal Article
Open access
BSBurcu Sayin
Burcu Sayin
TOMMASO ZOPPITOMMASO ZOPPI
TOMMASO ZOPPI
NMNicolò Marchini
Nicolò Marchini
FKFahad Ahmed Khokhar
Fahad Ahmed Khokhar
APAndrea Passerini
Andrea Passerini
Domain experts are striving to solve classification problems through the application of Machine Learning (ML) models, aiming for the highest accuracy. However, classifiers are inherently prone to misclassifications, especially when encountering unknown input data. This complicates their deployment in critical Cyber-Physical Systems (CPSs), where misclassifications can have detrimental effects on people, the environment, or infrastructure. This paper argues that ML classifiers should not be designed or evaluated in isolation but rather conceptualized as Critical System Classifiers (CSCs), which can reject uncertain predictions and trigger mitigation strategies in accordance to the encompassing CPS. We present an high-level architecture for CSCs that supports black-box integration, preprocessing for unknown detection, post-hoc calibration, and cost-sensitive thresholding, emphasizing the importance of evaluating classifiers using cost-aware metrics that account for rejections. We validate our proposal through experiments on tabular datasets related to failure prediction, intrusion and error detection, which are common applications of classifiers within CPSs. Key findings are: i) cost-sensitive evaluation promotes the usage of different classifiers compared to standard performance metrics, ii) tree-based models outperform statistical ones, classification-wise, iii) prediction calibration and rejection strategies provide a robust notion of confidence, and iv) combining multiple uncertainty-based rejectors achieves a favorable balance between high accuracy, low rejection rate, and cost. The analyses above were conducted using our publicly available CINNABAR GitHub repository for experimentations with CSCs. Overall, our study offers a system-level perspective, software architecture and implementation on classifier deployment in critical CPSs domains, paving the way for a trustworthy ML integration into real-world infrastructures.

27 March 2023
Verified
Conference Paper
Open access
SNShahin Sharifi Noorian
Shahin Sharifi Noorian
SQSihang Qiu
Sihang Qiu
BSBurcu Sayin
Burcu Sayin
ABAgathe Balayn
Agathe Balayn
UGUjwal Gadiraju
Ujwal Gadiraju
ABAlessandro Bozzon
Alessandro Bozzon
High-quality data plays a vital role in developing reliable image classification models. Despite that, what makes an image difficult to classify remains an unstudied topic. This paper provides a first-of-its-kind, model-agnostic characterization of image atypicality based on human understanding. We consider the setting of image classification “in the wild”, where a large number of unlabeled images are accessible, and introduce a scalable and effective human computation approach for proactive identification and characterization of atypical images. Our approach consists of i) an image atypicality identification and characterization task that presents to the human worker both a local view of visually similar images and a global view of images from the class of interest and ii) an automatic image sampling method that selects a diverse set of atypical images based on both visual and semantic features. We demonstrate the effectiveness and cost-efficiency of our approach through controlled crowdsourcing experiments and provide a characterization of image atypicality based on human annotations of 10K images. We showcase the utility of the identified atypical images by testing state-of-the-art image classification services against such images and provide an in-depth comparative analysis of the alignment between human- and machine-perceived image atypicality. Our findings have important implications for developing and deploying reliable image classification systems.

8 May 2025 in Artificial Intelligence Review
Verified
Journal Article
Open access
PDF file
BSBurcu Sayin
Burcu Sayin
Jie YangJie Yang
Jie Yang
XCXinyue Chen
Xinyue Chen
APAndrea Passerini
Andrea Passerini
FCFabio Casati
Fabio Casati
In this paper, we argue that the prevailing approach to training and evaluating machine learning models often fails to consider their real-world application within organizational or societal contexts, where they are intended to create beneficial value for people. We propose a shift in perspective, redefining model assessment and selection to emphasize integration into workflows that combine machine predictions with human expertise, particularly in scenarios requiring human intervention for low-confidence predictions. Traditional metrics like accuracy and f-score fail to capture the beneficial value of models in such hybrid settings. To address this, we introduce a simple yet theoretically sound “value” metric that incorporates task-specific costs for correct predictions, errors, and rejections, offering a practical framework for real-world evaluation. Through extensive experiments, we show that existing metrics fail to capture real-world needs, often leading to suboptimal choices in terms of value when used to rank classifiers. Furthermore, we emphasize the critical role of calibration in determining model value, showing that simple, well-calibrated models can often outperform more complex models that are challenging to calibrate.

22 June 2023
Verified
Book Chapter
Open access
PDF file
BSBurcu Sayin
Burcu Sayin
JYJie Yang
Jie Yang
APAndrea Passerini
Andrea Passerini
FCFabio Casati
Fabio Casati
In this paper, we argue that the way we have been training and evaluating ML models has largely forgotten the fact that they are applied in an organization or societal context as they provide value to people. We show that with this perspective we fundamentally change how we evaluate and select machine learning models.