Anca Dumitrache

Anca Dumitrache

Auto-generated profile, based on publicly available information

Anca Dumitrache

VU University Amsterdam

Research Focus & Metrics

Academic Career Length

16Years (2011-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
30 June 2018 in ACM Transactions on Interactive Intelligent Systems
Verified
Journal Article
Open access
PDF file
ADAnca Dumitrache
Anca Dumitrache
LALora Aroyo
Lora Aroyo
CWChris Welty
Chris Welty
Cognitive computing systems require human labeled data for evaluation and often for training. The standard practice used in gathering this data minimizes disagreement between annotators, and we have found this results in data that fails to account for the ambiguity inherent in language. We have proposed the CrowdTruth method for collecting ground truth through crowdsourcing, which reconsiders the role of people in machine learning based on the observation that disagreement between annotators provides a useful signal for phenomena such as ambiguity in the text. We report on using this method to build an annotated data set for medical relation extraction for the cause and treat relations, and how this data performed in a supervised training experiment. We demonstrate that by modeling ambiguity, labeled data gathered from crowd workers can (1) reach the level of quality of domain experts for this task while reducing the cost, and (2) provide better training data at scale than distant supervision. We further propose and validate new weighted measures for precision, recall, and F-measure, which account for ambiguity in both human and machine performance on this task.

7 July 2020
Verified
Conference Paper
Open access
FLFeng Lu
Feng Lu
ADAnca Dumitrache
Anca Dumitrache
DGDavid Graus
David Graus
With the uptake of algorithmic personalization in the news domain, news organizations increasingly trust automated systems with previously considered editorial responsibilities, e.g., prioritizing news to readers. In this paper we study an automated news recommender system in the context of a news organization's editorial values. We conduct and present two online studies with a news recommender system, which span one and a half months and involve over 1,200 users. In our first study we explore how our news recommender steers reading behavior in the context of editorial values such as serendipity, dynamism, diversity, and coverage. Next, we present an intervention study where we extend our news recommender to steer our readers to more dynamic reading behavior. We find that (i) our recommender system yields more diverse reading behavior and yields a higher coverage of articles compared to non-personalized editorial rankings, and (ii) we can successfully incorporate dynamism in our recommender system as a re-ranking method, effectively steering our readers to more dynamic articles without hurting our recommender system's accuracy.

1 January 2014
Verified
Book Chapter
Closed access
PDF file
OIOana Inel
Oana Inel
KKKhalid Khamkham
Khalid Khamkham
TCTatiana Cristea
Tatiana Cristea
ADAnca Dumitrache
Anca Dumitrache
ARArne Rutjes
Arne Rutjes
RSRobert-Jan Sips
Robert-Jan Sips

1 January 2019
Verified
Conference Paper
Open access
ADAnca Dumitrache
Anca Dumitrache
LALora Aroyo
Lora Aroyo
CWChris Welty
Chris Welty


9 March 2021 in Semantic Web: – Interoperability, Usability, Applicability
Verified
Journal Article
Open access
PDF file
ADAnca Dumitrache
Anca Dumitrache
OIOana Inel
Oana Inel
BTBenjamin Timmermans
Benjamin Timmermans
COCarlos Ortíz
Carlos Ortíz
RSRobert-Jan Sips
Robert-Jan Sips
CWChris Welty
Chris Welty
The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods for populating the Semantic Web. Crowdsourcing-based approaches are gaining popularity in the attempt to solve the issues related to volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a measure of quality. However, in many domains, such as event detection, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. We present an empirically derived methodology for efficiently gathering of ground truth data in a diverse set of use cases covering a variety of domains and annotation tasks. Central to our approach is the use of CrowdTruth metrics that capture inter-annotator disagreement. We show that measuring disagreement is essential for acquiring a high quality ground truth. We achieve this by comparing the quality of the data aggregated with CrowdTruth metrics with majority vote, over a set of diverse crowdsourcing tasks: Medical Relation Extraction, Twitter Event Identification, News Event Extraction and Sound Interpretation. We also show that an increased number of crowd workers leads to growth and stabilization in the quality of annotations, going against the usual practice of employing a small number of annotators.