Rickard Stureborg

Rickard Stureborg

Auto-generated profile, based on publicly available information

Rickard Stureborg

Duke University

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
2 May 2024 in arXiv
Verified
Preprint
Open access
PDF file
RSRickard Stureborg
Rickard Stureborg
DADimitris Alikaniotis
Dimitris Alikaniotis
YSYoshi Suhara
Yoshi Suhara
The zero-shot capability of Large Language Models (LLMs) has enabled highly flexible, reference-free metrics for various tasks, making LLM evaluators common tools in NLP. However, the robustness of these LLM evaluators remains relatively understudied; existing work mainly pursued optimal performance in terms of correlating LLM scores with human expert scores. In this paper, we conduct a series of analyses using the SummEval dataset and confirm that LLMs are biased evaluators as they: (1) exhibit familiarity bias-a preference for text with lower perplexity, (2) show skewed and biased distributions of ratings, and (3) experience anchoring effects for multi-attribute judgments. We also found that LLMs are inconsistent evaluators, showing low "inter-sample" agreement and sensitivity to prompt differences that are insignificant to human understanding of text quality. Furthermore, we share recipes for configuring LLM evaluators to mitigate these limitations. Experimental results on the RoSE dataset demonstrate improvements over the state-of-the-art LLM evaluators.

19 April 2023
Verified
Conference Paper
Open access
RSRickard Stureborg
Rickard Stureborg
BDBhuwan Dhingra
Bhuwan Dhingra
JYJun Yang
Jun Yang
Human data labeling is an important and expensive task at the heart of supervised learning systems. Hierarchies help humans understand and organize concepts. We ask whether and how concept hierarchies can inform the design of annotation interfaces to improve labeling quality and efficiency. We study this question through annotation of vaccine misinformation, where the labeling task is difficult and highly subjective. We investigate 6 user interface designs for crowdsourcing hierarchical labels by collecting over 18,000 individual annotations. Under a fixed budget, integrating hierarchies into the design improves crowdsource workers’ F1 scores. We attribute this to (1) Grouping similar concepts, improving F1 scores by +0.16 over random groupings, (2) Strong relative performance on high-difficulty examples (relative F1 score difference of +0.40), and (3) Filtering out obvious negatives, increasing precision by +0.07. Ultimately, labeling schemes integrating the hierarchy outperform those that do not — achieving mean F1 of 0.70.

23 March 2024 in Vaccine
Verified
Journal Article
Closed access
RSRickard Stureborg
Rickard Stureborg
JNJenna Nichols
Jenna Nichols
BDBhuwan Dhingra
Bhuwan Dhingra
JYJun Yang
Jun Yang
Walter A. OrensteinWalter A. Orenstein
Walter A. Orenstein
LVLavanya Vasudevan
Lavanya Vasudevan

23 August 2024
Verified
Book Chapter
Closed access
PDF file
CZChloe Qinyu Zhu
Chloe Qinyu Zhu
RSRickard Stureborg
Rickard Stureborg
BDBhuwan Dhingra
Bhuwan Dhingra

1 January 2024
Verified
Conference Paper
Open access
PDF file
RSRickard Stureborg
Rickard Stureborg
SCSanxing Chen
Sanxing Chen
RXRoy Xie
Roy Xie
APAayushi Patel
Aayushi Patel
CLChristopher Li
Christopher Li
BDBhuwan Dhingra
Bhuwan Dhingra
Rickard Stureborg, Sanxing Chen, Roy Xie, Aayushi Patel, Christopher Li, Chloe Zhu, Tingnan Hu, Jun Yang, Bhuwan Dhingra. Findings of the Association for Computational Linguistics: NAACL 2024. 2024.

1 January 2024
Verified
Conference Paper
Open access
PDF file
RSRickard Stureborg
Rickard Stureborg
DADimitris Alikaniotis
Dimitris Alikaniotis
YSYoshi Suhara
Yoshi Suhara
Rickard Stureborg, Dimitris Alikaniotis, Yoshi Suhara. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers). 2024.