Vasilisa Bashlovkina

Vasilisa Bashlovkina

Auto-generated profile, based on publicly available information

Vasilisa Bashlovkina

Google Research, New York, NY, USA

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
30 April 2023
Verified
Conference Paper
Open access
XDXiang Deng
Xiang Deng
VBVasilisa Bashlovkina
Vasilisa Bashlovkina
FHFeng Han
Feng Han
SBSimon Baumgartner
Simon Baumgartner
MBMichael Bendersky
Michael Bendersky
Market sentiment analysis on social media content requires knowledge of both financial markets and social media jargon, which makes it a challenging task for human raters. The resulting lack of high-quality labeled data stands in the way of conventional supervised learning methods. Instead, we approach this problem using semi-supervised learning with a large language model (LLM). Our pipeline generates weak financial sentiment labels for Reddit posts with an LLM and then uses that data to train a small model that can be served in production. We find that prompting the LLM to produce Chain-of-Thought summaries and forcing it through several reasoning paths helps generate more stable and accurate labels, while using a regression loss further improves distillation quality. With only a handful of prompts, the final model performs on par with existing supervised models. Though production applications of our model are limited by ethical considerations, the model’s competitive performance points to the great potential of using LLMs for tasks that otherwise require skill-intensive annotation.

30 April 2023
Verified
Conference Paper
Open access
XDXiang Deng
Xiang Deng
VBVasilisa Bashlovkina
Vasilisa Bashlovkina
FHFeng Han
Feng Han
SBSimon Baumgartner
Simon Baumgartner
MBMichael Bendersky
Michael Bendersky
Market sentiment analysis on social media content requires knowledge of both financial markets and social media jargon, which makes it a challenging task for human raters. The resulting lack of high-quality labeled data stands in the way of conventional supervised learning methods. In this work, we conduct a case study approaching this problem with semi-supervised learning using a large language model (LLM). We select Reddit as the target social media platform due to its broad coverage of topics and content types. Our pipeline first generates weak financial sentiment labels for Reddit posts with an LLM and then uses that data to train a small model that can be served in production. We find that prompting the LLM to produce Chain-of-Thought summaries and forcing it through several reasoning paths helps generate more stable and accurate labels, while training the student model using a regression loss further improves distillation quality. With only a handful of prompts, the final model performs on par with existing supervised models. Though production applications of our model are limited by ethical considerations, the model’s competitive performance points to the great potential of using LLMs for tasks that otherwise require skill-intensive annotation.

4 August 2023
Verified
Conference Paper
Open access
VBVasilisa Bashlovkina
Vasilisa Bashlovkina
RMRiley Matthews
Riley Matthews
ZKZhaobin Kuang
Zhaobin Kuang
SBSimon Baumgartner
Simon Baumgartner
MBMichael Bendersky
Michael Bendersky
We study the ability of transformer-based language models (LMs) to understand social media language. Social media (SM) language is distinct from standard written language, yet existing benchmarks fall short of capturing LM performance in this socially, economically, and politically important domain. We quantify the degree to which social media language differs from conventional language and conclude that the difference is significant both in terms of token distribution and rate of linguistic shift. Next, we introduce a new benchmark for Social MedIa Language Evaluation (SMILE) that covers four SM platforms and eleven tasks. Finally, we show that learning a tokenizer and pretraining on a mix of social media and conventional language yields an LM that outperforms the best similar-sized alternative by 4.2 points on the overall SMILE score.