Ilia Sucholutsky

Ilia Sucholutsky

Auto-generated profile, based on publicly available information

Ilia Sucholutsky

Department of Computer Science, New York University, New York, NY, USA

Research Focus & Metrics

Academic Career Length

10Years (2017-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
12 August 2024 in Proceedings of the National Academy of Sciences
Verified
Journal Article
Open access
SRSteve Rathje
Steve Rathje
DMDan-Mircea Mirea
Dan-Mircea Mirea
ISIlia Sucholutsky
Ilia Sucholutsky
RMRaja Marjieh
Raja Marjieh
CRClaire E. Robertson
Claire E. Robertson
JBJay J. Van Bavel
Jay J. Van Bavel
The social and behavioral sciences have been increasingly using automated text analysis to measure psychological constructs in text. We explore whether GPT, the large-language model (LLM) underlying the AI chatbot ChatGPT, can be used as a tool for automated psychological text analysis in several languages. Across 15 datasets ( n = 47,925 manually annotated tweets and news headlines), we tested whether different versions of GPT (3.5 Turbo, 4, and 4 Turbo) can accurately detect psychological constructs (sentiment, discrete emotions, offensiveness, and moral foundations) across 12 languages. We found that GPT ( r = 0.59 to 0.77) performed much better than English-language dictionary analysis ( r = 0.20 to 0.30) at detecting psychological constructs as judged by manual annotators. GPT performed nearly as well as, and sometimes better than, several top-performing fine-tuned machine learning models. Moreover, GPT’s performance improved across successive versions of the model, particularly for lesser-spoken languages, and became less expensive. Overall, GPT may be superior to many existing methods of automated text analysis, since it achieves relatively high accuracy across many languages, requires no training data, and is easy to use with simple prompts (e.g., “is this text negative?”) and little coding experience. We provide sample code and a video tutorial for analyzing text with the GPT application programming interface. We argue that GPT and other LLMs help democratize automated text analysis by making advanced natural language processing capabilities more accessible, and may help facilitate more cross-linguistic research with understudied languages.

27 November 2024 in Nature Human Behaviour
Verified
Journal Article
Open access
PDF file
XLXiaoliang Luo
Xiaoliang Luo
ARAkilles Rechardt
Akilles Rechardt
GSGuangzhi Sun
Guangzhi Sun
KNKevin K. Nejad
Kevin K. Nejad
FYFelipe Yáñez
Felipe Yáñez
BLBradley C. Love
Bradley C. Love
Scientific discoveries often hinge on synthesizing decades of research, a task that potentially outstrips human information processing capacities. Large language models (LLMs) offer a solution. LLMs trained on the vast scientific literature could potentially integrate noisy yet interrelated findings to forecast novel results better than human experts. Here, to evaluate this possibility, we created BrainBench, a forward-looking benchmark for predicting neuroscience results. We find that LLMs surpass experts in predicting experimental outcomes. BrainGPT, an LLM we tuned on the neuroscience literature, performed better yet. Like human experts, when LLMs indicated high confidence in their predictions, their responses were more likely to be correct, which presages a future where LLMs assist humans in making discoveries. Our approach is not neuroscience specific and is transferable to other knowledge-intensive endeavours.

20 February 2025 in Proceedings of the National Academy of Sciences
Verified
Journal Article
Open access
XBXuechunzi Bai
Xuechunzi Bai
AWAngelina Wang
Angelina Wang
ISIlia Sucholutsky
Ilia Sucholutsky
TGThomas L. Griffiths
Thomas L. Griffiths
Large language models (LLMs) can pass explicit social bias tests but still harbor implicit biases, similar to humans who endorse egalitarian beliefs yet exhibit subtle biases. Measuring such implicit biases can be a challenge: As LLMs become increasingly proprietary, it may not be possible to access their embeddings and apply existing bias measures; furthermore, implicit biases are primarily a concern if they affect the actual decisions that these systems make. We address both challenges by introducing two measures: LLM Word Association Test, a prompt-based method for revealing implicit bias; and LLM Relative Decision Test, a strategy to detect subtle discrimination in contextual decisions. Both measures are based on psychological research: LLM Word Association Test adapts the Implicit Association Test, widely used to study the automatic associations between concepts held in human minds; and LLM Relative Decision Test operationalizes psychological results indicating that relative evaluations between two candidates, not absolute evaluations assessing each independently, are more diagnostic of implicit biases. Using these measures, we found pervasive stereotype biases mirroring those in society in 8 value-aligned models across 4 social categories (race, gender, religion, health) in 21 stereotypes (such as race and criminality, race and weapons, gender and science, age and negativity). These prompt-based measures draw from psychology’s long history of research into measuring stereotypes based on purely observable behavior; they expose nuanced biases in proprietary value-aligned LLMs that appear unbiased according to standard benchmarks.

19 May 2023 in PsyArXiv
Verified
Preprint
Open access
SRSteve Rathje
Steve Rathje
DMDan-Mircea Mirea
Dan-Mircea Mirea
ISIlia Sucholutsky
Ilia Sucholutsky
RMRaja Marjieh
Raja Marjieh
CRClaire Robertson
Claire Robertson
JBJay Joseph Van Bavel
Jay Joseph Van Bavel
The social and behavioral sciences have been increasingly using automated text analysis to measure psychological constructs in text. We explore whether GPT, the large-language model underlying the artificial intelligence chatbot ChatGPT, can be used as a tool for automated psychological text analysis in several languages. Across 15 datasets (n = 47,925 manually annotated tweets and news headlines), we tested whether different versions of GPT (3.5 Turbo, 4, and 4 Turbo) can accurately detect psychological constructs (sentiment, discrete emotions, offensiveness, and moral foundations) across 12 languages. We found that GPT (r = 0.59-0.77) performs much better than English-language dictionary analysis (r = 0.20-0.30) at detecting psychological constructs as judged by manual annotators. GPT performs nearly as well as, and sometimes better than, several top-performing fine-tuned machine learning models. Moreover, GPT’s performance has improved across successive versions of the model, particularly for lesser-spoken languages. Overall, GPT may be superior to many existing methods of automated text analysis, since it achieves relatively high accuracy across many languages, requires no training data, and is easy to use with simple prompts (e.g., “is this text negative?”) and little coding experience. We provide sample code and a video tutorial for analyzing text with the GPT application programming interface. We argue that GPT and other large-language models may democratize automated text analysis by making advanced natural language processing capabilities more accessible, and may help facilitate more cross-linguistic research with understudied languages.

1 December 2017 in The Stata Journal: Promoting communications on statistics and Stata
Verified
Journal Article
Closed access
MSMatthias Schonlau
Matthias Schonlau
NGNick Guenther
Nick Guenther
ISIlia Sucholutsky
Ilia Sucholutsky
Text mining is the process of turning free text into numerical variables and then analyzing them with statistical techniques. We introduce the command ngram, which implements the most common approach to text mining, the “bag of words”. An n-gram is a contiguous sequence of words in a text. Broadly speaking, ngram creates hundreds or thousands of variables, each recording how often the corresponding n-gram occurs in a given text. This is more useful than it sounds. We illustrate ngram with the categorization of text answers from two open-ended questions.

18 May 2021 in Proceedings of the AAAI Conference on Artificial Intelligence
Verified
Journal Article
Open access
PDF file
ISIlia Sucholutsky
Ilia Sucholutsky
MSMatthias Schonlau
Matthias Schonlau
Deep neural networks require large training sets but suffer from high computational cost and long training times. Training on much smaller training sets while maintaining nearly the same accuracy would be very beneficial. In the few-shot learning setting, a model must learn a new class given only a small number of samples from that class. One-shot learning is an extreme form of few-shot learning where the model must learn a new class from a single example. We propose the 'less than one'-shot learning task where models must learn N new classes given only M