Melanie S. Brucks

Melanie S. Brucks

Auto-generated profile, based on publicly available information

Melanie S. Brucks

Columbia University, New York, NY 10027, USA

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
27 April 2022 in Nature
Verified
Journal Article
Closed access
PDF file
MBMelanie S. Brucks
Melanie S. Brucks
JLJonathan Levav
Jonathan Levav

28 April 2025 in PLOS ONE
Verified
Journal Article
Open access
PDF file
MBMelanie Brucks
Melanie Brucks
OTOlivier Toubia
Olivier Toubia
We examine how the seemingly arbitrary way a prompt is posed, which we term “prompt architecture,” influences responses provided by large language models (LLMs). Five large-scale, full-factorial experiments performing standard (zero-shot) similarity evaluation tasks using GPT-3, GPT-4, and Llama 3.1 document how several features of prompt architecture (order, label, framing, and justification) interact to produce methodological artifacts, a form of statistical bias. We find robust evidence that these four elements unduly affect responses across all models, and although we observe differences between GPT-3 and GPT-4, the changes are not necessarily for the better. Specifically, LLMs demonstrate both response-order bias and label bias, and framing and justification moderate these biases. We then test different strategies intended to reduce methodological artifacts. Specifying to the LLM that the order and labels of items have been randomized does not alleviate either response-order or label bias, and the use of uncommon labels reduces (but does not eliminate) label bias but exacerbates response-order bias in GPT-4 (and does not reduce either bias in Llama 3.1). By contrast, aggregating across prompts generated using a full factorial design eliminates response-order and label bias. Overall, these findings highlight the inherent fallibility of any individual prompt when using LLMs, as any prompt contains characteristics that may subtly interact with a multitude of hidden associations embedded in rich language data.

1 July 2020 in Journal of the Association for Consumer Research
Verified
Journal Article
Closed access
MBMelanie S. Brucks
Melanie S. Brucks
SHSzu-Chi Huang
Szu-Chi Huang

4 September 2026 in Science Advances
Verified
Journal Article
Open access
TPTianyi Peng
Tianyi Peng
MBMelanie Brucks
Melanie Brucks
GGGeorge Gui
George Gui
DMDaniel J. Merlau
Daniel J. Merlau
GFGrace Jiarui Fan
Grace Jiarui Fan
OTOlivier Toubia
Olivier Toubia
Scientists and practitioners are aggressively moving to deploy digital twins—large language model (LLM)-based models of individuals—across social science and policy research. We conducted 19 preregistered studies with 164 diverse outcomes (e.g., attitudes toward hiring algorithms and intention to share misinformation) and compared human responses with those of their digital twins (trained on each person’s previous answers to more than 500 questions). We establish an empirical benchmark for digital twin performance: Digital twins’ answers are only modestly more accurate than those from the (homogeneous) base LLM and correlate weakly with human responses (average correlation coefficient of 0.20). To guide future development, we document five ways in which digital twins distort human behavior: (i) insufficient individuation, (ii) stereotyping, (iii) representation bias, (iv) ideological biases, and (v) hyper-rationality. We make our full dataset and code public as a standardized testbed. Our results caution against premature deployment while laying the groundwork for the transparent, replicable, and iterative science necessary for responsible deployment of digital twins.

7 June 2022 in Nature
Verified
Journal Article
Closed access
PDF file
MBMelanie S. Brucks
Melanie S. Brucks
JLJonathan Levav
Jonathan Levav

3 December 2025 in Nature
Verified
Journal Article
Closed access
PDF file
MBMelanie S. Brucks
Melanie S. Brucks
JRJacqueline R. Rifkin
Jacqueline R. Rifkin
JJJeff S. Johnson
Jeff S. Johnson