Anaelia Ovalle

Dr. Anaelia Ovalle

orcid

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
12 June 2023
Verified
Conference Paper
Open access
AOAnaelia Ovalle
Anaelia Ovalle
PGPalash Goyal
Palash Goyal
JDJwala Dhamala
Jwala Dhamala
ZJZachary Jaggers
Zachary Jaggers
KCKai-Wei Chang
Kai-Wei Chang
RGRahul Gupta
Rahul Gupta
Warning: This paper contains examples of gender non-affirmative language which could be offensive, upsetting, and/or triggering. Transgender and non-binary (TGNB) individuals disproportionately experience discrimination and exclusion from daily life. Given the recent popularity and adoption of language generation technologies, the potential to further marginalize this population only grows. Although a multitude of NLP fairness literature focuses on illuminating and addressing gender biases, assessing gender harms for TGNB identities requires understanding how such identities uniquely interact with societal gender norms and how they differ from gender binary-centric perspectives. Such measurement frameworks inherently require centering TGNB voices to help guide the alignment between gender-inclusive NLP and whom they are intended to serve. Towards this goal, we ground our work in the TGNB community and existing interdisciplinary literature to assess how the social reality surrounding experienced marginalization of TGNB persons contributes to and persists within Open Language Generation (OLG). This social knowledge serves as a guide for evaluating popular large language models (LLMs) on two key aspects: (1) misgendering and (2) harmful responses to gender disclosure. To do this, we introduce TANGO, a dataset of template-based real-world text curated from a TGNB-oriented community. We discover a dominance of binary gender norms reflected by the models; LLMs least misgendered subjects in generated text when triggered by prompts whose subjects used binary pronouns. Meanwhile, misgendering was most prevalent when triggering generation with singular they and neopronouns. When prompted with gender disclosures, TGNB disclosure generated the most stigmatizing language and scored most toxic, on average. Our findings warrant further research on how TGNB harms manifest in LLMs and serve as a broader case study toward concretely grounding the design of gender-inclusive AI in community voices and interdisciplinary literature.

12 June 2023
Verified
Conference Paper
Open access
OQOrganizers Of Queerinai
Organizers Of Queerinai
AOAnaelia Ovalle
Anaelia Ovalle
ASArjun Subramonian
Arjun Subramonian
ASAshwin Singh
Ashwin Singh
CVClaas Voelcker
Claas Voelcker
LSLuke Stark
Luke Stark


26 April 2021 in Journal of Medical Internet Research
Verified
Journal Article
Open access
AOAnaelia Ovalle
Anaelia Ovalle
OGOrpaz Goldstein
Orpaz Goldstein
MKMohammad Kachuee
Mohammad Kachuee
EWElizabeth S C Wu
Elizabeth S C Wu
CHChenglin Hong
Chenglin Hong
MSMajid Sarrafzadeh
Majid Sarrafzadeh
Background: Social media networks provide an abundance of diverse information that can be leveraged for data-driven applications across various social and physical sciences. One opportunity to utilize such data exists in the public health domain, where data collection is often constrained by organizational funding and limited user adoption. Furthermore, the efficacy of health interventions is often based on self-reported data, which are not always reliable. Health-promotion strategies for communities facing multiple vulnerabilities, such as men who have sex with men, can benefit from an automated system that not only determines health behavior risk but also suggests appropriate intervention targets. Objective: This study aims to determine the value of leveraging social media messages to identify health risk behavior for men who have sex with men. Methods: The Gay Social Networking Analysis Program was created as a preliminary framework for intelligent web-based health-promotion intervention. The program consisted of a data collection system that automatically gathered social media data, health questionnaires, and clinical results for sexually transmitted diseases and drug tests across 51 participants over 3 months. Machine learning techniques were utilized to assess the relationship between social media messages and participants' offline sexual health and substance use biological outcomes. The F1 score, a weighted average of precision and recall, was used to evaluate each algorithm. Natural language processing techniques were employed to create health behavior risk scores from participant messages. Results: Offline HIV, amphetamine, and methamphetamine use were correctly identified using only social media data, with machine learning models obtaining F1 scores of 82.6%, 85.9%, and 85.3%, respectively. Additionally, constructed risk scores were found to be reasonably comparable to risk scores adapted from the Center for Disease Control. Conclusions: To our knowledge, our study is the first empirical evaluation of a social media–based public health intervention framework for men who have sex with men. We found that social media data were correlated with offline sexual health and substance use, verified through biological testing. The proof of concept and initial results validate that public health interventions can indeed use social media–based systems to successfully determine offline health risk behaviors. The findings demonstrate the promise of deploying a social media–based just-in-time adaptive intervention to target substance use and HIV risk behavior.

8 August 2023
Verified
Conference Paper
Open access
NDNathan Dennler
Nathan Dennler
AOAnaelia Ovalle
Anaelia Ovalle
ASAshwin Singh
Ashwin Singh
LSLuca Soldaini
Luca Soldaini
ASArjun Subramonian
Arjun Subramonian
JPJess De Jesus De Pinho Pinhal
Jess De Jesus De Pinho Pinhal
Bias evaluation benchmarks and dataset and model documentation have emerged as central processes for assessing the biases and harms of artificial intelligence (AI) systems. However, these auditing processes have been criticized for their failure to integrate the knowledge of marginalized communities and consider the power dynamics between auditors and the communities. Consequently, modes of bias evaluation have been proposed that engage impacted communities in identifying and assessing the harms of AI systems (e.g., bias bounties). Even so, asking what marginalized communities want from such auditing processes has been neglected. In this paper, we ask queer communities for their positions on, and desires from, auditing processes. To this end, we organized a participatory workshop to critique and redesign bias bounties from queer perspectives. We found that when given space, the scope of feedback from workshop participants goes far beyond what bias bounties afford, with participants questioning the ownership, incentives, and efficacy of bounties. We conclude by advocating for community ownership of bounties and complementing bounties with participatory processes (e.g., co-creation).

1 January 2024
Verified
Conference Paper
Open access
PDF file
AOAnaelia Ovalle
Anaelia Ovalle
NMNinareh Mehrabi
Ninareh Mehrabi
PGPalash Goyal
Palash Goyal
JDJwala Dhamala
Jwala Dhamala
KCKai-Wei Chang
Kai-Wei Chang
RGRahul Gupta
Rahul Gupta
Anaelia Ovalle, Ninareh Mehrabi, Palash Goyal, Jwala Dhamala, Kai-Wei Chang, Richard Zemel, Aram Galstyan, Yuval Pinter, Rahul Gupta. Findings of the Association for Computational Linguistics: NAACL 2024. 2024.