John J Hanna

Dr. John J Hanna

orcid

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
28 August 2023 in medRxiv
Verified
Preprint
Open access
PDF file
JHJohn J Hanna
John J Hanna
AWAbdi D. Wakene
Abdi D. Wakene
CLChristoph U. Lehmann
Christoph U. Lehmann
RMRichard J. Medford
Richard J. Medford
Large Language Models (LLM) are AI tools that can respond human-like to voice or free-text commands without training on specific tasks. However, concerns have been raised about their potential racial bias in healthcare tasks. In this study, ChatGPT was used to generate healthcare-related text for patients with HIV, analyzing data from 100 deidentified electronic health record encounters. Each patient’s data were fed four times with all information remaining the same except for race/ethnicity (African American, Asian, Hispanic White, Non-Hispanic White). The text output was analyzed for sentiment, subjectivity, reading ease, and most used words by race/ethnicity and insurance type. Results showed that instructions for African American, Asian, Hispanic White, and Non-Hispanic White patients had an average polarity of 0.14, 0.14, 0.15, and 0.14, respectively, with an average subjectivity of 0.46 for all races/ethnicities. The differences in polarity and subjectivity across races/ethnicities were not statistically significant. However, there was a statistically significant difference in word frequency across races/ethnicities and a statistically significant difference in subjectivity across insurance types with commercial insurance eliciting the most subjective responses and Medicare and other payer types the lowest. The study suggests that ChatGPT is relatively invariant to race/ethnicity and insurance type in terms of linguistic and readability measures. Further studies are needed to validate these results and assess their implications.

13 March 2025 in Journal of Medical Internet Research
Verified
Journal Article
Open access
JHJohn J Hanna
John J Hanna
AWAbdi D Wakene
Abdi D Wakene
AJAndrew O Johnson
Andrew O Johnson
Christoph Ulrich LehmannChristoph Ulrich Lehmann
Christoph Ulrich Lehmann
RMRichard Medford
Richard Medford
Background: Racial and ethnic bias in large language models (LLMs) used for health care tasks is a growing concern, as it may contribute to health disparities. In response, LLM operators implemented safeguards against prompts that are overtly seeking certain biases. Objective: This study aims to investigate a potential racial and ethnic bias among 4 popular LLMs: GPT-3.5-turbo (OpenAI), GPT-4 (OpenAI), Gemini-1.0-pro (Google), and Llama3-70b (Meta) in generating health care consumer–directed text in the absence of overtly biased queries. Methods: In this cross-sectional study, the 4 LLMs were prompted to generate discharge instructions for patients with HIV. Each patient’s encounter deidentified metadata including race/ethnicity as a variable was passed over in a table format through a prompt 4 times, altering only the race/ethnicity information (African American, Asian, Hispanic White, and non-Hispanic White) each time, while keeping all other information constant. The prompt requested the model to write discharge instructions for each encounter without explicitly mentioning race or ethnicity. The LLM-generated instructions were analyzed for sentiment, subjectivity, reading ease, and word frequency by race/ethnicity. Results: The only observed statistically significant difference between race/ethnicity groups was found in entity count (GPT-4, df=42, P=.047). However, post hoc chi-square analysis for GPT-4’s entity counts showed no significant pairwise differences among race/ethnicity categories after Bonferroni correction. Conclusions: A total of 4 LLMs were relatively invariant to race/ethnicity in terms of linguistic and readability measures. While our study used proxy linguistic and readability measures to investigate racial and ethnic bias among 4 LLM responses in a health care–related task, there is an urgent need to establish universally accepted standards for measuring bias in LLM-generated responses. Further studies are needed to validate these results and assess their implications.

24 July 2023 in Open Forum Infectious Diseases
Verified
Journal Article
Open access
PDF file
JHJohn J Hanna
John J Hanna
LGLiyu B Geresu
Liyu B Geresu
MDMarlon I Diaz
Marlon I Diaz
MHMilan Ho
Milan Ho
JCJulia A Casazza
Julia A Casazza
RMRichard J Medford
Richard J Medford
Background: Studies on COVID-19 in people with HIV (PWH) have had limitations. Further investigations on risk factors and outcomes of SARS-CoV-2 infection among PWH are needed. Methods: This retrospective cohort study leveraged the national OPTUM COVID-19 data set to investigate factors associated with SARS-CoV-2 positivity among PWH and risk factors for severe outcomes, including hospitalization, intensive care unit stays, and death. A subset analysis was conducted to examine HIV-specific variables. Multiple variable logistic regression was used to adjust for covariates. Results: Of 43 173 PWH included in this study, 6472 had a positive SARS-CoV-2 result based on a polymerase chain reaction test or antigen test. For PWH with SARS-CoV-2 positivity, higher odds were found for those who were younger (18–49 years), Hispanic White, African American, from the US South, uninsured, and a noncurrent smoker and had a higher body mass index and higher Charlson Comorbidity Index. For PWH with severe outcomes, higher odds were identified for those who were SARS-CoV-2 positive, older, from the US South, receiving Medicaid/Medicare or uninsured, a current smoker, and underweight and had a higher Charlson Comorbidity Index. In a subset analysis including PWH with HIV care variables (n = 5098), those with unsuppressed HIV viral load, a low CD4 count, and no antiretroviral therapy had higher odds of severe outcomes. Conclusions: This large US study found significant ethnic, racial, and geographic differences in SARS-CoV-2 infection among PWH. Chronic comorbidities, older age, lower body mass index, and smoking were associated with severe outcomes among PWH during the COVID-19 pandemic. SARS-CoV-2 infection was associated with severe outcomes, but once we adjusted for HIV care variables, SARS-CoV-2 was no longer significant; however, low CD4 count, high viral load, and lack of antiretroviral therapy had higher odds of severe outcomes.

28 May 2026 in Applied Clinical Informatics
Verified
Journal Article
Closed access
PDF file
JHJohn J Hanna
John J Hanna
CDChristopher R. Dennis
Christopher R. Dennis
AJAndrew O. Johnson
Andrew O. Johnson
LCLauren N. Cooper
Lauren N. Cooper
CEChristopher S. Evans
Christopher S. Evans
RMRichard J. Medford
Richard J. Medford
To support the artificial intelligence (AI) lifecycle in an integrated academic health system, we implemented a modular monitoring system to oversee electronic health record (EHR) vendor-provided clinical predictive models. This case study describes the lessons learned using this modular system to support sustainable oversight of the deployed predictive models across their lifecycle. We developed a modular monitoring system using automated data pipelines refreshed daily to support longitudinal oversight of four clinical predictive models developed by our EHR vendor (Epic Systems). An interactive monitoring application was designed to bridge the technical–operational gap for stakeholders involved in AI lifecycle decisions. Interactive reports include automated and assisted threshold-independent and threshold-dependent performance measures, flag rates, and problem-specific metrics crafted to support operational decision-making. Through the lenses of people, process, and technology, we describe lessons learned from the real-world implementation of a monitoring system. Effective oversight required clearly defined dyadic ownership (technical and operational) and embedded monitoring activities within existing AI governance decision-making processes across all AI lifecycle stages. Monitoring extended beyond performance drift to include alert burden, workflow alignment, and problem-level signals. Modularity enabled rapid adaptation as models and workflows evolved. Silent alert simulations proved valuable for selecting preimplementation thresholds that aligned with clinical capacity and workflows but were limited in predicting postimplementation performance when the models faced real-world human–alert interactions. Implementing a modular, governance-aligned monitoring system enabled sustained oversight of vendor-provided clinical predictive models and shifted monitoring beyond traditional technical performance metrics toward user-centric and operationally meaningful measures. These findings highlight the importance of integrated monitoring infrastructure as a core component of responsible AI governance in clinical settings.

26 June 2022 in Healthcare
Verified
Journal Article
Open access
PDF file
JHJohn J Hanna
John J Hanna
ANAnk E. Nijhawan
Ank E. Nijhawan
Christoph Ulrich LehmannChristoph Ulrich Lehmann
Christoph Ulrich Lehmann
RMRichard Medford
Richard Medford
Background: Undiagnosed human immunodeficiency virus (HIV) infection remains a public health challenge. We explore Facebook (FB) advertisement (Ads) cost per new HIV diagnosis using non-targeted Ads, a routine testing model against targeted Ads, and a focused testing model in Texas. Methods: On 14 October 2021, we created (without launching) Texas-based, USD 10 targeted (using criteria matching HIV populations at risk) and non-targeted FB Ads for 10 days. In the process of creating the Ads, we collected estimated audience size, daily reach, and daily clicks. We estimated Ad cost for each new HIV diagnosis for targeted and non-targeted Ads using new HIV diagnosis rates from focused and routine testing campaigns. Results: The Ad costs per new HIV diagnosis from the targeted model were 4.74, 2.86, 5.28, and 2.88 times lower for men, Black, Hispanic, and all age groups, respectively, when compared to the non-targeted model. The wider the gap was between new HIV diagnosis rates in a population for focused and routine testing, the more cost-effective targeted Ads became. Conclusions: Among HIV populations at risk, targeted FB Ads are more cost-effective for detecting new HIV infections than non-targeted Ads. This cost-effectiveness increases in locations where focused testing increases new HIV diagnosis rates, compared to routine testing.

9 January 2025 in Antimicrobial Stewardship & Healthcare Epidemiology
Verified
Journal Article
Open access
PDF file
LCLauren N. Cooper
Lauren N. Cooper
MDMarlon I. Diaz
Marlon I. Diaz
JHJohn J Hanna
John J Hanna
ZMZachary M. Most
Zachary M. Most
Christoph Ulrich LehmannChristoph Ulrich Lehmann
Christoph Ulrich Lehmann
RMRichard J. Medford
Richard J. Medford
Objective: Social media has become an important tool in monitoring infectious disease outbreaks such as coronavirus disease 2019 and highly pathogenic avian influenza (HPAI). Influenced by the recent announcement of a possible human death from H5N2 avian influenza, we analyzed tweets collected from X (formerly Twitter) to describe the messaging regarding the HPAI outbreak, including mis- and dis-information, concerns, and health education. Methods: We collected tweets involving keywords relating to HPAI for 5 days (June 04 to June 08, 2024). Using topic modeling, emotion, sentiment, and user demographic analyses, we were able to describe the population and the HPAI-related topics that users discussed. Results: With an original pool of 14,796 tweets, we analyzed a final data set of 13,319 tweets from 10,421 unique X users, with 50.4% of the tweets exhibiting negative sentiments (< 0 on a scale of −4 to +4). Predominant emotions were anger and fear shown in 36.4% and 29.5% of tweets, respectively. We identified 5 distinct, descriptive topics within the tweets. The use of emotionally charged language and spread of misinformation were substantial. Conclusions: Mis- and dis-information about the causes of and ways to prevent HPAI infections were common. A large portion of the tweets contained references to a planned epidemic or “plandemic” to influence the upcoming 2024 US presidential election. These tweets were countered by a limited number of tweets discussing infection locations, case reports, and preventive measures. Our study can be used by public health officials and clinicians to influence the discourse on current and future outbreaks.