Nicoleta Economou

Dr. Nicoleta Economou

orcid

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
29 November 2023 in Journal of the American Medical Informatics Association
Verified
Journal Article
Closed access
PDF file
NENicoleta Economou
Nicoleta Economou
SBSophia Bessias
Sophia Bessias
MCMichael P Cary
Michael P Cary
ABArmando Bedoya
Armando Bedoya
BGBenjamin A Goldstein
Benjamin A Goldstein
EPEric G Poon
Eric G Poon
Objective: The complexity and rapid pace of development of algorithmic technologies pose challenges for their regulation and oversight in healthcare settings. We sought to improve our institution’s approach to evaluation and governance of algorithmic technologies used in clinical care and operations by creating an Implementation Guide that standardizes evaluation criteria so that local oversight is performed in an objective fashion. Materials and Methods: Building on a framework that applies key ethical and quality principles (clinical value and safety, fairness and equity, usability and adoption, transparency and accountability, and regulatory compliance), we created concrete guidelines for evaluating algorithmic technologies at our institution. Results: An Implementation Guide articulates evaluation criteria used during review of algorithmic technologies and details what evidence supports the implementation of ethical and quality principles for trustworthy health AI. Application of the processes described in the Implementation Guide can lead to algorithms that are safer as well as more effective, fair, and equitable upon implementation, as illustrated through 4 examples of technologies at different phases of the algorithmic lifecycle that underwent evaluation at our academic medical center. Discussion: By providing clear descriptions/definitions of evaluation criteria and embedding them within standardized processes, we streamlined oversight processes and educated communities using and developing algorithmic technologies within our institution. Conclusions: We developed a scalable, adaptable framework for translating principles into evaluation criteria and specific requirements that support trustworthy implementation of algorithmic technologies in patient care and healthcare operations.

25 November 2023 in npj Digital Medicine
Verified
Journal Article
Open access
PDF file
SOShauna M. Overgaard
Shauna M. Overgaard
MGMegan G. Graham
Megan G. Graham
TBTracey Brereton
Tracey Brereton
MPMichael J. Pencina
Michael J. Pencina
JHJohn D. Halamka
John D. Halamka
NENicoleta Economou
Nicoleta Economou
The integration of Quality Management System (QMS) principles into the life cycle of development, deployment, and utilization of machine learning (ML) and artificial intelligence (AI) technologies within healthcare settings holds the potential to close the AI translation gap by establishing a robust framework that accelerates the safe, ethical, and effective delivery of AI/ML in day-to-day patient care. Healthcare organizations (HCOs) can implement these principles effectively by embracing an enterprise QMS analogous to those in regulated industries. By establishing a QMS explicitly tailored to health AI technologies, HCOs can comply with evolving regulations and minimize redundancy and rework while aligning their internal governance practices with their steadfast commitment to scientific rigor and medical excellence.

10 March 2025 in Journal of the American Medical Informatics Association
Verified
Journal Article
Closed access
Chuan HongChuan Hong
Chuan Hong
ACAnand Chowdhury
Anand Chowdhury
ASAnthony D Sorrentino
Anthony D Sorrentino
HWHaoyuan Wang
Haoyuan Wang
MAMonica Agrawal
Monica Agrawal
MPMichael J Pencina
Michael J Pencina
Objectives: Large language models (LLMs) are increasingly utilized in healthcare, transforming medical practice through advanced language processing capabilities. However, the evaluation of LLMs predominantly relies on human qualitative assessment, which is time-consuming, resource-intensive, and may be subject to variability and bias. There is a pressing need for quantitative metrics to enable scalable, objective, and efficient evaluation. Materials and Methods: We propose a unified evaluation framework that bridges qualitative and quantitative methods to assess LLM performance in healthcare settings. This framework maps evaluation aspects—such as linguistic quality, efficiency, content integrity, trustworthiness, and usefulness—to both qualitative assessments and quantitative metrics. We apply our approach to empirically evaluate the Epic In-Basket feature, which uses LLM to generate patient message replies. Results: The empirical evaluation demonstrates that while Artificial Intelligence (AI)-generated replies exhibit high fluency, clarity, and minimal toxicity, they face challenges with coherence and completeness. Clinicians’ manual decision to use AI-generated drafts correlates strongly with quantitative metrics, suggesting that quantitative metrics have the potential to reduce human effort in the evaluation process and make it more scalable. Discussion: Our study highlights the potential of a unified evaluation framework that integrates qualitative and quantitative methods, enabling scalable and systematic assessments of LLMs in healthcare. Automated metrics streamline evaluation and monitoring processes, but their effective use depends on alignment with human judgment, particularly for aspects requiring contextual interpretation. As LLM applications expand, refining evaluation strategies and fostering interdisciplinary collaboration will be critical to maintaining high standards of accuracy, ethics, and regulatory compliance. Conclusion: Our unified evaluation framework bridges the gap between qualitative human assessments and automated quantitative metrics, enhancing the reliability and scalability of LLM evaluations in healthcare. While automated quantitative evaluations are not ready to fully replace qualitative human evaluations, they can be used to enhance the process and, with relevant benchmarks derived from the unified framework proposed here, they can be applied to LLM monitoring and evaluation of updated versions of the original technology evaluated using qualitative human standards.

1 November 2023 in Circulation: Cardiovascular Quality and Outcomes
Verified
Journal Article
Closed access
PMPratheek Mallya
Pratheek Mallya
LSLaura M. Stevens
Laura M. Stevens
JZJuan Zhao
Juan Zhao
Chuan HongChuan Hong
Chuan Hong
RHRicardo Henao
Ricardo Henao
JHJennifer L. Hall
Jennifer L. Hall
BACKGROUND: High-quality research in cardiovascular prevention, as in other fields, requires inclusion of a broad range of data sets from different sources. Integrating and harmonizing different data sources are essential to increase generalizability, sample size, and representation of understudied populations—strengthening the evidence for the scientific questions being addressed. METHODS: Here, we describe an effort to build an open-access repository and interactive online portal for researchers to access the metadata and code harmonizing data from 4 well-known cohort studies—the REGARDS (Reasons for Geographic and Racial Differences in Stroke) study, FHS (Framingham Heart Study), MESA (Multi-Ethnic Study of Atherosclerosis), and ARIC (Atherosclerosis Risk in Communities) study. We introduce a methodology and a framework used for preprocessing and harmonizing variables from multiple studies. RESULTS: We provide a real-case study and step-by-step guidance to demonstrate the practical utility of our repository and interactive web page. In addition to our successful development of such an open-access repository and interactive web page, this exercise in harmonizing data from multiple cohort studies has revealed several key themes. These themes include the importance of careful preprocessing and harmonization of variables, the value of creating an open-access repository to facilitate collaboration and reproducibility, and the potential for using harmonized data to address important scientific questions and disparities in cardiovascular disease research. CONCLUSIONS: By integrating and harmonizing these large-scale cohort studies, such a repository may improve the statistical power and representation of understudied cohorts, enabling development and validation of risk prediction models, identification and investigation of risk factors, and creating a platform for racial disparities research. REGISTRATION: URL: https://precision.heart.org/duke-ninds .

24 February 2026 in Journal of the American Medical Informatics Association
Verified
Journal Article
Open access
PDF file
MCMichael P Cary
Michael P Cary
RRRegina G Russell
Regina G Russell
CSChristina Silcox
Christina Silcox
KLKay S Lytle
Kay S Lytle
LLLisa Soleymani Lehmann
Lisa Soleymani Lehmann
VGVincent Guilamo-Ramos
Vincent Guilamo-Ramos
Background and Approach: In 2025, the National Academy of Medicine released an Artificial Intelligence Code of Conduct (AICC). In this commentary, we examine how the AICC introduces governance mechanisms to oversee AI applications and how it can support the ethical development and responsible use of AI in healthcare, paying special attention to the role of nurses. Findings: One shortcoming of the AICC is its lack of explicit acknowledgment of nurses, which risks obscuring their indispensable role in the safe, equitable, and effective use of AI in healthcare. We offer practical steps for health leaders to operationalize the AICC. Conclusion: Implementation of the AICC can support robust AI governance in health systems, but nurse expertise must be incorporated. The AICC offers a framework into which nursing perspectives can be embedded to ensure that AI tools positively transform healthcare delivery and enhance the quality and equity of care.

24 February 2025 in The American Journal of Bioethics
Verified
Journal Article
Closed access
MEMatthew Elmore
Matthew Elmore
NENicoleta Economou
Nicoleta Economou
MPMichael Pencina
Michael Pencina