Laleh Laleh Seyyed-Kalantari

Dr. Laleh Laleh Seyyed-Kalantari


Research Focus & Metrics

Academic Career Length

10Years (2017-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
10 December 2021 in Nature Medicine
Verified
Journal Article
Open access
PDF file
LSLaleh Laleh Seyyed-Kalantari
Laleh Laleh Seyyed-Kalantari
HZHaoran Zhang
Haoran Zhang
MMMatthew B. A. McDermott
Matthew B. A. McDermott
ICIrene Y. Chen
Irene Y. Chen
MGMarzyeh Ghassemi
Marzyeh Ghassemi
Artificial intelligence (AI) systems have increasingly achieved expert-level performance in medical imaging applications. However, there is growing concern that such AI systems may reflect and amplify human bias, and reduce the quality of their performance in historically under-served populations such as female patients, Black patients, or patients of low socioeconomic status. Such biases are especially troubling in the context of underdiagnosis, whereby the AI algorithm would inaccurately label an individual with a disease as healthy, potentially delaying access to care. Here, we examine algorithmic underdiagnosis in chest X-ray pathology classification across three large chest X-ray datasets, as well as one multi-source dataset. We find that classifiers produced using state-of-the-art computer vision techniques consistently and selectively underdiagnosed under-served patient populations and that the underdiagnosis rate was higher for intersectional under-served subpopulations, for example, Hispanic female patients. Deployment of AI systems using medical imaging for disease diagnosis with such biases risks exacerbation of existing care biases and can potentially lead to unequal access to medical treatment, thereby raising ethical concerns for the use of these models in the clinic.

24 November 2020 in Biocomputing 2020
Verified
Conference Paper
Open access
PDF file
LSLaleh Laleh Seyyed-Kalantari
Laleh Laleh Seyyed-Kalantari
GLGuanxiong Liu
Guanxiong Liu
MMMatthew McDermott
Matthew McDermott
ICIrene Y. Chen
Irene Y. Chen
MGMarzyeh Ghassemi
Marzyeh Ghassemi
Machine learning systems have received much attention recently for their ability to achieve expert-level performance on clinical tasks, particularly in medical imaging. Here, we examine the extent to which state-of-the-art deep learning classifiers trained to yield diagnostic labels from X-ray images are biased with respect to protected attributes. We train convolution neural networks to predict 14 diagnostic labels in 3 prominent public chest X-ray datasets: MIMIC-CXR, Chest-Xray8, CheXpert, as well as a multi-site aggregation of all those datasets. We evaluate the TPR disparity − the difference in true positive rates (TPR) − among different protected attributes such as patient sex, age, race, and insurance type as a proxy for socioeconomic status. We demonstrate that TPR disparities exist in the state-of-the-art classifiers in all datasets, for all clinical tasks, and all subgroups. A multi-source dataset corresponds to the smallest disparities, suggesting one way to reduce bias. We find that TPR disparities are not significantly correlated with a subgroup’s proportional disease burden. As clinical models move from papers to products, we encourage clinical decision makers to carefully audit for algorithmic disparities prior to deployment. Our supplementary materials can be found at, http://www.marzyehghassemi.com/chexclusion-supp-3/.

8 April 2021
Verified
Conference Paper
Open access
HZHaoran Zhang
Haoran Zhang
NDNatalie Dullerud
Natalie Dullerud
LSLaleh Laleh Seyyed-Kalantari
Laleh Laleh Seyyed-Kalantari
QMQuaid Morris
Quaid Morris
SJShalmali Joshi
Shalmali Joshi
MGMarzyeh Ghassemi
Marzyeh Ghassemi
Clinical machine learning models experience significantly degraded performance in datasets not seen during training, e.g., new hospitals or populations. Recent developments in domain generalization offer a promising solution to this problem by creating models that learn invariances across environments. In this work, we benchmark the performance of eight domain generalization methods on multi-site clinical time series and medical imaging data. We introduce a framework to induce synthetic but realistic domain shifts and sampling bias to stress-test these methods over existing non-healthcare benchmarks. We find that current domain generalization methods do not achieve significant gains in out-of-distribution performance over empirical risk minimization on real-world medical imaging data, in line with prior work on general imaging datasets. However, a subset of realistic induced-shift scenarios in clinical time series data exhibit limited performance gains. We characterize these scenarios in detail, and recommend best practices for domain generalization in the clinical setting.

1 September 2023 in Radiology: Artificial Intelligence
Verified
Journal Article
Open access
MAMonish Ahluwalia
Monish Ahluwalia
MAMohamed Abdalla
Mohamed Abdalla
JSJames Sanayei
James Sanayei
LSLaleh Laleh Seyyed-Kalantari
Laleh Laleh Seyyed-Kalantari
MHMohannad Hussain
Mohannad Hussain
BFBenjamin Fine
Benjamin Fine
To externally test four chest radiograph classifiers on a large, diverse, real-world dataset with robust subgroup analysis. In this retrospective study, adult posteroanterior chest radiographs (January 2016–December 2020) and associated radiology reports from Trillium Health Partners in Ontario, Canada, were extracted and de-identified. An open-source natural language processing tool was locally validated and used to generate ground truth labels for the 197 540-image dataset based on the associated radiology report. Four classifiers generated predictions on each chest radiograph. Performance was evaluated using accuracy, positive predictive value, negative predictive value, sensitivity, specificity, F1 score, and Matthews correlation coefficient for the overall dataset and for patient, setting, and pathology subgroups. Classifiers demonstrated 68%–77% accuracy, 64%–75% sensitivity, and 82%–94% specificity on the external testing dataset. Algorithms showed decreased sensitivity for solitary findings (43%–65%), patients younger than 40 years (27%–39%), and patients in the emergency department (38%–60%) and decreased specificity on normal chest radiographs with support devices (59%–85%). Differences in sex and ancestry represented movements along an algorithm’s receiver operating characteristic curve. Performance of deep learning chest radiograph classifiers was subject to patient, setting, and pathology factors, demonstrating that subgroup analysis is necessary to inform implementation and monitor ongoing performance to ensure optimal quality, safety, and equity. Keywords: Conventional Radiography, Thorax, Ethics, Supervised Learning, Convolutional Neural Network (CNN), Machine Learning Algorithms Supplemental material is available for this article. © RSNA, 2023 See also the commentary by Huisman and Hannink in this issue.

5 June 2023 in IEEE Access
Verified
Journal Article
Open access
SAS. Akbarian
S. Akbarian
LSLaleh Laleh Seyyed-Kalantari
Laleh Laleh Seyyed-Kalantari
FKFarzad Khalvati
Farzad Khalvati
Elham DolatabadiElham Dolatabadi
Elham Dolatabadi
The performance of deep learning models, such as convolutional neural networks (CNN)s, is highly dependent on the size of the dataset used for training. In low-data environments, it can be challenging to achieve good performance when training models from scratch. This is where knowledge transfer approaches from pre-trained networks can be particularly useful. In this study, we implement different experiments for standard transfer learning approaches as our baseline and introduce the adoption of a novel knowledge transfer approach, called teacher-student learning framework, to improve the performance of diagnostic predictive models in medical imaging. Specifically, we investigate various configurations in the teacher-student learning framework inspired by the activation attention transfer in computer vision models to help address some of the challenges faced in medical imaging, such as the limited availability of annotated data and limited computing resources. We show that the teacher-student learning approach has the potential to significantly improve the performance of diagnostic predictive models. Our findings could have a positive impact on healthcare accessibility and affordability, as they may enable the development of more cost-effective and widely available medical imaging technologies under a limited data environment.

23 December 2024 in IEEE Access
Verified
Journal Article
Open access
FKFaiza Khan Khattak
Faiza Khan Khattak
VSVallijah Subasri
Vallijah Subasri
AKAmrit Krishnan
Amrit Krishnan
CPChloe Pou-Prom
Chloe Pou-Prom
SASedef Akinli-Kocak
Sedef Akinli-Kocak
FRFrank Rudzicz
Frank Rudzicz
Machine Learning Health Operations (MLHOps) is the combination of processes for reliable, efficient, usable, and ethical deployment and maintenance of machine learning models in healthcare settings. This paper provides both a survey of work in this area and guidelines for developers and clinicians to deploy and maintain their own models in clinical practice. We cover the foundational concepts of general machine learning operations and describe the initial setup of MLHOps pipelines (including data sources, preparation, engineering, and tools). We then describe long-term monitoring and updating (including data distribution shifts and model updating) and ethical considerations (including bias, fairness, interpretability, and privacy). This work therefore provides guidance across the full pipeline of MLHOps from conception to initial and ongoing deployment. We also include a checklist to ensure thorough verification of each step in the process.