Regulatory approval of immune checkpoint inhibitors (ICIs) was based on results of large, randomized clinical trials, resulting in limited outcomes data in patient cohorts typically underrepresented in such trials. The objective of this study was to evaluate the efficacy and safety of ICIs in these unique patient cohorts. This is a multicenter, retrospective analysis of real-world data at six academic and community clinics in the United States from 1 January 2011 to 1 April 2018. Patients were included if they had received at least one cycle of ICI treatment. Unique patient cohorts included age > 75 years, non-White race, positive smoking history, ECOG performance status (PS) ≥ 2, BMI ≥ 30 kg/m², autoimmune diseases (AIDs), chronic viral infections (CVI), extensive prior lines of therapy (LOTs), or >three metastatic sites. Immune-related adverse events (irAEs), overall survival (OS), and time to treatment failure were evaluated in the entire cohort and in NSCLC patients treated with PD-(L)1 monotherapy. Outcomes and their association with unique patient cohorts were compared on univariate analysis and multivariate analysis to those without a particular characteristic in the entire NSCLC PD-(L)1 monotherapy cohorts. In total, 1453 patients were included: 56.5%—smokers, 30.4%—non-White, 22.8%—elderly, 20.8%—ECOG PS ≥ 2, 15.7%—history of AIDs, and 4.7%—history of CVI. The common ICIs were nivolumab (37.1%) and pembrolizumab (22.2%). Black patients, compared to White patients, experienced fewer irAEs (OR 0.54, p < 0.001). An ECOG PS of ≥2 (HR = 2.01, p < 0.001) and an increased number of previous LOTs were associated with poor OS (the median OS of 26.2 vs. 16.2 vs. 9.6 months for one vs. two vs. three prior LOTs, p < 0.001). The above results were confirmed in anti-PD-(L)1 monotherapy non-small cell lung cancer patients (n = 384). Overall, ICIs were safe and efficacious in these typically underrepresented patient cohorts. We noted ECOG PS ≥ 2 and an increased prior LOTs were associated with poor ICI efficacy, and Black patients, compared to White patients, experienced fewer irAEs.