Background Efforts to mitigate unwarranted variation in the quality of care require insight into the ‘level’ (eg, patient, physician, ward, hospital) at which observed variation exists. This systematic literature review aims to synthesise the results of studies that quantify the extent to which hospitals contribute to variation in quality indicator scores. Methods Embase, Medline, Web of Science, Cochrane and Google Scholar were systematically searched from 2010 to November 2023. We included studies that reported a measure of between-hospital variation in quality indicator scores relative to total variation, typically expressed as a variance partition coefficient (VPC). The results were analysed by disease category and quality indicator type. Results In total, 8373 studies were reviewed, of which 44 met the inclusion criteria. Casemix adjusted variation was studied for multiple disease categories using 144 indicators, divided over 5 types: intermediate clinical outcomes (n=81), final clinical outcomes (n=35), processes (n=10), patient-reported experiences (n=15) and patient-reported outcomes (n=3). In addition to an analysis of between-hospital variation, eight studies also reported physician-level variation (n=54 estimates). In general, variation that could be attributed to hospitals was limited (median VPC=3%, IQR=1%–9%). Between-hospital variation was highest for process indicators (17.4%, 10.8%–33.5%) and lowest for final clinical outcomes (1.4%, 0.6%–4.2%) and patient-reported outcomes (1.0%, 0.9%–1.5%). No clear pattern could be identified in the degree of between-hospital variation by disease category. Furthermore, the studies exhibited limited attention to the reliability of observed differences in indicator scores. Conclusion Hospital-level variation in quality indicator scores is generally small relative to residual variation. However, meaningful variation between hospitals does exist for multiple indicators, especially for care processes which can be directly influenced by hospital policy. Quality improvement strategies are likely to generate more impact if preceded by level-specific and indicator-specific analyses of variation, and when absolute variation is also considered. PROSPERO registration number CRD42022315850.

Background: Quality indicators (QIs) are widely used to benchmark hospital performance and improve quality of care but are often based on expert opinion rather than data-driven assessment. This study aims to evaluate QIs, using a framework that assesses four criteria: Feasibility, discriminative ability, validity, and reliability. Methods: We used data from the Dutch breast cancer registry (NABON Breast Cancer Audit, NBCA) and included all surgically treated breast cancer patients in the Netherlands between 2021-2023. Eighteen QIs were evaluated. Feasibility was determined by QI numerator completeness, with >90% data availability considered feasible. Discriminative ability was assessed by between-hospital variation in QI scores, where an interquartile range (IQR) >10% indicated good discrimination. Validity was evaluated by the impact of case-mix adjustment and considered low when (pseudo-) R² 75% was considered high. Results: After exclusion of one QI with feasibility <1%, feasibility ranged from 80.2%-100%, and 15 QIs were feasible. Overall, the IQR ranged from 1 to 36, with 8 QIs having an IQR higher than 10, indicating good discriminative ability. The (pseudo-)R² ranged from 0.01-0.53, with 11 QIs showing low case-mix impact. Rankability ranged from 0-69%, with none of the QIs having a high rankability. None of the QIs met all preset criteria, but six QIs met at least three out of four criteria. Conclusion: The QIs employed by the Dutch breast cancer registry fulfilled most criteria, but rankability is a concern and requires specific attention, especially for public reporting. Our results show the importance of considering feasibility, discriminative ability, validity, and reliability when evaluating QIs, and these should also be taken into account when developing new QIs.

Survival rates of adolescents and young adults (AYAs) with acute lymphoblastic leukemia (ALL) are inferior to those of pediatric ALL patients. In part, this may be caused by differences in treatment setting. Generally, children are treated in specialized pediatric hemato-oncology settings, whereas AYAs are treated in adult hemato-oncology settings. Since 2005, adult treatment protocols have included pediatric-inspired chemotherapy, which has been the standard of care for AYAs from 2008 onwards. This study aims to assess whether, despite protocols in both settings having become more similar, there remains an effect of treatment in specialized pediatric hemato-oncology settings on 5-year survival for ALL patients in the Netherlands. We used nationwide registry data (2004–2013) on 472 ALL patients aged between 10 and 30 years old. A fuzzy regression discontinuity design was applied to estimate the treatment effect using two-stage least squares regression with the treatment threshold at 17 years and 7 months of age, adjusting for sex, age at diagnosis, and immunophenotype. We found a risk difference of 0.419 (p = 0.092; 95% CI = −0.0686; 0.907), meaning a 41.9 percentage point greater probability of surviving five years after diagnosis for ALL patients treated in specialized pediatric hemato-oncology settings. Our results suggest that ALL patients around the threshold could benefit from increased collaboration between pediatric and adult hemato-oncology in terms of survival.

Clinical and pathological outcomes of oesophagogastric cancer surgery are used for benchmarking hospital performance. The extent to which case-mix adjustment is required for valid hospital comparisons is unknown. This study aimed to develop distinct case-mix adjustment models for multiple outcomes of oesophageal and gastric cancer surgery, and to assess the impact of case-mix adjustment on between-hospital comparisons. We included all patients who underwent oesophagogastric cancer resections in the Netherlands between 2017 and 2022. We developed distinct case-mix adjustment models for ten outcomes. Model performance was evaluated with the area-under-the-receiving-operator-curve (AUC) and pseudo-R-squared, representing how strongly case-mix factors predict the outcomes. We used the Wald χ² test to assess relative predictor importance per model. The impact of case-mix adjustment on between-hospital comparisons on outcome was quantified using unadjusted and adjusted observed/expected ratios. In total, 4354 oesophageal cancer patients and 2109 gastric cancer patients were included. The most informative predictors in the models for oesophageal cancer were ASA-score, salvage surgery, peripheral vascular disease/aortic aneurysm, chronic lung disease, and tumour histology. For gastric cancer these were age, preoperative weight loss, tumour location, and clinical M-category. All case-mix models showed low to moderate performance, with AUCs ranging between 0.58 and 0.73 and between 0.58 and 0.74 for oesophageal and gastric cancer, respectively. Overall, case-mix adjustment had a limited impact on between-hospital comparisons, but more pronounced for 30-day mortality, failure-to-cure and failure-to-rescue. Given low to moderate model performance and the limited impact on between-hospital comparisons, case-mix adjustment may not always be necessary for valid benchmarking on outcomes in oesophagogastric cancer surgery.

Objective To construct a data-driven composite from (a subset of) currently used quality indicators for oesophagogastric cancer surgery and to evaluate whether this approach enhances the reliability of between-hospital comparisons on outcome relative to the expert-driven composite indicator ‘textbook outcome (TO)’. Design In this retrospective cohort study, we applied Item Response Theory (IRT) to construct a data-driven continuous composite indicator reflecting a single latent variable—the quality of surgical care—and estimated latent variable scores for all individual patients. Reliability was compared between the expert-driven (TO) and data-driven (IRT) composite indicators. Setting All Dutch hospitals providing oesophagogastric cancer surgery. Participants All patients who underwent oesophagectomy (n=3588) or gastrectomy (n=1782) between 2018 and 2022 as registered in the Dutch Upper GI Cancer Audit (DUCA). Primary and secondary outcome measures We evaluated the reliability of between-hospital comparisons using ‘rankability’, which quantifies the proportion of observed variation in indicator scores between hospitals not attributable to chance. Results Seven out of 15 quality indicators were included in the IRT composite indicator. Most of the patients were assigned the artificial maximum of the continuous quality score (ie, ceiling effect), resulting in similar average hospital scores. Relative to TO, rankability increased when using the IRT composite for oesophagectomy (57% vs 41%) but declined for gastrectomy (38% vs 47%). Conclusions The selected seven quality indicators for oesophageal and gastric cancer surgery represent a single latent variable but are not yet optimal for differentiating surgical care quality due to ceiling effects. Despite using fewer indicators, the continuous IRT score showed a promising increase in rankability for oesophagectomy, suggesting that data-driven composite indicators may enhance hospital benchmarking reliability.