Objectives Our main objective is to assess the inter-reviewer reliability (IRR) reported in published systematic literature reviews (SLRs). Our secondary objective is to determine the expected IRR by authors of SLRs for both human and machine-assisted reviews. Methods We performed a review of SLRs of randomised controlled trials using the PubMed and Embase databases. Data were extracted on IRR by means of Cohen’s kappa score of abstract/title screening, full-text screening and data extraction in combination with review team size, items screened and the quality of the review was assessed with the A MeaSurement Tool to Assess systematic Reviews 2. In addition, we performed a survey of authors of SLRs on their expectations of machine learning automation and human performed IRR in SLRs. Results After removal of duplicates, 836 articles were screened for abstract, and 413 were screened full text. In total, 45 eligible articles were included. The average Cohen’s kappa score reported was 0.82 (SD=0.11, n=12) for abstract screening, 0.77 (SD=0.18, n=14) for full-text screening, 0.86 (SD=0.07, n=15) for the whole screening process and 0.88 (SD=0.08, n=16) for data extraction. No association was observed between the IRR reported and review team size, items screened and quality of the SLR. The survey (n=37) showed overlapping expected Cohen’s kappa values ranging between approximately 0.6–0.9 for either human or machine learning-assisted SLRs. No trend was observed between reviewer experience and expected IRR. Authors expect a higher-than-average IRR for machine learning-assisted SLR compared with human based SLR in both screening and data extraction. Conclusion Currently, it is not common to report on IRR in the scientific literature for either human and machine learning-assisted SLRs. This mixed-methods review gives first guidance on the human IRR benchmark, which could be used as a minimal threshold for IRR in machine learning-assisted SLRs. PROSPERO registration number CRD42023386706.
As of March 2021, COVID-19 has claimed the lives of more than 2.7 million people worldwide. Vaccination has started in most countries around the world. In this study, we estimated the cost-effectiveness of strategies for COVID-19 vaccination for Turkey compared to a baseline in the absence of vaccination and imposed measures by using an enhanced SIRD (Susceptible, Infectious, Recovered, Death) model and various scenarios for the first year after vaccination. The results showed that vaccination is cost-effective from a health care perspective, with an incremental cost-effectiveness ratio (ICER) of 511 USD/QALY and 1045 USD/QALY if vaccine effectiveness on transmission is equal or reduced to only 50% of effectiveness on disease, respectively, at the 90% baseline effectiveness of the vaccine. From a societal perspective, cost savings were estimated for both scenarios. Other results further showed that the minimum required vaccine uptake to be cost-effective would be at least 30%. Sensitivity and scenario analyses, as well as the iso-ICER curves, showed that the results were quite robust and that major changes in cost-effectiveness outcomes cannot be expected. We can conclude that COVID-19 vaccination in Turkey is highly cost-effective or even cost-saving.
A diabetes risk score cannot directly be translated and applied in different populations, and its performance should be evaluated in the target population. This study aimed to translate the Finnish Diabetes Risk Score (FINDRISC) instrument and compare its performance with the modified version for detecting undiagnosed type 2 diabetes mellitus (T2DM) and dysglycaemia among the Indonesian adult population. Forward and backward translations were performed and followed by cultural adaptation. In total, 1,403 participants were recruited. The FINDRISC-Bahasa Indonesia (FINDRISC-BI) was scored according to the original FINDRISC instrument, while a Modified FINDRISC-BI was analyzed using a specific body mass index and waist circumference classification for Indonesians. The area under the receiver operating characteristic curve, sensitivity, specificity, and the optimal cut-offs of both instruments were estimated. The area under the receiver operating characteristic curve for detecting undiagnosed T2DM was 0.73 (0.67–0.78) for the FINDRISC-BI with an optimal cut-off score of ≥9 (sensitivity = 63.0%; specificity = 67.3%) and 0.72 (0.67–0.78) for the Modified FINDRISC-BI with an optimal cut-off score of ≥11 (sensitivity = 59.8%; specificity = 74.9%). The area under the receiver operating characteristic curve for detecting dysglycaemia was 0.72 (0.69–0.75) for the FINDRISC-BI instrument with an optimal cut-off score of ≥8 (sensitivity = 66.4%; specificity = 67.0%), and 0.72 (0.69–0.75) for the Modified FINDRISC-BI instrument with an optimal cut-off score ≥9 (sensitivity = 63.8%; specificity = 67.6%). The Indonesian version of the FINDRISC instrument has acceptable diagnostic accuracy for screening people with undiagnosed T2DM or dysglycaemia in Indonesia. Modifying the body mass index and waist circumference classifications in the Modified FINDRISC-BI results in a similar diagnostic accuracy; however, the Modified FINDRISC-BI has a higher optimal cut-off point than the FINDRISC-BI. People with an above optimal cut-off score are suggested to take a further blood glucose test.
1 August 2016 in Annals of Allergy, Asthma & Immunology
Background: Evidence on costs and health benefits of pneumococcal conjugate vaccine (PCV) for children in Asian countries is limited but growing. As a region with a considerably high burden of pneumococcal disease, it is prominent to have a comprehensive overview on the cost-effectiveness of implementing and adopting a PCV vaccination program. Methods: We conducted a systematic review from Pubmed and Embase to identify economic evaluation studies of PCV for children in Asian countries up to May 2020. Data extraction included specific characteristics of the study, input parameters, cost elements, cost-effectiveness results, and key drivers of uncertainty. The Preferred Reporting Items for Systematic Reviews and Meta Analyses (PRISMA) statement was followed for this systematic review. The reporting quality of the included studies was evaluated using the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. Results: After the screening process on both the title and abstract and full text of 518 records, a total of 25 studies fulfilled the inclusion criteria, and were included in the review. The majority of included studies demonstrates that PCV for children is cost-effective in most of the Asian region, and even cost-saving in some countries. Most of the included studies implemented cost utility analysis (CUA) using either quality-adjusted life years (QALYs) or disability-adjusted life years (DALYs). Overall, the main drivers affecting the cost effectiveness were vaccine price, burden regarding pneumonia-related parameters, and the inclusion of herd effects. Conclusion: The children pneumococcal vaccination program appears to be a cost-effective intervention in Asia, and even cost-saving in certain conditions. Vaccine price, pneumonia-related disease burden, and the inclusion of the herd effect are observed as important key drivers in estimating cost-effectiveness in this region. Incorporating PCV in vaccination programs in this region was found to be highly favorable.