Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Despite the availability of free routine immunizations in low and middle-income countries (LMICs), many children are incompletely vaccinated, vaccinated late for age or drop out over the course of the immunization schedule. Without the technology to model and visualize risk on large datasets, vaccinators and policy-makers are unable to identify and target groups and individuals at high risk of dropping out thus default rates remain high preventing universal immunization coverage. Predictive analytics algorithm leverages Artificial Intelligence (AI), and uses statistical modeling, machine learning and multi-dimensional data mining to accurately identify children who are most likely to delay or miss their follow-up immunization visits. Objective: To conduct feasibility testing and validation of a predictive analytics algorithm to identify which children are likely to miss the follow-up vaccines through risk profiling of clients. Methods: The algorithm was developed using 47,554 longitudinal immunization records, split into a training and validation cohort. Four machine learning models (random forest, recursive partitioning, support vector machines and C-forest) were used to generate the algorithm to predict, for each child, the likelihood of defaulting on the follow-up immunization visit. The models were encapsulated in the predictive engine, which decided the most appropriate method to apply in a given case. Each of the models was assessed on: accuracy, precision (positive predictive value), sensitivity, specificity and negative predictive value, and area under curve. Results: Out of 11,889 cases in validation dataset, the random forest model correctly predicted 8,994 cases yielding 94.9% sensitivity and 54.9% specificity. The C-Forest Model, Support Vector Machines and Recursive Partitioning Models improved prediction respectively by achieving 352, 376 and 389 visits above the predictions made by the random forest model. All models had a C-statistic of 0.750 or above while the highest statistic (AUC 0.791, 95% CI: 0.784– 0.798) was seen for the Recursive Partitioning algorithm. Conclusions: This feasibility study demonstrates predictive analytics can accurately identify children who are at higher risk for defaulting on follow-up immunization visits. Correct identification of potential defaulters opens a window for evidence-based targeted interventions in resource limited settings to achieve optimal immunization coverage and timeliness.

Background: Inability to track children’s vaccination history coupled with parents’ lack of awareness of vaccination due dates compounds the problem of low immunization coverage and timeliness in developing countries. We evaluated the impact of two types of silicone immunization reminder bracelets for children in improving immunization coverage and timeliness of Pentavalent-3 and the Measles-1 vaccines. Methods: Children < 3 months were enrolled in either of the 2 intervention groups (Alma Sana Bracelet Group and Star Bracelet Group) or the Control group. Children in the intervention groups were provided the two different bracelets at the time of recruitment. Each time the child visited the immunization center, a hole was perforated in the silicone bracelet to denote vaccine administration. Each child was followed up till administration of Measles-1 vaccine or till 12 months of age (if they did not come to the center for vaccination). Data was analyzed using the intention-to-treat population between groups. The unadjusted and adjusted Risk Ratios (RR) and 95% confidence interval (CI) for Pentavalent-3 and Measles-1 coverage at 12 months of age were estimated through bivariate and multivariate analysis. Time-to-Pentavalent-3 and Measles-1 immunization curves were calculated using the Kaplan–Meier method. Results: A total of 1,445 children were enrolled in the study between July 19, 2017 and October 10, 2017. Baseline characteristics among the three groups were similar. Up-to-date coverage for the Pentavalent-3 /Measles-1 vaccine at 12 months of age was 84.6%/72.0%, 85.4%/70.5% and 83.0%/68.5% in Alma Sana Bracelet group, Star Bracelet group and Control group respectively but the differences were not statistically significant. In the multivariate analysis, neither the Alma Sana bracelet (adjusted RR = 1.01; 95% CI: 0.96-1.06), (adjusted RR: 1.05; 95% CI: 0.97-1.13) nor the Star bracelet (adjusted RR = 1.01; 95% CI: 0.96-1.06) (adjusted RR: 1.03; 95% CI: 0.95-1.11) was significantly associated with Pentavalent-3 vaccination or Measles-1 vaccination. Conclusion: Although we did not observe any significant impact of the bracelets on improved immunization coverage and timeliness, our findings add to the existing literature on innovative, low cost reminders for health and make several suggestions for enhancing practical implementation of these tools. Trial registration: ClinicalTrials.gov NCT03310762. Retrospectively Registered on October 16, 2017.

Background: Routine childhood immunization coverage in Pakistan remains sub-par, in part, due to suboptimal utilization of existing vaccination services. Quality of vaccine delivery can affect both supply and demand for immunization, but data for immunization center quality in Pakistan is sparse and in Sindh province in Southern Pakistan, no comprehensive health facility assessment has ever been conducted at a provincial level. We assessed health facilities, specifically immunization centers, and their associated health workers throughout the province to summarize quality of immunization centers. Methods: An exhaustive list of health facilities obtained from Sindh’s provincial government was included in our analysis, comprising a total of 1396 public, private, and public-private health facilities. We adapted a health facility and health worker assessment survey developed by BASICS and EPI-Sindh to record indicators pertaining to health facility infrastructure, processes and human resources. Using expert panel ranking, we developed critical criteria (the presence of a cold box/refrigerator, vaccinator and vaccination equipment at the immunization center) to indicate the bare minimum items required by immunization centers to vaccinate children. We also categorized other infrastructure, process, and human resource items to determine high, low and moderate function requirements to ascertain quality. We evaluated presence of critical criteria, calculated scores for high, moderate and low function requirements, and displayed frequencies of infrastructure, process and human resource indicators for all immunization centers across Sindh. We analyzed results at the division level and utilized a two-sample independent clustered t-test to test differences in average function requirement scores between facilities that met critical criteria and those that did not. Results: Out of the 1396 health facilities assessed across Sindh province from October 2017 to January 2018, 1236 (88.5%) were operational while 1209 (86.6%) offered vaccination services (immunization centers). Only 793 (65.6%; 793/1209) immunization centers met the critical criteria of having all the following items: vaccinator, a cold box or refrigerator and vaccine supplies. Of the 416 (34.4%; 416/1209) immunization centers that did not meet the critical criteria, most of the centers did not have a cold box or refrigerator (28.3%; 342/1209), followed by lack of vaccines (19.9%; 240/1209), and a vaccinator (13.0%; 157/1209). Of the 2153 healthcare workers interviewed, 1875 (87.1%) were vaccinators, of which 1745 (81.0%; 1745/2153) were male, and had an average of 12.4 years of schooling. A total of 1805 (96.3%; 1805/1875), 1655 (88.3%; 1655/1875) and 1387 (74.0%; 1387/1875) of the vaccinators were trained in vaccination, cold chain and inventory management respectively. Conclusion: One out of three immunization centers in Sindh lack the critical components essential for quality vaccination services. While the majority of health workers (>80%) were trained on vaccination and cold chain management, the proportion trained on inventory management was comparatively low. Our findings therefore suggest that suboptimal immunization center quality is partly due to inadequate infrastructure and inefficient processes contributed to an extent, by low levels of inventory management training among vaccinators. Our study presents critical research findings with high-impact policy implications for identifying and addressing gaps to improve vaccination uptake within a low-middle income country setting.

Gaps in information access impede immunization uptake, especially in low-resource settings where cutting-edge and innovative digital interventions are limited given the digital inequity. Our objective was to develop an Artificially Intelligent (AI) chatbot to respond to caregiver's immunization-related queries in Pakistan and investigate its feasibility and acceptability in a low-resource, low-literacy setting. We developed Bablibot (Babybot), a local language immunization chatbot, using Natural Language Processing (NLP) and Machine Learning (ML) technologies with Human in the Loop feature. We evaluated the bot through a sequential mixed-methods study. We enrolled caregivers visiting the 12 selected immunization centers for routine childhood vaccines. Additional caregivers were reached through targeted text message communication. We assessed Bablibot's feasibility and acceptability by tracking user engagement and technological metrics, and through thematic analysis of in-depth interviews with 20 caregivers. Between March 9, 2020, and April 15, 2021, 2,202 caregivers were enrolled in the study, of which, 677 (30.7%) interacted with Bablibot (users). Bablibot responded to 1,877 messages through 874 conversations. Conversation topics included vaccination due dates (32.4%; 283/874), side-effect management (15.7%;137/874), or delaying vaccination due to child's illness or COVID-lockdown (16.8%;147/874). Over 90% (277/307) of responses to text-based exit surveys indicated satisfaction with Bablibot. Qualitative analysis showed caregivers appreciated Bablibot's usefulness and provided feedback for further improvement of the system. Our results demonstrate the feasibility and acceptability of local-language NLP chatbots in providing real-time immunization information in low-resource settings. Text-based chatbots canminimize the workload on helpline operators, in addition to instantaneously resolving caregiver queries that otherwise lead to delay or default.

Background: The Zindagi Mehfooz (safe life; ZM) electronic immunization registry (EIR) is a comprehensive suite of digital health interventions that aims to improve equitable access, timeliness, and coverage of child immunizations through a smartphone-based app for vaccinators, web-based dashboards for supervisors and managers, text message alerts and reminders for caregivers, and a call center. It has been implemented at scale in Sindh Province, Pakistan. Objective: This study aimed to present findings from an evaluation of the ZM-EIR suite of digital health interventions in order to improve data availability and use as a contribution, among other immunization program interventions, to enhanced immunization outcomes for children aged 12-23 months in Sindh Province. Methods: The mixed methods study included (1) analysis of ZM-EIR system data to identify high-, moderate-, and low-adoption and compliance sites; (2) in-depth interviews with caregivers, vaccinators, supervisors, and managers in the Expanded Program for Immunization (EPI); and (3) pre-post outcome evaluation using vaccine coverage from the Multiple Indicator Cluster Surveys (MICS) 2014 and 2018-2019. Key outcomes of interest were improved data availability, use and contribution to immunization outcomes, including receipt of individual antigens (Bacillus Calmette-Guérin [BCG], pentavalent [Penta] 1-3, measles), full immunization (all antigens), and zero-dose children defined as children aged 6-23 months who have not received the first dosage of the diphtheria-pertussis-tetanus 1/Penta vaccine. Results: By registering newborns, providing alerts and reminders, and tracking their immunization completion, the ZM-EIR improved data availability and use in the EPI. The ZM-EIR was well received by EPI administrators, supervisors, vaccinators, and caregivers. The key benefit highlighted by ZM-EIR users was a list of children who missed scheduled vaccines (defaulters). Through greater availability and use of data, the ZM-EIR implementation, as part of a broader package of immunization program–strengthening activities in Sindh Province, may have contributed to an increase in immunization coverage and timeliness for BCG vaccinations and a decrease in zero-dose children in 2018-2019 from 2014. Additional findings from the study included the dual burden of reporting on paper and gender-related considerations of female caregivers not wanting to provide their phone numbers to male vaccinators, creating barriers to greater uptake of the ZM-EIR. Conclusions: The ZM-EIR is a promising technology platform that has increased the availability and use of immunization data, which may have contributed, along with other intensive immunization program interventions, to improvements in immunization outcomes through systematic registration of children, alerts and reminders, and increased use of data for planning and monitoring by the EPI. Trial Registration: ISRCTN Registry ISRCTN23078223; https://doi.org/10.1186/ISRCTN23078223

Despite the potential of geospatial technologies to track and monitor coverage, they are underutilized for guiding immunization program strategy and implementation, especially in low-and-middle-income countries. We conducted geospatial analysis to explore the geographic and temporal trends of immunization coverage, and examined the pattern of immunization service access (outreach and facility based) by children. We extracted data to analyze coverage rates across different dimensions (by enrolment year, birth year and vaccination year) from 2018 till 2020 in Karachi, Pakistan using the Sindh Electronic Immunization Registry (SEIR). We conducted geospatial analysis to assess variation in coverage rates of BCG, Pentavalent (Penta)-1, Penta-3, and Measles-1 vaccines using Government targets. We also analyzed the proportion of children receiving their routine vaccinations at fixed centers and outreach and examined whether children received vaccinations at the same or multiple immunization centers. A total of 1,298,555 children were born, enrolled or vaccinated from 2018 till 2020. At the district level, analysis by enrollment and birth year showed coverage increased between 2018 and 2019 and declined in 2020, while analysis by vaccination year showed consistent increase in coverage. However, micro-geographic analysis revealed pockets where coverage persistently declined. Notably 27/168, 39/168 and 3/156 Union councils showed consistently declining coverage when analyzing by enrollment, birth and vaccination year respectively. More than half (52.2%, 678,280/1,298,555) of the children received all their vaccinations exclusively through fixed centers and, 71.7% (499,391/696,701) received all vaccinations from the same centers. Despite overall improving vaccination coverage between 2018 and 2020, certain geographic areas have consistently declining coverage rates, which is detrimental for equity. Making immunization inequities visible through geospatial analysis is the first step to ensure resources are allocated optimally. Our study provides impetus for immunization programs to develop and invest in geospatial technologies, harnessing its potential for improved coverage and equity.