Child immunization data quality in Rwanda: an assessment of routine health information system data

Abstract

Background

Documentation and reporting of routine data by health workers is the backbone of the childhood immunization program. Immunization data from health management information systems (HMIS) in low-and middle-income countries (LMICs) are often incomplete and unreliable. In Rwanda, the immunization e-Tracker, an individual-level health management information system (HMIS) built on DHIS2 open-source software, has been implemented and scaled nationwide since 2019. The aim of this study was to assess the quality of the routine HMIS immunization data over time.

Method

Data were derived from four HMIS sources for January to December 2020 from 24 health facilities from four districts: health facility registers (paper-based), district aggregated reports (paper-based), national HMIS reports (electronic), and e-Tracker reports (electronic). We then obtained e-Tracker reports and national HMIS reports from 2022 for the same facilities and assessed changes over time. Data quality assessments were conducted for four selected childhood immunization indicators: Bacille Calmette-Guérin (BCG), Pentavalent 3 (Penta 3) and Measles & Rubella 1 (MR1). We calculated frequencies and percentage differences. Accuracy ratios were computed for HMIS reports against facility registers for 2020 and e-Tracker for 2022.

Results

In 2020, varying degrees of inconsistencies between facility registers and HMIS reports were observed, ranging from − 2.57 to 0.67% for BCG, -13.85% to -1.45% for Penta3, and − 8.30–2.00% for MR1. Only BCG data were entered in the e-Tracker in 2020. By 2022, e-Tracker completeness of Penta3 and MR1 had also increased substantially.

Conclusions

Data quality in the paper based HMIS was variable across districts and health facilities. Improvements in quality of e-Tracker data over time demonstrate increased uptake of e-Tracker use by health workers, possibly explained by the removal of paper documentation and reporting. Further improvements in data quality can be achieved by purposefully designed implementation strategies to support health workers with digital data entry.

Text box 1. Contributions to the literature
• Evaluating childhood immunization data is essential for ensuring quality healthcare, particularly in crises and for high-risk populations.
• Electronic immunization registries can improve health information system efficiency, but systematic evaluations are needed for scalability and sustainability.
• Evidence on the impact of immunization health information systems in enhancing data quality, particularly in low- and middle-income countries (LMICs), remains limited.
• This study contributes to implementation science by exploring how digital health innovations can strengthen health systems and improve immunization programs in LMICs.

Background

Routine health information systems (RHIS) or health management information systems (HMIS) are a crucial part of national health systems to ensure routine data collection and reporting by primary health care facilities [1]. For the childhood immunization program, routine data, typically collected during service provision by health workers, are used to inform health program managers and to provide health statistics on the coverage of immunization, for instance, in a country, a district, or a health facility. Data of good quality, defined as “data that are accurate, precise, relevant, complete, and timely enough for its intended purpose” [2], are essential for monitoring and implementing changes to the immunization program. High-quality immunization data help service providers make clinical decisions, planning, guide public health actions to improve immunization rates, and efficiently prioritize immunization program resources [3].

Health workers spend considerable amounts of time on recording and reporting of data. However, the accuracy of routine data collected from national vaccination programs is often disputed [4, 5], and immunization data from HMIS in low- and middle-income countries (LMICs) are often incomplete and unreliable [2]. In an LMIC context, routine data recording in health facilities has been primarily and predominantly paper-based [1, 6]. Typically, immunization data are collected using facility books or registers, in standardized tally sheets and forms. These data are subsequently compiled and summarized into counts of indicators according to standard reporting forms. Nevertheless, in several LMICs, including Rwanda, digital HMIS have been implemented for digitalizing the routine reporting, where the summarized indicators are entered into the digital HMIS. Digital HMIS refers to a tool that helps healthcare organizations to electronically collect, store, process, and share healthcare data [7]. It can transmit data faster compared to paper-based information systems and can promote data quality by reducing errors in the way data are transmitted and aggregated from the facility, to district level and to higher levels of the health system [8,9,10]. The District Health Information Software (DHIS2) has been adopted in over 50 countries [1, 8], and is one of the most widely used system for capturing routine health programs data including for childhood immunization. Despite some clear benefits of digital data collection systems, such as real-time program monitoring and easy data accessibility [11], data quality problems such as overreporting and incomplete reporting have persisted in many African countries [3, 12]. As a result, leveraging this data effectively remains a challenge. As healthcare systems increasingly adopt digital solutions to meet the growing demands and expectations of individuals—particularly in patient-centered care—ensuring the accuracy and reliability of collected data is essential [11, 13,14,15].

In Rwanda, the childhood immunization program covers children under 2 years, and it includes BCG vaccine (tuberculosis), DPT-HepB-Hib or pentavalent (diphtheria, tetanus, pertussis, hepatitis B, and Hemophilus influenza type b), oral polio vaccine or OPV (poliomyelitis), inactivated polio vaccine or IPV (poliomyelitis), pneumococcal conjugate vaccine or PCV, rotavirus or RV, and measles and rubella (MR). The national coverage of child immunization has steadily increased over time and has hovered around 98% for complete immunization since 2012 [16]. DHIS2 was first introduced into the routine HMIS for aggregate reporting of indicators. An individual-level digital data collection system, a DHIS2-based e-Tracker, has been implemented since 2019 with the goal of improving the quality of routine immunization data and ultimately strengthen the performance of the immunization program [17]. However, problems with data quality continue to be reported such as data inconsistencies between coverage rate reported at national level using survey data and HMIS reports [18, 19]. Comprehensive evaluations of the use of the e-Tracker in terms of data quality and comparisons with other, paper-based data sources in the routine HMIS have not been performed. The aim of this study was to assess the quality of the routine immunization data in the routine HMIS at the start of implementation of the e-Tracker and 3 years after implementation.

Methods

Study setting

Our first data collection was carried out between April and December 2021 in five Rwandan districts, one in each of the five provinces of Rwanda. The districts were selected based on national data reports from the e-Tracker and the national HMIS, covering the first three months of 2020 for all districts in Rwanda. The analysis focused on three immunization indicators—Bacille Calmette-Guérin (BCG), Measles 1, and Pentavalent 3—to assess data completeness in the e-Tracker compared to HMIS-reported figures. This assessment was conducted by the immunization program manager and researcher (TU). To ensure a diverse representation of e-Tracker usage and geographic coverage, five districts were selected: Rwamagana (best performer, > 80%), Rubavu (worst performer, < 15%), and Gasabo, Gicumbi, and Kamonyi (moderate performers, 50-60%). Additionally, six health centers were randomly chosen from each of the five districts. Rwamagana district serves a population of 484,953 populations through 15 health facilities and Rubavu district has a population of 546,483 populations with 15 health facilities. Gicumbi district has 14 health facilities serving a population of 448,824 while Kamonyi district has 13 health facilities serving a population of 450,849. Gasabo district in the City of Kigali has 16 health facilities and serves a population of 879,505 [20].

The HMIS workflow at the primary health care facility is set up in such a way that an immunization nurse at the health facility collects all routine immunization data in facility paper registers (books) during immunization sessions and immunization cards held by clients (Fig. 1). Once a month, the immunization nurse summarized reports of all immunization indicators in a standard paper form, based on the nurse’s book registers, which is then sent to the district. A copy of this aggregated paper-based health facility report submitted to the district, is handed to the data manager, who enters the information into the digital HMIS at the end of the month. At the district level, the district supervisor consolidates all the health facility reports and sends the compiled report to the national level (Fig. 1). The e-Tracker was implemented in all primary health facilities and in all districts of Rwanda from 2019. For the first 3 years (2020,2021,2022) after implementation, data in the paper-based registers were then entered into the e-Tracker at the end of the day or sometimes later by the nurse or the data manager at the health facility.

Fig. 1

Workflow in the Health Management Information Systems (HMIS) for the routine immunization program in health facilities in Rwanda during 2019–2021

From October 2022, the e-Tracker replaced paper-based documentation and served as a primary data collection tool for the immunization program, as per a ministerial decision (Fig. 2). The e-Tracker has functionalities to generate automated reports of aggregate counts, which are accessed by the immunization nurse and handed over to the data manager who then completes the digital aggregate HMIS reports.

Fig. 2

Workflow in the Health Management Information Systems (HMIS) for the routine immunization program in health facilities in Rwanda from 2022

Study design and data collection

We conducted a descriptive cross-sectional study design to assess the data captured in the child immunization program for January-December 2020. Data quality was only assessed using the dimensions of accuracy and consistency. The verification of reporting consistency involved the review of health facility source documents (facility registers) in order to assess the reporting accuracy for selected indicators. Data were collected for three key immunization indicators, Bacille Calmette-Guérin (BCG), Pentavalent 3 (Penta3), and Measles & Rubella 1 (MR1). Data collectors, who were researchers, or master students of health informatics, visited each of the 30 health facilities and collected data from the facility’s paper registers and monthly aggregated paper reports. For each immunization indicator, data collectors counted the number of children that received the vaccine as available from these two data sources. Data were entered in Microsoft Excel software. Two data collectors verified a random sample of data after data entry to check for errors. Access to e-Tracker reports and HMIS national reports for the same period for the selected health facilities was requested from the joint Divisions of Research Innovation & Data Science, Maternal-Child health, and Immunization Department of the Rwanda Biomedical Centre (RBC) in the Ministry of Health. Regarding the follow-up assessment during January-December 2022, we obtained access to the e-Tracker reports and HMIS national reports for the selected health facilities, from the relevant authorities. The first author transferred these data into Microsoft Excel, and the second author verified the data for data entry errors.

Data analysis

For 2020, we analyzed data from four sources: the facility registers, the facility reports sent from the health facility to the district, HMIS reports and e-Tracker reports. First, we generated frequencies per district per immunization indicator using data from each of the four sources. We calculated percentage difference first between counts from facility registers and facility reports, and then between facility registers and HMIS reports. For 2022, we analyzed data from two sources: the e-Tracker reports and HMIS reports, and generated frequencies per district for each immunization indicator. Further, we calculated accuracy ratios to compare the source health facility-based document (facility registers in 2020 and e-Tracker in 2022) to the HMIS reports, using the following formula:

$$\begin{array}{l}Accuracy\:ratio\\= {\rm{(}}\left( {recounted\:number\:of\:events\:from\:the\:source\:document} \right)\\/(reported\:number\:of\:events\:from\:HMIS{\rm{))}}*100\end{array}$$

According to the WHO toolkit on data quality assessment [21], an accuracy ratio of 100% indicates a perfect match between recounted figures from the source document and the HMIS. An accuracy ratio less than 100% or greater than 100% is considered over-reporting and under-reporting, respectively. The acceptable margin error for the discrepancy is an accuracy ratio between 90% and 110%. An accuracy ratio < 90% indicated “data over-reporting”, which means more data being reported to the higher level than found in the source data (facility registers in 2020 and e-Tracker in 2022). An accuracy > 110% indicated “data under-reporting”, which demonstrates lower numbers being reported to the higher level than recorded in the source data (facility registers in 2020 and e-Tracker in 2022).

Results

We analyzed data from 24 health facilities in 4 districts. Health facility reports for all selected health facilities (n = 6) from one district were missing, for reasons beyond our control and independent to our project. Hence, these facilities were excluded from comparative analysis, and our data were focused on 4 districts. Percentage differences between data from facility registers and facility reports were smallest for BCG and ranged from 0.02% in district 1 to 1.18% in district 3 (Table 1). Lower number of immunizations were found in the health facility reports of the 4 districts for Penta3 and MR1 compared to the facility registers, as shown by the negative percentage difference. District 4 recorded higher numbers for all immunization indicators in the facility reports than facility registers (Table 1).

Table 1 Recounted number of immunization indicators (BCG, Penta 3 and MR1) reported during 2020 and differences by data sources per district

In contrast, HMIS reports from district 4 had fewer numbers of immunization records compared to the facility registers (percentage difference of -2.57% for BCG, -13.85% for Penta3 and − 8.30% for MR1), pointing to less information submitted to HMIS reports than the one reported in the facility registries in each health facility (Table 2).

The accuracy ratio for BCG, calculated as number of immunizations in the facility register divided by the number of immunizations in the HMIS report multiplied by 100, ranged between 99.3% and 101.6% in districts 1–3 while in district 4, the accuracy ratio was 102.5%. Overall accuracy ratios for Penta3 were 105.2% and MR1 were 102.1%. These results show relatively good accuracy, and we observed no indication of systematic under- or over-reporting.

Table 2 Recounted number of BCG, Penta 3 and MR1 during 2020: differences in facility registers versus HMIS reports per district

Regarding the comparison with the e-Tracker data, in 2020, only data on BCG were entered into the e-Tracker (n = 20,547), which was 75% of the data recorded in the source document (n = 27,511 in facility registers) (Fig. 3). Data on Penta3 and MR1 were largely not entered in the e-Tracker (n = 3,591) and (n = 1,363) but reported and submitted through paper-based forms (facility reports) (Fig. 3). A comparison of all recounted and submitted reports by indicators across districts in 2020 were summarized in a separate file (Additional file 1).

Fig. 3

Total number of immunizations retrieved from four data sources: the e-Tracker, facility registers, facility reports, and HMIS reports for n = 24 health facilities in 4 districts in 2020

In 2022, we found that the use of the e-Tracker had improved significantly. Numbers recorded in the e-Tracker was more comparable to the HMIS reports for October-December 2022 compared to January-September 2022 for all indicators (BCG: 29,390 in e-Tracker reports vs. 30,047 in HMIS reports, Penta3: 17,047 in e-Tracker reports vs. 26,067 in HMIS reports, MR1: 14,362 in e-Tracker reports vs. 24,933 in HMIS reports). The differences between the number of immunizations reported in the e-Tracker and HMIS reports were smallest for health facilities in district 2 (Fig. 4) for all indicators in 2022, as shown in the box plots in Fig. 4 indicating median and interquartile range (IQR) (BCG: median = 13, IQR = 41; Penta 3: median = 23, IQR = 17; MR1 = 22, IQR = 149) followed by district 3 (BCG: median = 60, IQR = 47; Penta 3: median = 141, IQR = 322; MR1 = 257, IQR = 413) and district 1 (BCG: median = 204, IQR = 319; Penta 3: median = 346, IQR = 558; MR1: median = 437, IQR = 593). The differences between the number of immunizations were highest for health facilities in district 4 (BCG: median = 260, IQR = 540; Penta 3: median = 303, IQR = 1506; MR1 = 437, IQR = 1669) (Fig. 4). Complete results of comparison of e-Tracker data and HMIS data by indicators across districts in 2022 are provided as an additional file (Additional file 2).

Overall, there was an improvement in the accuracy ratio during 2022 with the e-Tracker data becoming more consistent with the HMIS reports. Accuracy ratio for 2022, were over a 100% for BCG for districts 1 (114.4%) and 3 (105.3%), showing that this indicator was under-reported in the HMIS reports compared to the e-Tracker recorded data. An over reporting was observed for the district 2 and 4 with the ratios of 96.0% and 89.2% respectively. Penta3 (accuracy ratio 68% in district 1, 90.0% in district 2, 79.7% in district 3 and 44.7% in district 4) and MR1 indicators (accuracy ratio of 60.3% in district 1, 81.6% in district 2, 68.9% in district 3, and 37.87% in district 4) were over-reported in all four districts, showing sub-optimal completeness of these data in the e-Tracker. Accuracy ratios during the last three months of 2022 were higher than the rest of the year (Fig. 5), at 102.2% for BCG, 83.9% for Penta3 and 78.3% for MR1.

Fig. 4

Boxplot showing differences in number of immunizations between e-Tracker data and HMIS reports in 2022– results by district

Fig. 5

Accuracy ratios of e-Tracker reports for BCG, Penta3 and MR1 in 2022 in n = 24 health facilities in 4 districts

Discussion

In this study, we assessed the quality of routine data from the childhood immunization program in Rwanda by evaluating the consistency and accuracy of aggregate reports versus source data– paper-based facility registers in 2020 and the immunization e-Tracker in 2022. Our results showed varying degrees of inconsistencies for paper-based data sources, although within acceptable range. The e-Tracker was used to only record BCG immunizations in 2020. This situation, however, has improved in the year 2022, where Penta3 and MR1 were recorded in addition to BCG.

Data quality remains an issue in HMIS in LMICs. Inconsistencies in the HMIS data have been reported in several comparable health system settings in Eastern and Southern Africa [22]. Data in aggregate reports were incomplete as observed in studies in Kenya and Ghana [12], data discrepancies, mostly over-reporting, were observed in Nigeria [2], and poor data quality was reported in Ethiopia [23]. Similar discrepancies between recorded and reported data were observed in our study, both in the paper-based HMIS and in the e-Tracker-based HMIS. This could probably be due to the data entry repetition at various stages of paper-based data recording and reporting, and parallel use of paper-based and electronic based documentation [24]. Countries are transitioning from paper-based to digital systems for recording and reporting of immunization data. There is evidence in previous studies and a systematic review for a comprehensive improved documentation and reporting after the introduction of electronic based system [25,26,27,28]. The systematic review highlights the opportunity for new technologies and digital transformation for improving the quality of data and care [29].

In 2020, when the HMIS was predominantly paper based, the accuracy ratios of HMIS reports against facility reports was over-reported within acceptable margin of accuracy, with 97.3% for BCG, 99.8% for Penta3 and 98.3% for MR1. Similar to our study, a study conducted in upper region of Ghana assessing the data quality of routine immunization found inconsistencies between facility registers and facility reports, often showing over-reporting of data in submitted reports at higher levels of HMIS [2], although within the acceptable range [2]. Studies conducted in other settings found similar inconsistencies with over-reporting of reported Penta3 and MR vaccines data [30] and under-reporting of the same indicators [31] also within the acceptable range. Other previous studies, for example those conducted in Volta region of Ghana and Mozambique [32, 33], showed inconsistencies outside the acceptable range for Penta 3 and BCG with accuracy ratios of 64% and 44% respectively. In our study, in the paper-based HMIS, the accuracy ratio for all immunization indicators was within the acceptable range and was nearing acceptable range against the e-Tracker data for BCG.

Inconsistencies found in our study between facility registers, facility reports and HMIS reports could probably be linked to data loss during aggregation, arithmetic errors during monthly compilation, or data manipulation [2]. Point-of-care digital tools such as the e-Tracker can potentially eliminate these issues but often take time for health workers to optimally use them. At the start of implementation in 2020, the e-Tracker was mostly used for recording and reporting of BCG immunizations, to ensure every newborn is registered in the e-Tracker as soon as possible after birth as shown by our study. It was used as a secondary data collection tool, where health workers were required to enter data in the e-Tracker in addition to the existing documentations in the paper-based HMIS. Research conducted in Zambia and Tanzania revealed that the use of electronic immunization registry decreased over time in the context where it was used in parallel with paper based documentation compared to when used exclusively [34]. Lack of sufficient training of immunization nurses (users), and lack of technical functionalities to fit immunization clinical practice, as well as the provision of incentives on BCG indicator in so-called performance-based financing have been reported in a recent qualitative study of health worker experiences [35]. The 2020 covid-19 pandemic and the subsequent restrictions in the year following the implementation of the e-Tracker could have affected the planned trainings, that lead to low use of e-Tracker and poor data quality of e-Tracker data. On the other hand, the new covid-19 tracker has boosted the implementation of similar trackers by the public health authorities. Significant improvement in e-Tracker reporting were observed in our results in the year 2022, particularly after the issue of a ministerial order to phase-out paper documentation, and provision of additional trainings to the users. During this time, accuracy ratio for BCG indicator reached acceptable levels of discrepancy (102%), and nearly acceptable range for other indicators (83.9% for Penta3 and 78.3% for MR1). Although not as high as when the HMIS was fully paper-based, the HMIS with the e-Tracker appears to show improvements in terms of data quality. Similar findings have been reported in earlier research that compared paper-based documentations to digital systems [27, 36,37,38,39], with digital systems showing better data quality, completeness, timeliness and consistency.

An important strength of our study is that it addresses a gap in research on data quality and scale-up of individual-level digital tools for immunization HMIS, as several LMICs health systems are transitioning from paper to digital. Our research on data quality of electronic immunization registries is a first step toward more effective data utilization. When coupled with big data analytics, machine learning, and data integration systems, these tools have the potential to enhance public health surveillance and empower data-driven decision-making [11]. The selection of the indicators for our analysis was based on the WHO recommendation that all children should receive at least one dose of Bacille Calmette-Guérin (BCG), three doses of the diphtheria-tetanus-pertussis (Penta3) vaccine, and at least one dose of the measles vaccine [40, 41], as well as stakeholder priorities in Rwanda. We selected districts and health facilities that were diverse, both geographically and performance-wise, which we believe makes our findings more generalizable. We analyzed all the data sources in the HMIS in 2020, both paper-based and digital, that provided a baseline for data quality and allowed interpretations of changes over time. Our quantitative results can be interpreted in light of findings from qualitative interviews of health workers on e-Tracker use conducted in the same health facilities [35].

Health workers reported several barriers to transition from paper to digital, such as connectivity problems, lack of sufficient training, high staff turnover, and lack of sufficient technical support, which may explain the underlying cause for the discrepancies between e-Tracker and paper reports shown in our analysis. Similar barriers have been reported in other resource-limited settings, and can ultimately impact data quality [42]. Since paper registers were removed from health facilities in October 2022, we were unable to replicate analysis of 2020 data and make comparisons of data from facility registers against e-Tracker reports, and HMIS reports. This is one of the limitations of the study. We assumed that the automated reports generated by the e-Tracker accurately reflected data recorded in individual client records. Any potential discrepancies in automated aggregation of e-Tracker reports were, therefore, not uncovered by our analysis.

Conclusion

Paper-based reports on childhood immunization in the HMIS in Rwanda were generally good in terms of its data quality. The data shows clear improvements in the adoption and use of the e-Tracker, with a tendency towards further improvements in data quality on removal of repetitive data entry in 2022. Tailored implementation strategies to enhance e-Tracker use by immunization nurses can further improve use and data quality.