Implementation of an information system for tuberculosis in healthcare facilities in Indonesia: evaluation of its effectiveness and challenges

Abstract

Background

Tuberculosis (TB) surveillance in Indonesia is currently supported by the recording and reporting technology Tuberculosis Information System (Sistem Informasi Tuberkulosis [SITB]). SITB is used by all health workers responsible for TB programs in health facilities or hospitals throughout Indonesia. The satisfaction and ease of use of SITB affect the speed and accuracy of TB case reporting. Therefore, evaluating the information system’s quality is crucial. SITB utilization is comprehensively assessed using the End-User Computing Satisfaction (EUCS) method, which considers the dimensions of content, accuracy, timeliness, and ease of use. Furthermore, the effectiveness of the user interface using heuristic evaluation methods is critical. The primary objective of this study is to conduct a comprehensive evaluation of the System for Information on Tuberculosis (SITB) focusing on its usability and user satisfaction.

Methods

This study employed a multi-method design approach. First, a heuristic evaluation was conducted by three expert user experience designers. Second, a cross-sectional survey of 115 SITB users was carried out across all health facilities in the Special Region of Yogyakarta, Indonesia, for reporting TB cases, using the EUCS method. Finally, in-depth interviews were conducted in 10 health facilities, focusing on identifying specific attributes that needed improvement to enhance user satisfaction.

Results

The evaluation indicated that user convenience and timeliness require improvement, as well as the match between the system and the real world and error prevention, as shown in the heuristic evaluation of the SITB user interface. These findings were confirmed through interviews that identified the need for user manuals, server repairs, and information and SITB forms that are suitable for TB management in the field. The overarching assessment of the SITB reveals persistent challenges in data entry, attributed to the functionality of the application, thereby potentially impacting user satisfaction.

Conclusions

SITB application requires “major” improvements to make recording, reporting, and tracing activities more efficient and precise. Moreover, the integration of existing information systems in healthcare facilities is essential to support the tuberculosis surveillance reporting system in Indonesia.

Text box 1. Contributions to the literature
• This study comprehensively evaluates SITB implementation, offering insights into its real-world effectiveness and challenges through a multi-method approach. By combining three distinct evaluation methods, it provides a holistic understanding of user satisfaction and system usability.
• The findings highlight the potential of digital health information systems in enhancing TB case detection and management, as demonstrated by the record-high case notification rates achieved after SITB implementation.
• This study pinpoints key challenges in system implementation across diverse healthcare settings, laying the groundwork for targeted improvements in future iterations of SITB and similar systems.
• By focusing on Indonesia—a country with a high TB burden and geographical diversity—this research offers valuable lessons for other developing nations grappling with similar challenges in health information system implementation.

Background

Indonesia ranks 2rd among countries with the highest number of tuberculosis (TB) cases worldwide. The government seeks to accelerate the elimination of TB by 2030 [1]. To address these challenges, Indonesia aims to implement six strategies, one of which is to strengthen TB management by improving recording and reporting using a TB information system called Tuberculosis Information System (Sistem Informasi Tuberkulosis [SITB]). The development of a TB information system in Indonesia commenced in 1995, during a period when the system was manual and relied on TB register forms. In 2004, the recording and reporting system transitioned to an electronic form using excel and email. Then, in 2009, the government adopted the E-TB manager for recording drug-resistant TB [DR-TB]. In 2012, the government gradually developed a web-based TB information system called Sistem Informasi Terpadu Tuberkulosis [SITT] for recording drug-sensitive TB [DS-TB].

The implementation of recording and reporting using SITT has been ongoing for approximately 5 years, with a hybrid process due to reports being sent via email from primary health facilities and hospitals to the health office. In 2017, the government initiated the development of SITB to integrate records for both drug-resistant and drug-sensitive TB. After a two-year development process, the new system’s implementation began in 2020, supported by extensive workshops for TB managers in health facilities (Fig. 1).

Fig. 1

SITB roadmap and milestones (1995–2022)

The variables in SITB align with manual TB registers. While manual registers include 16 types (Table 1) [2], SITB streamlines the process by defaulting data for TB 01–03 in each facility’s system. This eliminates the need to specify data for register 1 (TB 01). Officers only need to enter the patient’s medical record number or NIK, following the illustrated variables in SITB. Figure 2 shows the TB register recording and reporting process, while Fig. 3 illustrates the SITB reporting interface. The website http://www.sitb.id/ facilitates TB surveillance through SITB. Implemented in 2020, SITB replaces manual TB registers and the previous partially paper-based Integrated TB Information System.

Table 1 Types of registers for recording and reporting TB cases

Fig. 2

Registers used for recording and reporting

Fig. 3

SITB user interface: reporting section

In line with these strategies, SITB serves as the primary reporting system for recording TB cases nationwide. Health workers managing TB programs at health service centers use SITB for tuberculosis surveillance in Indonesia. This national-level TB system streamlines tuberculosis case management, covering the entire patient journey from initial visits and laboratory tests to medication administration, contact tracing, and follow-ups. SITB enables real-time data management, improving TB control efficiency [3]. It facilitates seamless reporting of TB incidence rates from primary health facilities to the Ministry of Health. With sections for Admin & Data Officer [DO], Laboratory, and Pharmacy or Logistics, SITB collects, manages, and analyzes TB-related data across Indonesia. Figure 4 illustrates the SITB recording and reporting flow.

Fig. 4

SITB user recording and reporting flow

The implementation of SITB in Indonesia has become crucial for TB management. However, since 2021, few studies have evaluated SITB implementation. Previous research primarily focused on three areas specifically qualitative evaluations through SITB, identification of technical implementation factors through in-depth user interviews [4, 5] and assessment of implementation readiness using the Hot Fit method [6,7,8]. Despite the valuable insights from previous research, gaps remain in comprehensively evaluating SITB. Previous studies have not addressed user satisfaction, SITB application usability, or other implementation challenges [4,5,6,7,8]. These gaps necessitate further research for a more comprehensive evaluation of SITB implementation in Indonesia, potentially enhancing its effectiveness and efficiency in tuberculosis management. Our study aims to address these gaps by thoroughly assessing SITB, focusing on previously unexplored user satisfaction and usability aspects.

Usability is critical for user satisfaction, with system ease of use determining successful implementation [9]. Various methods can evaluate usability, but heuristic evaluation generally identifies more usability issues than cognitive walkthrough or user testing [10,11,12]. Each method has strengths: heuristic evaluation excels in uncovering general problems, while think-aloud protocol and user testing better identify specific task performance barriers [13]. Heuristic evaluation offers faster problem identification, though user testing reveals slightly more severe issues [12]. For comprehensive usability evaluation, combining expert perspectives (such as heuristic evaluation) with system user insights is highly recommended [13].

Usability significantly impacts user satisfaction, with system ease of use crucial for successful implementation [9]. Severity ratings help allocate resources to critical problems and estimate additional usability efforts needed. Usability problem severity combines frequency, impact, and persistence [14]. Heuristic evaluation is widely used for assessing patient safety features and identifying usability problems in medical devices, chosen for its cost-effectiveness and efficiency in uncovering issues [14, 15].

Information technology relies on human interaction for functionality, making human behavior crucial for effective technology use and user satisfaction. Users of technology are individuals with specific psychological behaviors [15]. Therefore, user behavioral aspects are key in determining technology usage success and satisfaction. The End-User Computing Satisfaction [EUCS] model can assess user satisfaction with information system implementation [16]. As SITB is the primary tool for TB recording and reporting in Indonesia, operated by health workers nationwide, evaluating user satisfaction is essential, particularly since its implementation in 2020 to replace manual processes.

The EUCS model complements the heuristic evaluation by providing a framework to assess user satisfaction with the implementation of the information system [16]. While not directly measuring user experiences, this model helps us understand the potential impact of the system’s usability on user satisfaction. By combining heuristic evaluation with the EUCS model, we aim to provide a comprehensive evaluation of SITB that addresses both usability issues and potential user satisfaction. This approach allows us to leverage the strengths of each method: the efficiency of heuristic evaluation in identifying a broad range of usability issues, and the insights from EUCS in understanding potential user satisfaction.

Information technology requires human intervention for operation, making human behavior crucial for technology use and user satisfaction. Users have specific psychological behaviors that influence technology adoption [9]. The EUCS model helps capture these behavioral aspects in our evaluation. In Indonesia, numerous health information systems are used for various diseases, making ease of use critical for SITB implementation. Therefore, this study also SITB constraints and attributes requiring improvement to enhance effectiveness and user adoption in tuberculosis management nationwide.

This study presents the first comprehensive evaluation of SITB, combining three distinct approaches: heuristic evaluation of the user interface, assessment of potential user satisfaction using the End-User Computing Satisfaction [EUCS] model, and identification of constraints and attributes that need to be in-depth interview method with individuals who are in charge of or responsible for SITB. The findings of this multifaceted study serve as a foundation for future decision-making in our ongoing multiyear research project focused on developing integration models between SITB and Electronic Medical Records [EMR] systems. This comprehensive evaluation provides crucial insights that will guide our long-term strategy for enhancing SITB’s functionality and user experience. By integrating these three aspects, we aim to evaluate SITB’s effectiveness and challenges, assess its usability and user satisfaction, and identify specific areas for improvement from the users’ perspective.

Methods

Study design

A multimethod design approach was used. First, a heuristic evaluation was performed. Then, a cross-sectional survey using the End-User Computing Satisfaction [EUCS] method was conducted. Lastly, qualitative data collection methods were employed. This study was conducted between November 2022 and May 2023.

Context, study population, and ethical considerations

In Indonesia, the reporting of TB information is a multistage process that begins at the grassroots level and encompasses public health centre (PHC) facilities, clinics, and hospitals. Information is then relayed to district or city health offices, followed by provincial health offices, and finally reach the Directorate General of Health Services of the Indonesian Ministry of Health. At every level, health and technical officers are responsible for operating the TB information systems. National-level reporting is conducted monthly, with evaluations conducted quarterly. In SITB, the business process encompasses several key stages: identifying suspected TB cases, conducting laboratory checks, confirming diagnoses, monitoring the administration and side effects of drugs, screening household contacts or areas with confirmed TB cases, and final reporting to the government.

At the health facility level, the personnel responsible for data entry in the SITB include the person in charge of the SITB, TB ward nurses, laboratory workers, pharmacists, and medical record officers and at the health office level, the TB supervisors (Wasors), both at the district and provincial levels.

This study was conducted in the first line of healthcare. The initial stage consisted of a heuristic evaluation of the SITB’s user interface by three expert user experience designers. Following these individual evaluations, we organized a discussion among these experts to consolidate their findings, address discrepancies, and reach a consensus on the most critical usability issues. The second stage involved a cross-sectional survey of 115 SITB users across all health facilities in the Special Region of Yogyakarta (DIY). We employed the One Proportion Sample formula as our research objective was to determine the proportion of individuals satisfied with SITB [17]. A sufficiently large sample was necessary to establish a more valid proportion across categories. Our focus was on the “very satisfied” category, for which we set an expected proportion of 0.5 due to the unknown true proportion in the population. To ensure the accuracy of our results, we utilized a 95% confidence level with a 0.05 margin of error. The final stage employed qualitative data collection methods in 10 health facilities from the 5 regencies of the DIY province, aiming to identify specific attributes that require improvement to enhance user satisfaction.

Variables, techniques, and collection tools

This study focuses on several key variables: the usability of the TB information system SITB; user satisfaction evaluated through five dimensions—content, accuracy, format, ease of use, and timeliness; and the study identifies specific attributes that require improvement to enhance the effectiveness of SITB from the user’s perspective.

In the present study, the second stage employed a total sampling technique. Of the 136 health facilities, 115 agreed to participate, a decision influenced by accreditation factors. The third stage involved conducting interviews in 10 health facilities, 1 from each district in 5 regions: Sleman, Bantul, Kota Yogyakarta, Gunung Kidul, and Kulon Progo. These facilities were selected through purposive sampling based on the recommendations of the Provincial Health Office, targeting areas with the highest number of cases in each region.

In the first stage of the study, SITB evaluation was based on “10 Severity Ratings for Usability Problems,” which focused on assessing specific aspects of the system. One of the methods used to identify problems in a functional system is heuristic evaluation, which involves assessing whether a design violates the 10 usability principles [18, 19]; H-1, visibility of system status; H-2, match between system and the real world; H-3, user control and freedom; H-4, consistency and standard; H-5, error prevention; H-6, recognition rather than recall; H-7, flexibility and efficiency of use; H-8, aesthetic and minimalist design; H-9, help users recognize, diagnose, and recover from error; and H-10, help and documentation.

For the second stage, data was collected by administering questionnaires to healthcare personnel who operate the SITB, with these questionnaires being adapted from “The Measurement of End-User Computing Satisfaction.” The EUCS model identifies five factors that can affect user satisfaction with the implementation of an information system: content, accuracy, format, ease of use, and timeliness. EUCS is an overall evaluation of user experience in using the information system. Moreover, user satisfaction can be measured using an information system (EUCS) to evaluate the effectiveness of the design and implementation of the system [20]. Therefore, according to research, the EUCS model is widely recognized and validated as the most useful for measuring end-user satisfaction and information system implementation. In the final stage, data was obtained using in-depth interview techniques with individuals who are in charge of or responsible for the SITB.

Statistical analyses

The descriptive statistics used in this study were the averages for each variable. In the first stage, severity rating was used to determine which usability issues should be addressed first, based on the highest rating of the most serious usability problems [14, 18]. The scale of the severity rating is as follows: 1, not a functionality issue; 2, a superficial issue that does not need fixing unless extra time is available (cosmetic problem); 3, a minor issue thus the fix has low priority; 4, a major issue that is important to fix and should be given high priority; and 5, a catastrophic issue that should be fixed before the product can be released.

For the second stage, user satisfaction with the SITB application was measured using the EUCS method, examining five factors, with different question formats [20]. These factors include content, consisting of six questions; accuracy, consisting of six questions; format, comprising seven questions; ease of use, consisting of six questions; and timeliness, which includes four questions. The validity of the instrument was confirmed using the Fornell-Larcker criterion. All constructs demonstrated satisfactory convergent validity with Average Variance Extracted (AVE) values exceeding 0.5, and reliability was established with Cronbach’s alpha coefficients greater than 0.7, as illustrated in Fig. 5. These results indicate that the instrument used in this study meets the necessary standards for validity and reliability in measuring the constructs of interest [21,22,23]. To evaluate user satisfaction with the SITB system, we employed Partial Least Squares Structural Equation Modeling (PLS-SEM) using SmartPLS software. This advanced statistical technique allows us to examine complex relationships between variables and is particularly suitable for analyzing the multidimensional nature of user satisfaction.

Fig. 5

Pathway model of end-user computing satisfaction for SITB application

Content analysis was applied to analyze the qualitative data gathered from the in-depth interviews. This process involved transcription, coding, categorization, theme identification, and interpretation of results. The analysis followed a structured approach, categorizing information by topic. Based on the content analysis, three main themes emerged: (1) The high volume of applications overwhelming processing officers, (2) The SITB application’s design enhancing the monitoring of TB patient care processes, which has been shown to improve cure rates, and (3) SITB’s facilitation of reporting and its potential to improve policy-making processes. These themes provide valuable insights into the user experience and impact of the SITB system [24].

Results

This section consists of three interrelated chapters that form a comprehensive triangulated evaluation approach. The chapters include a usability evaluation of the SITB user interface using the heuristic evaluation method, an evaluation of SITB user satisfaction in health facilities using the EUCS model, and in-depth interviews with SITB users to identify attributes requiring improvement to enhance the effectiveness of electronic and centralized TB reporting.

Heuristic evaluation

Based on the heuristic evaluation conducted by three evaluators with similar findings, nine problems were identified. However, in H-1 and H-9, no issues were found. The first problem was identified in H-2, The inconsistency in terminology and the interchangeable use of abbreviations, like “TB” and “TBC,” are acknowledged. Nevertheless, these issues are considered less critical compared to the functional challenges. Functionality issues were identified that led to discrepancies between the system’s behavior and real-world scenarios. Specifically, when there is one individual in a family within a house suspected of being exposed to the disease, data entry in the SITB for tracing is limited to that single household. However, tracing should encompass a larger area, including approximately 20 houses on either side. These limitations stem from the server’s capacity to handle data; increased data entry slows down the application.

The second problem was identified in H-3. It was discovered that the manual page did not have a back navigation feature. In H-4, three problems were identified, including the absence of a search menu on the laboratory page, incorrect use of the delete icon (x) instead of the trash can icon, and lack of an expand data icon on the “Reports” page.

The sixth problem identified in H-5 was related to error prevention. It was found that there was no reset feature for returning to default and that no usage instructions were provided in the manual book. Furthermore, the instructional videos for form completion cannot be downloaded from http://www.sitb.id/sitb/manual/. The seventh problem was determined in H-6. It was discovered that there was no “action” information on all pages and no “no” information in tables that had not been filled with data.

The eighth problem was identified during H-7. The minibar appears inconsistent and irregular, which affects the aesthetic and minimalist design discussed in H-8. The final problem was identified in H-10. It was found that there was no user guide on the “manual” page.

The severity rating assessment of the problems found in the SITB application indicates that the most common issues are related to the H-4 principle. Although this principle had three findings, it falls under the minor category and has a low priority for repair, representing 33% of all findings.

However, the H-2 and H-5 heuristic principles have a high average severity rating and a major category, with ratings of 2.73 and 2.72, respectively As indicated in (Table 2). This indicates that these two principles should be prioritized for improvement.

Table 2 Category severity rating evaluation of SITB application heuristics

EUCS evaluation

Description of answers by dimension

Response on User Characteristic dimension of EUCS instrument.

In the province of Yogyakarta, Indonesia, 136 health facilities are implementing directly observed treatment, short course. Of the 115 health facilities included in this study, 48 are private and government-owned hospitals and clinics, and the remaining health facilities are public health centres (PHCs).

The SITB users in these health facilities have diverse backgrounds, both regarding their educational background and working units. According to the study findings, of the 115 SITB users, majority (72%) are women who work as nurses.

The number of Health Human Resources, particularly nurses, in Yogyakarta is 10,757, with female nurses being more dominant, accounting for 75% [25]. Majority (64.3%) of SITB users in DIY health facilities have worked for > ten years and have a diploma background, and most of them are aged < 50. Details regarding the characteristics of SITB users are depicted in (Table 3).

Table 3 Characteristics of respondents of end-user computer satisfaction evaluation

SITB usage training exhibits variation among health facility personnel. Of the 115 respondents surveyed, 52 (45.2%) acquired SITB proficiency through training conducted by the health department. Most trained personnel (78.2%) participated in 1–2 training sessions. However, 13% of respondents reported no formal SITB training, necessitating independent learning through online resources or consultation with TB supervisors. Table 4 illustrates the methods utilized by users to learn how to operate SITB and the frequency of their training.

Table 4 Methods for learning the use of SITB and intensity of training

The satisfaction level of SITB application users was measured using the EUCS method, which evaluates satisfaction based on five factors, each with different types of questions. The range of satisfaction scores is divided into five categories: very satisfied (4.2–5.0), satisfied (3.5–4.1), moderately satisfied (2.6–3.4), dissatisfied (1.8–2.5), and very dissatisfied (1.0–1.7). A description of the EUCS Method Pathway Model used to evaluate user satisfaction with the SITB application is provided (Fig. 5).

Overall, users are satisfied with the SITB application with an average score of 4.08, because it has helped to facilitate the reporting process that was previously manual and required users to physically submit reports to the district/city and provincial health offices, particularly during the pandemic. However, two dimensions had an average score below 4 points: ease of use (3.83) and timeliness (3.98). Some users are dissatisfied with these two components, especially in the indicators “SITB: Fast to learn” and “Loading response speed when switching menus/pages,” as shown in (Table 5). However, statistically, all indicators of the dimensions, including content, accuracy, format, ease of use, and timeliness, significantly influence end-user satisfaction (T-statistic > 1.96 or P-value < 0.05). The following (Table 6) is the frequency distribution of the overall user satisfaction with SITB, as measured by using the EUCS method.

Table 5 SITB user satisfaction from five dimensions (content, accuracy, format, ease of use, timelines)

Table 6 SITB user satisfaction using the end-user computing satisfaction (EUCS) method

*Satisfaction level scale, Mean 3.5-4 Very Satisfied; 3-3.49 Satisfied; 2.5–2.99 Moderately Satisfied; 1.5–2.49 Dissatisfied; 0-1.49 Very Dissatisfied.

Attributes that should be improved from the user’s viewpoint

Structured interviews were conducted with TB managers from 10 health facilities involving 20 SITB users, using in-depth interview method. The results revealed that several problems frequently arise when operating the SITB system (Table 7). These problems are expected to provide input and suggestions for improving SITB attributes from the user’s perspective to enable effective and efficient data collection, tracking, and reporting of TB cases. SITB remains controversial owing to reporting that cannot be done during working hours because of server load at the center, frequent system errors, and several TB service management forms that are not included in SITB.

Table 7 Attributes that need to be improved from the user’s point of view

Since January 2023, issues related to incomplete patient identity in data entry have been resolved according to user expectations. The government has integrated SITB with Population and Civil Registration Agency (Dukcapil), enabling users to complete their personal and social identity by entering an identity number (NIK). The TB programmers expressed their satisfaction with this integration, as it has made capturing cases easier and more efficient, as stated by one of the informants A 42-year-old female nurse from a public hospital expressed optimism about this feature:

“.Yes, I believe this form of integration with the NIK will enhance data completeness and facilitate case capture…” (Informant 02_female, 42 years old, nurse at public hospital).

SITB integration with Dukcapil coincided with the launch of the Indonesian Health Service (Satu Sehat) platform owned by the Ministry of Health, which made it easier for SITB users in all health facilities to access patient identity information. However, a few things have not been integrated, such as referral, treatment, and integration of SITB into EMR. Thus, users still have to make double entries in different information systems. One of the informants (Informant 10) explained, “.We still have to enter the same data to the EMR and to SITB, so entering data into SITB cannot be done in real time because we have a lot of work. We have to wait for the patient service to finish, then we will enter the SITB data by looking at the information on the EMR…”(Informant 10_female, 34 years old, nurse at Academic Hospital).

Discussion

The evaluation of the System for Information on TB (SITB) in Indonesia, including the Evaluation of User Convenience and Satisfaction (EUCS), has shed light on several critical areas that warrant attention for enhancing the system’s effectiveness and user satisfaction. This section discusses the key findings related to user convenience, timeliness, system-real world alignment, error prevention, and challenges in data entry, as identified through heuristic evaluation and interviews, including the EUCS. Additionally, the discussion explores potential strategies to address these issues, such as the development of user manuals, server maintenance, and the design of information and forms tailored for TB management in field settings. By addressing these aspects, it is possible to improve the overall usability and functionality of the SITB, ultimately contributing to more effective TB control efforts in Indonesia.

Heuristic evaluation

Heuristic evaluations are crucial in implementing health information systems because they can identify several challenges, including those with major and catastrophic severity categories. It is crucial for system designers and developers to address these issues [26, 27].

Research has shown that heuristic methods are more effective than cognitive tracing methods in identifying problems in health information systems [28]. Therefore, heuristic evaluation is a critical and effective technique for diagnosing existing issues and improving patient safety and treatment quality. This evaluation provides more accurate results and helps identify the actual problems in the system [28]. Moreover, numerous studies have successfully employed heuristic evaluation to assess the utility of health service information systems [28].

Importance of creating a system that matches the real health system and services

The study revealed an incompatibility between the TB management process and system, including the absence of a menu to change drug regimens in patients with DS-TB. Thus, officers had to assume that therapy had failed, and the patient’s status was changed to DR-TB. Another finding was the lack of a menu for contact history with DR-TB in patients receiving TB prevention therapy, which is a critical component of the tracing process.

The radiology system interface did not align with real world medical practices [28]. For example, the radiographic implementation protocol listed on the system did not match the actual implementation standard operating procedure, which can cause confusion for users and hinder task completion timeliness. Similarly, it was found that 57% of errors in a heuristic evaluation involved a mismatch between the system and the real world [28]. To enhance this compatibility, health information systems should use language that is easy for users to understand, employing familiar words, phrases, and concepts instead of system-oriented terms [28].

The match between the system and the real world is a crucial indicator [28], which stressed that systems should employ language that is understandable for users. This includes using words, phrases, and everyday language concepts, rather than relying solely on systematic terms that may only be understood by system designers.

Preventing errors in the health system to ensure patient safety and effective treatment

SITB application includes a help menu to assist users with problems or errors; however, an open access book version of the user manual is not yet available. Currently, the user guide is only available as a video and can only be accessed using an SITB account. Therefore, when health facility users experience problems or errors, they should rely on colleagues or TB managers at the district office level for assistance. Furthermore, the SITB application does not have a reset menu or return-to-default feature, meaning that users should save and then edit data to make changes. Although a warning pop-up appears when saving incomplete data, this can still be improved to prevent adverse events during TB patient treatment.

Although electronic health records (EHRs) have been recognized for their potential to reduce health costs and improve care quality in hospitals, improper design and use can lead to serious, unwanted impacts. Poor EHR system design and improper use can lead to EHR-related errors that compromise the integrity of the information, endangering patient safety, or reducing care quality [28]. Furthermore, user errors and bad interfaces can interfere with receiving information, leading to bookkeeping errors during decision-making [29].

Therefore, ensuring the safety of technology and its use in clinical settings is a crucial challenge in the development of health information technology. To achieve this, we highlight nine key points, including the methods and technology used in information systems, user and function standards, security software and network-enabled clinical environments, decision support, real time methods for automated surveillance and monitoring, information on hazards and adverse events, and models to improve patient safety [30].

EUCS evaluation

According to the findings of the present study, a significant proportion of users of the SITB application were female nurses. This observation aligns with previous research that reported that 70% of female nurses are involved in the management of TB cases [31]. Notably, other studies on TB have shown that most healthcare workers involved in managing such cases are female [32,33,34]. These findings reveal that the SITB application can reach and benefit a group of healthcare workers who play a vital role in the management of TB. Future research should explore the reasons behind the high usage of SITB among female nurses and the way it can be further tailored to their specific needs.

The management of TB patients is the responsibility of TB program implementers, namely, program managers, doctors, nurses, pharmacists, and laboratory and pharmacy staff. This is consistent with the findings of recent studies by [34], which highlight the role of radiographers in TB treatment. In the province of DIY, TB program implementers have over 10 years of work experience, are of productive age, and have a diploma-level education background. These findings underscore the importance of having experienced and educated healthcare professionals to manage TB, a complex and challenging disease that requires specialized knowledge and skills [32, 35].

Studies have shown that knowledge of TB infection control among healthcare workers is a concern, with several studies focusing on the demographic characteristics of health workers responsible for TB management. Research on the level of knowledge of TB infection control found that 48.5% of health workers are diploma graduates, and the rest are either undergraduate or postgraduate [34]. Similarly, studies have found that healthcare workers responsible for TB management have a working duration of 10 years and over, with the highest age range being between 31 and 40 years [32, 36,37,38]. A study involving nine countries, including Indonesia, has revealed that most electronic TB (eTB) manager users fall within the age range of 30–49 years [39]. These findings suggest that healthcare workers with significant work experience and diploma-level education play a critical role in TB management, and interventions to improve knowledge of TB infection control should target this group of healthcare workers. Further research should explore the most effective methods for improving TB infection control knowledge among healthcare workers, particularly those in the 31–40 age range, and the impact of such interventions on patient outcomes.

The EUCS evaluation model is a suitable tool for assessing the satisfaction of users of information systems in the healthcare sector, such as hospitals and health centers. Five EUCS dimensions affect user satisfaction with information systems in the healthcare sector [40].

The EUCS model has been shown to be both valid and reliable in measuring end-user satisfaction with IT systems. Content, accuracy, format, and timeliness play important roles in determining end-user satisfaction and the quality of the information provided [41]. Moreover, the EUCS method has been successfully applied to different contexts as a valid measurement of computer user satisfaction, including in the healthcare sector, while considering the policy implementation process and the context of the health information system [42].

Our findings revealed varying levels of satisfaction across different rural areas in Yogyakarta. Staff satisfaction was generally positive for content and accuracy dimensions, with scores above 3.5 on a 5-point scale. Most indicators on the content dimension were rated “very satisfied” by users, although one indicator was disputed by the assessor due to a discrepancy between the system and the manual register. However, satisfaction levels were lower for ease of use (3.2) and timeliness (3.0), often hindered by limited computer literacy and frequent server downtime of the SITB. Previous studies have highlighted the significant influence of the content dimension on user satisfaction and identified it as one of the dominant factors alongside the accuracy dimension [43, 44]. Recent research has confirmed that all indicators on the content dimension were classified as “satisfied,” with one indicator rated “very satisfied” because the system provides sufficient information.

Although most indicators in the accuracy dimension were rated “very satisfied,” the system failed to meet user expectations, particularly about frequent errors, which are often caused by server downtime. This forces officers to manually input data outside of working hours to prevent system errors. The accuracy dimension is crucial in maintaining user satisfaction, as shown by a survey of users [45, 46]. The findings are consistent with that of a research [9], which emphasizes that the suitability of information and the continuity of the system without error constraints are critical factors in ensuring patient satisfaction.

The user interface’s clarity is a dominant factor contributing to the ease of use of the SITB application. Two groups of factors affect the user interface, consisting of positive and significant variables, such as content, format, and accuracy, with content being the most positive and significant influence [47]. The format dimension, which relates to website design appearance, has the most significant impact on results [48]. A recent study has confirmed that the format dimension had the most dominant results, with a mean value of 3.59, falling into the “very satisfied” category [49]. These reveals that the clarity of the user interface is critical to the success of the SITB application and that improving the content, accuracy, and format dimensions can enhance user satisfaction. Further research is required to explore strategies for addressing server downtime issues and improving the accuracy dimension to enhance user satisfaction.

The use of SITB in these health facilities can have a significant impact on the management and control of TB cases in the region. This is evidenced by the enhanced detection and reporting capabilities of SITB, which have contributed to record-high case notification achievements in 2022 and 2023. Specifically, over 724,000 new TB cases were identified in 2022, with this number increasing to 809,000 cases in 2023 [3]. Thus, the use of digital information systems strongly supports addressing health issues [50]. However, Indonesia consists of urban and rural areas with varying facilities across hospitals, which may lead to variations in the acceptance of such information systems. Therefore, it is crucial to ensure that the SITB system is designed and implemented in a way that accommodates the needs of different types of health facilities and their users, regardless of their background. Therefore, it is crucial to ensure that the SITB system is designed and implemented in a way that accommodates the needs of different types of health facilities and their users, regardless of their background.

The SITB is mandatory to be implemented across all healthcare facilities in Indonesia, of which consisted of 10.180 primary health care (Puskesmas) and more than 1000 hospitals covering rural, urban and remote areas [51]. This comprehensive implementation highlights the importance of examining the system’s effectiveness and challenges in diverse contexts. Yogyakarta, with its mix of urban and rural areas, serves as a crucial microcosm for understanding these aspects on a national scale [52]. We measured user satisfaction using the End-User Computing Satisfaction (EUCS) model, adapted to the rural context. While we did not conduct a formal infrastructure readiness assessment specifically evaluate SITB users’ computer proficiency in this research.

Attributes that should be improved from the user’s viewpoint.

• 1. High volume of applications overwhelms processing officers.

  Health facilities are currently required to input disease data into ten different applications, including SITB for TB; SIARVI for DHF; SISMAL for malaria; SILANTOR for chikungunya; E-FIlCA for filariasis; SKDR for worms, pneumococci, yellow fever, influenza, rotavirus, Mers-COv, typhoid, cholera, and gastrointestinal and respiratory tract infections; E-zoonosis for rabies, anthrax, plague, leptospirosis, avian influenza, and toxoplasma; SIHA for HIV and sexual infections; SI PD3I for diphtheria, pertussis, tetanus, meningitis, and polio; PWS for measles; and SITASIA for leprosy and yaws. However, the World Health Organization has acknowledged that the current health information system subsystems lack coordination, leading to an excessive burden on health workers who must collect and report data.

• 2. The SITB application is designed to enhance the monitoring of TB patient care processes, which has been shown to improve the cure rate.

  SITB facilitates the identification of gaps between TB suspects and the diagnostic examinations that are performed, resulting in improved monitoring of TB patient care processes and increased cure rates. However, the electronic implementation of TB (ETR.Net) in South Africa often generates a high rate of error reporting, which negatively impacts the TB control program. Owing to the suboptimal function of e-TB, no significant effect was found on the cure rate of TB patients [43]. This highlights the need to optimize the use of electronic TB management systems to reduce error rates and improve TB control programs.

• 3. SITB facilitates reporting and can improve the process of determining policies.

  SITB provides health facilities and city or provincial health offices with critical information for developing policies, such as trends in TB diagnoses made by health facilities, trends in TB cases (DS-TB, DR-TB, HIV-TB, and DM-TB), patient treatment monitoring processes, involvement of cadres/community in contact investigations, and availability of TB logistics in each health facility. Information technology can save healthcare professionals time in tasks such as recording, maintaining, and preparing reports [44]. These reports are crucial for decision-making in both administration and patient care.

This study has substantial implications for research and practice in health information systems. We introduce a novel, comprehensive approach for evaluation by combining heuristic evaluation, the EUCS model, and in-depth interview method, which can be applied to other health information systems. Our findings underscore the importance of user-centered design in system development, potentially influencing future designs. The study’s results can inform policy decisions regarding the implementation and improvement of health information systems in Indonesia, particularly for tuberculosis management. Additionally, this work provides a foundation for future studies on integrating SITB with Electronic Medical Records systems, contributing to long-term improvements in tuberculosis management. Our comprehensive evaluation offers actionable insights for enhancing SITB’s usability and user satisfaction, potentially leading to more effective tuberculosis management across Indonesia.

Limitation of this study include the absence of formal assessment of SITB users’ readiness in terms of technology use and computer skills, nor did we perform a comprehensive infrastructure evaluation. While our study has successfully evaluated SITB using a combined evaluation method, these gaps present opportunities for future research. We recommend that future studies conduct thorough infrastructure assessments to identify potential barriers to SITB adoption across diverse healthcare settings in Indonesia. Such studies will provide a more complete understanding of SITB’s effectiveness and user satisfaction, ultimately informing improved implementation strategies.

Ethical clearance

Ethical approval for this study was obtained from the Institutional Review Board of the Medical Faculty of Gadjah Mada University (ref. no. KE/FK/1404/EC/2022). Participation in the study was voluntary, and informed consent was obtained from all participants.

Conclusions

The implementation of SITB in healthcare facilities in DIY Province Indonesia was evaluated using three methods: user side (EUCS and interviews), user interface (heuristics), and data collection and observation. The implementation was smooth, and most users found SITB easy to use for recording, reporting, and collecting data. However, several issues were identified and were resolved according to the evaluation method used. For example, the EUCS evaluation method highlighted the need for more information to enable faster learning and improve the loading speed of menu pages. The heuristic evaluation method identified error prevention as a top priority for improvement, with recommendations for creating usage instructions for TB managers. To enhance data recording efforts, the system needs updating and correcting, particularly for significant issues, to ensure that all indicators meet at least a cosmetic level. Furthermore, users hope that SITB will be integrated with EMR to avoid double entry and that the system will be supported by an additional server to prevent slow performance.