Usability of a digital tool to support long-lasting insecticide net distribution in Northern Bahr el Ghazal State, South Sudan

Abstract

Background

Long-lasting insecticidal nets (LLINs) have been a widely used malaria prevention method for decades. In South Sudan, LLINs are typically distributed by volunteers who use paper-based systems to collect distribution data. Paper-based systems are simple to use but have a higher occurrence of data inaccuracies and can hinder the timely use of data for decision-making. In 2022, a digital tool was introduced to collect data during the LLIN campaign in Northern Bahr el Ghazal (NBeG). The tool aimed to improve the accuracy of data entry and enable data to be used in real-time for decision making during the campaign. The digital tool was developed with offline functionality and interoperability with DHIS2 tracker version 2.8 in DHIS2 version 2.38. This study assessed the usability of the tool according to user perspectives.

Methods

A questionnaire containing open- and closed-ended questions was conducted with users of the digital tool, supervisors and other key stakeholders in five counties of NBeG. The questionnaire was administered using Malaria Consortium’s Projects Results System Android mobile application. Usability was determined through a modified and validated System Usability Scale (SUS) approach.

Results

A total of 93 participants responded to the usability questionnaire. The mean (± standard deviation) usability score across 10 SUS-scoring items was 60.91 (12.87), indicating a modest level of usability. The majority of users agreed the tool was useful for managing the LLIN distribution workflow, was easy to use, reduced workload, and supported stock management and real-time campaign monitoring. There was no significant difference in the usability scores across genders, roles, and counties. Respondents with experience of both paper-based and the digital tool tended to express a preference for the digital tool over paper-based systems. The majority of respondents also reported they would recommend the digital tool to colleagues.

Conclusion

Digital tools are perceived to improve data collection during LLIN campaigns, even in remote areas where network coverage is challenging. Additional improvements can be implemented to overcome operational challenges and improve usability of the tool. Further study is needed to assess the impact of the digital tool on data quality and real-time data use.

Background

Malaria is a serious public health problem, predominantly affecting countries in sub-Saharan Africa [1]. In South Sudan, malaria accounts for 66% of outpatient consultations, 50% of inpatient consultations and approximately 30% of all mortality [2]. The Ministry of Health and National Malaria Control Programme in South Sudan have developed a National Malaria Strategy which includes measures to prevent, diagnose and treat malaria, including the distribution of long-lasting insecticidal nets (LLINs) [3].

LLINs are a proven method for preventing malaria, and have been the backbone of malaria control in Africa for over two decades [4]. LLINs are typically distributed through mass campaigns and continuous distribution channels including antenatal care and essential programme on immunization (EPI) clinics. In South Sudan, LLIN distribution campaigns are conducted every three years, as recommended by the World Health Organization [5]. During the campaigns, LLINs are typically distributed by volunteers, and distribution data are collected using a paper-based system.

Although paper-based methods are simple to use, handwritten data collection can result in inaccuracies at collection and collation stages, and data reporting is slower which hinders timely decision making [6]. Findings from the introduction of an electronic data system in seven African counties showed electronic data systems can improve data completeness, facilitate data submission and analysis, and reduce the time between data entry and data reporting, when compared to paper-based systems [7].

Despite the reported benefits of electronic data systems, digital tools have not previously been introduced in South Sudan due to a perception that volunteers with low literacy levels and limited experience with technology would be unable to use them [8]. Internet connectivity is also a challenge as only 10.9% of the population in South Sudan have access to the internet, which limits the use of digital tools in remote areas [9].

To improve data accuracy and timely decisions, a digital application was developed to collect data during the LLIN campaign. To facilitate the functionality of this digital system in South Sudan, the application was designed with two key requirements: offline functionality, so data can be collected without an internet connection, and interoperability with the Digital Health Information System 2 (DHIS2), to improve real-time decision-making [10]. Global Positioning System (GPS) functions were also incorporated into digital tool to enable supervisors to confirm that targeted households had been reached.

In 2022, a small-scale pilot of the tool was conducted. A total of 106 volunteers were trained to use the application to collect data during the LLIN campaign in Central Equatoria. After the completion of the campaign, interviews were conducted with 13 volunteers to gather feedback, with findings showing that volunteers experienced challenges with using the digital tool to correctly upload data directly to DHIS2. In 2023, a larger study was carried out during the LLIN campaign in Northern Bahr el Ghazal state. A total of 965 volunteers were trained to use the digital tool for data collection. This paper reports the findings from assessing the usability of the digital tool among a larger sample of users.

Methods

Description of the digital tool

The LLIN digital tool consists of an application developed using the Tracker version 2.8 in DHIS2 version 2.38 (see additional file 1). Data were collected using the digital tool on a Galaxy Tab A7 Lite tablet. The digital tool contains a database of state-, county-, payam- (the administrative level below county containing a minimum 25,000 people) and boma- (the lowest administrative level consisting of a collection of 4–8 villages) level information for the whole South Sudan. Users selected all administrative units i.e. state, county, payam and boma information from a dropdown list and the digital tool generates unique identifier for the household. Users then input number of people in the household and the tool generates the number of LLINs to be distributed to the residents. The tool also collects GPS location data for each household which allows supervisors to monitor the distribution in real-time and summary reports are automatically generated, collated and uploaded to the DHIS2 dashboard which is visible to decision-makers. The inclusion of location data was an important factor for the Ministry of Health and The Global Fund to fight AIDS, Tuberculosis, and Malaria as this enabled them to confirm all targeted households were reached with LLINs. In addition, the tool’s real-time reporting functionality allowed decision-makers to identify issues and mobilize a response during the campaign.

Study design

This study was designed to assess the usability of the digital tool in Northern Bahr el Ghazal to determine whether wider deployment may be beneficial. A questionnaire was designed to gather responses from users on the usability of the tool based on a modified system usability scale (SUS) approach [11].

Study setting

Northern Bahr el Ghazal state is part of the Greater Bahr el Ghazal region. The state has a total area of 30,543 km² and is made up of five counties: Aweil South, Aweil East, Aweil West, Aweil North and Aweil Centre (Fig. 1). Ethnically, most of the state’s population is composed of Dinka and Jurchol tribe members, with a minority of Luo tribe members. The predominant means of livelihood are agriculture and livestock farming for the two tribes. Floods occur annually from June to November and hinder routine life, causing internal displacements. According to the most recent Malaria Indicator Survey, in 2017, the malaria prevalence in Northern Bahr el Ghazal was 53%, the highest of any state in South Sudan, and around 66% of the population have access to LLINs [12]. Northern Bahr el Ghazal also has the lowest literacy rate in the country, with estimates suggesting only 21% of the population aged fifteen years and above are literate [13].

Fig. 1

Map showing the Northern Bahr el Ghazal region in South Sudan and the study counties

LLIN distribution implementation

The digitalized campaign was carried out in all five counties of Northern Bahr el Ghazal state from March to June 2023. A total of 965 volunteers (Table 1) including Registrars, Site Managers, Payam Supervisors, and County Health Department (CHD) and State Ministry of Health (SMOH) staff were trained to use the digital tool. The LLIN campaign was conducted using a house-to-house distribution method and COVID-19 protocols were observed.

Table 1 Participants’ distribution across survey in all counties

During the planning for the LLIN campaign figures were calculated for the population, number of households and the number of LLINs required for each area. This information was used to determine the number of volunteers and supervisors needed in each location. Prior to the distribution a sensitization meeting was conducted for government officials at state and county level and community leaders were tasked with delivering sensitization meetings at payam level. Volunteers were selected by community leaders and a list was given to the CHD and implementing partner for verification and screening of volunteers. After verification, volunteers were trained for three days.

The LLIN distribution was conducted over the course of 10 days. Registrars went door-to-door registering each households details in digital tool and distributed nets. Site managers supervised the registrars and performed the role of stock managers at Boma level issuing nets from stores entering data into the digital tool. Payam supervisor supervised site managers and troubleshooted any technical glitches with the digital tool. If further support was needed technical officers were called to support. SMOH staff were responsible for the overall campaign planning, supervision, monitoring and technical support at state level, while the CHD’s role was to provide monitoring, supervision, implementation and technical support at county level.

Sampling and data collection

The study questionnaire (Additional file 2) was designed and deployed using Malaria Consortium’s Projects Results System mobile application and administered to participants in the field by Malaria Consortium and CHD staff. A minimum sample size of 80 participants was determined to be sufficient for the study, assuming an expected mean usability score (standard deviation) of 68 (12.5) based on a previous study [14]. The sample size was calculated to determine the mean SUS score with a 95% level of confidence, precision of ± 3 and non-response rate of 10%. The questionnaire containing the SUS rating scale was administered to a total 93 respondents (Table 1). Respondents were randomly selected from a pool of 965 potential participants who were trained to use the digital tool.

Adaptations were made to the original 10-item SUS Likert-type rating scale, which combines an equal number of negatively and positively framed Likert items for assessing system usability. In addition, three open-ended questions were included in the questionnaire to collect additional data on user perspectives to aid contextualization of the SUS scores. The adapted SUS rating scale was piloted and validated with 20 respondents who did not participate in the main study. A principal component analysis was used to assess the reliability and internal validity of the adapted SUS tool. The SUS tool was deemed to have good reliability and internal validity based on satisfactory component loadings of each of the 10 items in the usability correlation matrix, Cronbach’s alpha values (ranging from 0.63 to 0.72) and Eigenvalues [15].

The 10-item SUS rating scale covered a variety of aspects of system usability, such as ease of use, need for support, complexity and perceived usefulness of the tool. Open-ended questions covered challenges experienced using the digital tool, suggestions for improvements, and overall perception and experience with the digital tool compared to paper-based systems. The questionnaire also enabled the collection of data on participants’ characteristics, including location, gender and role. The survey was carried out by seven enumerators from Malaria Consortium’s field team based in Northern Bahr el Ghazal state through visits to data collection sites. Accounts were created for each of the survey participants, and enumerators provided participants with a tablet to complete the survey and then log out of their account. The identity of the respondents was kept anonymous.

Data analysis

Descriptive statistics were used to summarize participant characteristics, expressed as frequencies and percentages for categorical variables, and means and standard deviations for continuous variables. Individual usability scores were calculated for each item and participant, which were pooled to generate mean usability scores and standard deviations across the entire study sample in accordance with the SUS scoring framework [11]. According to this framework, SUS scores have a range of 0 to 100, computed based on the 10 items in the rating scale covering a variety of aspects of system usability, with scores of excellent (> 72.5), good (62.7–72.5), ok (51.7–62.6) and poor (< 51.7) [16]. Differences in SUS scores across participant characteristics were assessed using ANOVA tests of comparison of means. Statistical significance was determined at p value < 0.05. Statistical analyses were conducted using Stata (version 16) [17]. Responses from the open-ended questions were grouped by theme.

Results

During the LLIN campaign a total of 773,387 LLINs were distributed to households across the five counties (Additional file 3).

Participants’ characteristics

A total of 93 individuals participated in the questionnaire. Respondents’ characteristics are summarized in Table 2 together with the composite SUS score for the study sample. For interpreting SUS scores, this represents an ‘ok’ level of usability [16]. Geographically, SUS scores ranged from 56.32 (11.44) in Aweil West county to 73.00 (4.11) in Aweil Central county. They ranged from 50.83 (17.74) among female respondents to 61.25 (12.67) among male respondents. In terms of user roles, scores were lowest among SMOH users and highest among CHD users. However, usability scores did not vary significantly by county (p = 0.542), gender (p = 0.204) or role (p = 0.062).

Table 2 SUS score distributions across respondents’ characteristics

Summary of item-level participants’ usability responses

Over 85% of participants responded either ‘agree’ or ‘strongly agree’ to the positively framed questions (Fig. 2). The highest scoring statement was the tool improved the quality of reporting, followed closely by respondents reporting the tool was easy to use. In addition, respondents disagreed with the majority of the negatively framed questions (Fig. 3). The only negatively framed question where the majority of participants answered ‘agree’ or ‘strongly agree’ was regarding the need for improvements to the tool. When asked to compare the digital tool with the former paper-based system the majority of participants (65.6%) did not respond to the question (Fig. 4). However, of the 32 responses, 30 (93.8%) reported the digital tool was an improvement on the paper-based system.

Fig. 2

Responses to the five positively framed questions regarding use of the digital tool

Fig. 3

Responses to the negatively framed questions regarding use of the digital tool

Fig. 4

Volunteers’ comparison of digital tool versus paper-based system

Additional user perspectives from open-ended responses

Table 3 summarizes participants’ responses to the open-ended questions. All participants responded to at least one of the open questions, however many of the responses were not detailed. Overall, respondents with experience of both paper-based and the digital tool reported preferring the digital tool for use during LLIN distribution campaigns. In addition, users reported the tool was user-friendly, made their work easier and reduced their workload. The most frequently mentioned challenges were limited access to a power source for charging tablet batteries. Other challenges included GPS reading delays and poor internet connectivity. Participants suggested that to improve the tool the digital form should be shortened. Overall, the majority of respondents (57%) considered the tool an improvement and said they would recommend it for future use.

Table 3 Thematic summary of participants’ responses to open questions

Discussion

The aim of this study was to investigate the usability of a digital data collection tool among volunteers collecting data during LLIN campaigns in South Sudan. The study demonstrated the tool has a modest level of usability among respondents. Although the score is below the globally accepted threshold for high usability (68.5), achieving even an “ok” usability score contradicts the assumption that volunteers with low literacy levels and limited experience with technology would be unable to use digital tools [14]. The usability score is also supported by the responses to the open questions, which reported the majority of participants found the tool easy to use. In addition, the majority of respondents who had experience with both paper-based and digital data collection reported a preference for the digital tool. These findings are consistent with similar studies in Benin and the Democratic Republic of the Congo where the introduction of digital data collection tools during LLIN campaigns were also found to be valuable [18, 19].

This study also identified several challenges and contextual factors influencing the tool’s usability and overall user experience. The most commonly reported challenges were issues with charging the devices, as the capacity of the power banks supplied was not high enough to charge the devices GPS loading and internet connectivity. Despite the responses regarding poor connectivity, the tool did not require internet connectivity to function, when probing further respondents mentioned they perceived the slow device loading was due to connectivity issues. Volunteers incorrectly assumed the slow loading speed was a problem with internet connectivity rather than the device itself. Through further testing we found the slow loading was caused by starting a new form before completing the previous one. Additional training may be useful to improve volunteer’s understanding of how the tool works and to enable them to troubleshoot any issues. Furthermore, power supply limitations can be remedied by providing higher-capacity power banks and mobile charging systems, which could be solar powered. Another area suggested for improvement was to shorten the digital form, however this form is based on standard reporting templates for LLIN campaigns and therefore is unlikely to change. To enhance the tool’s usability and overall benefits, it is imperative to tailor its functionality and future deployment strategies to overcome the challenges and contextual constraints identified.

Study strengths, limitations, and implications for future research

The SUS scoring method provides a quick and highly efficient way of assessing usability. It is easy to apply and is proven to quickly measure user perception with considerable precision. Notwithstanding the strength of the methods used, the study has some limitations worth acknowledging. While the data collection tool enabled the collection of open and closed responses on usability and user perceptions, the open responses were brief and not detailed enough to provide deeper insights for contextualizing the quantitative SUS scores. Secondly, SUS scores rely on respondent's perceptions and judgements which may not always be objective. Furthermore, many respondents’ did not complete the open-ended questions which could have influenced the results either way. Consequently, responses are prone to social desirability bias. The study’s lack of a control group poses an additional limitation. A controlled study design would have enabled the comparison of usability between digital and paper based LLIN campaigns and would have provided more informative insight on the comparative advantage of digitizing public health interventions like LLIN distribution. The study’s ability to determine significant differences in usability across user characteristics was constrained by sample size limitations. Further studies should consider using a larger sample to explore variations in SUS scores across user-level and contextual characteristics.

The disparity in the distribution of study participants’ gender, with an underrepresentation of women is perhaps a reflection of the gender imbalance in LLIN digitization and distribution campaigns in the study locations. Gender gaps in digital health interventions, if left unaddressed, have the potential to undermine the impact of digital tools and exacerbate existing inequities [20]. The gender imbalance seen in this study therefore warrants efforts to involve female stakeholders across the entire value-chain of public health interventions.

In addition, the usability of digital tools may be influenced by a complex interplay of technological and contextual factors. Data on other contextual factors that could potentially influence SUS scores, such as those relating to users’ literacy, level of education and other socio-economic characteristics, were not captured and considered as covariates in the analysis. This makes it difficult to compare results with globally defined thresholds and usability scores in other study settings. Furthermore, the findings from this study may not be generalizable to the whole of South Sudan due to the unique context of Northern Bahr el Ghazal state.

Finally, this study investigated the usability of the digital tool from users' perspectives and was not designed to assess of the effectiveness of the tool and its ability to improve data quality or optimize decision-making, which requires further research. Further, in-depth, qualitative assessments of acceptability, challenges and user perceptions are needed to deepen understanding of the feasibility and usability of digitalizing LLIN distribution campaigns.

Conclusion

This study demonstrates the usability of a digital tool for data collection during LLIN distribution campaigns in South Sudan, including in remote rural areas like Northern Bahr el Ghazal. Digitising LLIN campaigns has the potential to be feasible in resource-limited settings. Further work is needed to address the challenges identified during this study including changes to the tool that could improve user experience. In addition, further research assessing the degree to which digitization improves data quality and decision-making during campaigns would be valuable.