Estimating the hole surface area of insecticide-treated nets using image analysis, manual hole counting and exact hole measurements

Abstract

Background

The physical integrity of insecticidal-treated nets (ITNs) is important for creating a barrier against host-seeking mosquitoes and, therefore, influences people's perception of the net's effectiveness and their willingness to use it. Monitoring the physical integrity of ITNs over time provides information for replenishment schedules and purchasing decisions. Currently, the assessment of physical integrity of ITNs is conducted by manually counting holes and estimating their size to class the net as functional or not. This approach is laborious to routinely conduct during field surveys of ITNs. Automated image analysis may provide a rapid assessment of the physical integrity of ITNs but it is not known if the images can capture sufficient information. As a first step, this study aimed to assess the agreement between estimated hole surface areas derived from (1) manually segmented images, (2) manual hole counting compared to (3) ground truth obtained by calibrated close-up shots of individual holes.

Methods

The physical integrity of 75 ITNs purposely selected from an ongoing study was assessed by manual hole counting, image analysis and ground truth. For the image analysis, a graphical user interface was developed and used for the segmentation of holes visible in photographs taken from each side of the net. The hole surface area was then computed from this data. The agreement between the estimates from image analysis and manual hole counting was compared to the ground truth using the Bland–Altman method.

Results

There was substantial agreement between the manually segmented image analysis estimates and the ground truth hole surface areas. The overall bias was small, with a mean ratio of the hole surface area from image analysis to the ground truth of 0.70, and the 95% limits of agreement ranging from 0.35 to 1.38. Manual hole counting underestimated the hole surface area compared to the ground truth, particularly among nets with holes above 10 cm in diameter.

Conclusion

Images coupled with manual segmentation contain sufficient information to calculate hole surface area. This lays the groundwork for incorporating automatic hole detection, and then assessing whether this method will offer a fast and objective method for routine assessment of physical integrity of ITNs. While the WHO method underestimated the hole surface area, it remains useful in classifying nets as either serviceable or too torn because the cut-off is specific to this method.

Background

Insecticide-treated nets (ITNs) contributed substantially to reducing malaria cases between 2004 and 2015 and remain the cornerstone of malaria control [1]. Since 2004, about 2.9 billion ITNs have been distributed globally, of which 86% were distributed in sub-Saharan Africa [2]. Even so, in sub-Saharan Africa the percentage of the population with access to an ITN within their household is only 56% [2]. This coverage is far below the World Health Organization (WHO) target of 80% of people living in malaria endemic areas sleeping under an ITN that is needed for sustained control and elimination of malaria [3, 4].

A functional ITN is one that is present, is in good physical condition, and remains insecticidal, thereby protecting against vector-borne diseases by creating a physical and insecticidal barrier between humans and host-seeking malaria vectors that both reduces bites and kills mosquitoes [5, 6]. The WHO Prequalification Team (WHO-PQ) has listed over 25 ITN brands [7], with an anticipated lifespan of three years under typical usage [8]. However, field data shows that these ITNs typically last just 2 years [9,10,11,12,13,14,15]. Damage to the fabric of ITNs reduces their effectiveness [16], and is also the leading cause of nets being discarded from households [17,18,19,20] contributing to reduced ITN coverage [21, 22]. Longer-lasting ITNs would require less frequent replacement, reducing the need for ongoing investments in materials, labour, and distribution [23]. This would result in lower operational costs over time. Because vector control depends upon sustained coverage of interventions over time, ITNs with longer lifespans may increase the likelihood of continuous protection against vectors. Users may also be more likely to adopt and adhere to interventions that offer long term efficacy with minimal compliance or maintenance requirements.

Monitoring the physical integrity of ITNs is critical to guiding procurement decisions and devising replacement strategies [5, 19, 24]. According to the WHO guidelines [8, 24], ITN physical integrity monitoring is conducted manually by counting holes and categorizing them by their sizes (larger than a finger, thumb, fist or head) [8]. Standard hole surface areas are then assigned to each category of hole size to estimate the overall hole surface area of the ITNs [8]. While this method (hereafter WHO method) is straightforward and commonly used, it has some disadvantages. It cannot provide information on the actual hole size [25] as it assumes that all holes are accurately represented by the standard WHO size corresponding to the category and are circular in shape [26], whereas studies have shown that holes vary in shape [27] and are not circular [25]. This method may also be labour-intensive, time-consuming [25] and prone to errors during routine field surveys. These limitations contribute to the lack of fabric integrity assessments conducted after ITNs are distributed.

There is a need to develop a faster and more objective method for assessing the physical integrity of ITNs so that product-specific and location-specific differences in durability can be regularly monitored in routine field surveys. Previous studies have considered the potential of image analysis [25, 28] for the assessment of the fabric integrity of ITNs. When the WHO method and image analysis using Image J software [29] were compared head-to-head, the number of holes was comparable but the estimated total hole surface area was larger with the WHO method [25, 27]. This discrepancy was due to the assumption used in the WHO method: a circular shape for the holes [8, 24], and the use of one general formula for holes within a hole size category regardless of their shape and actual size when calculating hole surface areas.

Image analysis was reported to be labour-intensive due to the demanding image processing steps before the final hole surface area was estimated [25, 27, 30]. To address this challenge, machine learning algorithms can be combined with digital images for fast hole surface area estimation as described elsewhere [31,32,33]. This study aimed to evaluate whether these images contain sufficient information for accurate hole area calculation as a first step before introducing automated hole detection. This study compared the accuracy of image analysis using manual segmentation and the WHO method to measurements taken from close-up images of individual holes as the ground truth “reference method”.

Methods

Study area

The ITNs were assessed at the Vector Control Product Testing Unit (VCPTU) facilities located at the Bagamoyo branch of the Ifakara Health Institute (IHI), Tanzania. The ITNs were collected from a durability study which took place in Bagamoyo, Tanzania [34].

Study design

This was a cross-sectional study embedded in an ITN durability study [34]. A total of 75 rectangular, white, double-sized ITNs (190 cm × 180 cm × 150 cm) from five WHO pre-qualified [7] brands were selected based on their levels of physical damage as measured using the proportional hole index (pHI) method [8] after 36 months of community use [34]. In the present study, the area of damage for each ITN was assessed by three methods: by using the standard WHO method [8], by image analysis using images of the whole net (four sides plus the roof panel), and image analysis using a single close-up image for each hole taken next to the most appropriate size of the ArUcO marker of known size, which provided the reference values for calibration of estimated area. An ArUcO marker is a square marker composed by a wide black border and an inner binary matrix, which encodes a unique identifier (ID) for each marker. This marker provides a precise scale reference and enables the use of image correction techniques like perspective correction. The ground truth was used as a reference method because the close-up shot of each individual hole was scaled correctly using ArUcO markers to provide the most detailed and accurate results.

Frame setup and net alignment

The nets were carefully fitted onto a metal frame with a contrasting black cloth background to ensure a smooth, flat, and wrinkle-free surface for inspection and photography. This setup was essential for making each hole in the net visible in the photographs. A white grid made of Velcro was attached to the black cloth to achieve precise alignment of the net. Before capturing the photographs, the setup underwent a thorough inspection by the field team. This inspection confirmed that all holes were visible and ensured there were no unintended folds or overlapping areas of netting.

The rectangular frame measured 160 cm in length, 145 cm in width, and 150 cm in height. It was slightly smaller than the size of the new nets to accommodate potential size variations among the field-used nets caused by shrinkage (Fig. 1A) The black cloth was fitted with a white grid on each side, and the roof panel was gridded on the outside so that the grid could be seen through the net. Each grid formed a rectangular shape on the black cloth (Fig. 1B). The dimensions of the white grid were 30 cm × 30 cm or 30 cm × 10 cm on the long sides of the net, 30 cm × 30 cm or 30 cm × 25 cm on the short sides, and 30 cm × 30 cm, 30 cm × 25 cm, or 25 cm × 10 cm on the top view (roof panel).

Fig. 1

The bed net frame is set up (A) and covered with a black cloth with white gridlines before the net is placed on top (B)

The WHO method

The numbers of holes falling into size categories 1, 2, 3 and 4 representing 0.5–2 cm, 2–10 cm, 10–25 cm, and > 25 cm, respectively, were recorded on a standard hole tally sheet following established procedures [8]. The holes were not directly measured but were classified based on the WHO method as being larger than a little finger, thumb, fist and head. Holes smaller than a little finger were not counted as they are considered by the who to be too small to allow mosquitoes through. The count of holes in each size category was used to estimate the approximate hole surface area for each ITN [8]. Assuming all holes were circular, their diameter was determined to be equal to the mid-point of the hole category which was 1.25 cm, 6 cm, 17.5 cm and 30 cm for category 1, 2, 3 and 4 holes, respectively. As category 4 does not have an upper limit, a 30 cm diameter was chosen for calculating the hole surface area [8, 24]. The area of each hole was estimated as per the WHO guidelines [8]. Based on the estimated hole surface area, nets were classified as “serviceable” if the hole surface area was equal to or below 1000 cm² and “too torn” if the hole surface area was greater than 1000 cm² [26].

Taking photographs for the image analysis method

After manual hole counts following the WHO method were completed, each ITN was photographed while it remained on the metal frame covered with the black cloth. A Samsung Galaxy Tab A8 tablet computer was employed, with its camera placed approximately 250 cm away and directly facing the center of ITN panel. Before taking photographs, the trained field team ensured that all holes were clearly visible and not obscured and that there was no folding or wrinkled surface of the net. Velcro was used to ensure correct net alignment to the frame and the visibility of all holes are fixed to its visible position during photograph taking. They also checked that the cameras were correctly focused. Five photographs were taken, each capturing one side or the roof panel of the mosquito net (Fig. 2A). In order to take photographs of the roof panel, the frame was turned 90 degrees, positioning the roof panel to the side facing the camera. The photographs were sized 2448 × 2448 pixels and were taken with standard camera settings (model: SM-P613, focal length: 3 mm, flash mode: no flash, automatic white balance). Subsequently, the photographs were labelled with a unique identification number corresponding to the side of the particular mosquito net using Bulky Rename Utility software version 3.4.4.

Fig. 2

Full side photograph of the net hanging over a black cloth with a white grid (A) and a photograph of an individual hole showing both the hole and a known size reference ArUcO marker placed beside the hole (B)

Taking photographs for the ground truth method

After capturing images of the whole net, close-up photographs of each individual hole with a diameter exceeding 0.5 cm were taken with a camera standing ~ 30 cm away from the net. The standard camera settings described above were used. Each image included an ArUcO marker to serve as a calibration reference. ArUcO markers are easily detected by computer vision algorithms. They are high-contrast, square markers with a thick black border and an internal binary pattern encoding a unique identifier (ID), allowing for precise image correction techniques, including perspective correction. To accommodate various hole sizes, four different sized ArUcO markers were employed (i.e. 2 × 2 cm², 20 × 20 cm², 50 × 50 cm² and 98 × 98 cm²) with only one marker placed on the edge of the hole and included in the image (Fig. 2B). The sizes of ArUcO markers were selected because they correspond to the possible range of hole sizes observed in the damaged nets. Each ArUcO marker had a unique ID. This approach minimized the error by using markers closer in size to each hole, preventing the need to increase the camera distance to capture both the marker and the hole in the same image.

Photograph processing for image analysis

Hole segmentation from photographs was conducted using a graphical user interface (GUI) specifically developed for this study (Fig. 3). During segmentation, photographs of the mosquito nets were loaded into the GUI, and frame grid points were marked to indicate the corners of the mosquito net frame. Holes were first segmented using a thresholding function within the GUI, and then manually corrected using a “Draw” function present in the GUI (Fig. 3A). A thresholding function is a mathematical function with outputs which are values (0 or 1) based on whether an input value was above or below a threshold value. This converts a grayscale image into a mask containing the values 0 or 1. The thresholds can be manually adjusted in the GUI, and were chosen individually for every image. To find the real area of the holes in close-up shots, the size reference ArUcO marker was used (Fig. 3B).

Fig. 3

The graphical user interface (GUI) for hole labelling and segmentation. A Interface with one whole side of an ITN. B Interface with a close-up of a hole with ArUcO marker together with file status window. The GUI enables users to place landmarks to mark frame edges. The interface includes a threshold function to assist with manual segmentation by identifying pixels above a specified threshold value as hole areas. Users can then refine this segmentation using the "draw" and "erase" tools to label selected pixels as hole or background, respectively. Finally, the users can tag the image based on the photographed net side (short side, long side, or top side)

Hole surface area estimation for the ground truth close-up shots

The ArUcO corners were detected, and perspective correction was applied to the close-up shots. The known dimensions of the ArUcO markers were then used to compute the area per pixel for each image. The number of pixels was counted for each hole segmentation mask and multiplied by the area per pixel to obtain the total hole surface area for each close-up shot. A hole segmentation mask refers to an image of the same dimensions as its reference photograph, with pixels corresponding to the background class set to a value of 0 and pixels of the hole class set to 1.

Hole surface area estimation on the whole side images

From the images of the whole side of the net, as shown in Fig. 3A, first the corners of the frame were detected. Using these points, a perspective correction was applied to the image, and to find the area per pixel, the known dimensions of the frame were used to correctly scale the image. By multiplying the area per pixel by the total number of colored pixels in the segmentation mask, the total hole surface was computed for each hole per net.

Analysis to assess agreement

Data analysis was conducted using Stata version 16.1 (StataCorp LLC, College Station, Texas 77845 USA). Both the areas computed by the image analysis method as well as the WHO method were compared to the area computed based on the close-up shots as the reference method. The estimated surface areas of the holes in the net for the WHO method and the segmented whole side image method were plotted against the ground truth. The agreement between each method and the ground truth was estimated using the Bland–Altman method [35, 36]. The overall bias was estimated as the mean of the ratios of the estimated net hole area for each method divided by the ground truth hole area. The variability in the individual ratios was captured by the 95% limits of agreement: these are bounds between which 95% of the individual ratios were expected to lie. Finally, the sensitivity and specificity for determining nets as serviceable or too-torn, as defined according to the WHO method [8], were estimated using a range of values for the ground truth hole surface area.

Results

Description of the study nets

All seventy-five field-aged ITNs had holes on them identified by all three methods: image analysis, WHO manual hole counting, and the ground truth method (Table 1). According to the WHO method [8] of classification of holes, 53.5% of holes were classified as ‘size 1’, 41.0% as ‘size 2’, 2.7% as ‘size 3’, and 2.8% as ‘size 4’. Seventy-five percent of the nets were in serviceable condition with hole surface area less than 1000 cm² while 25% were classified as badly damaged, with a hole surface area of at least 1000 cm² per net.

Table 1 Detected number of holes and their surface area for the 75 ITNs using the WHO method, image analysis and the ground truth

Agreement between methods for the total net hole surface areas

The estimates of the total hole surface area differed for the three methods. Scatter plots of the estimated hole surface area against the ground truth indicated reasonable agreement between image analysis and ground truth (Fig. 4A) and lower values of the WHO hole counting method estimates in general compared to the ground truth (Fig. 4B).

Fig. 4

Scatter plots displaying the hole surface area per ITNs as measured by image analysis compared to the ground truth (A) and the WHO method compared to the ground truth (B) using the log scale

The bias and variability were estimated using the Bland–Altman method for assessing agreement. For the manual segmentation image analysis, the bias and variability in the ratio of image analysis to the ground truth was reasonably constant over the range of the ground truth values. The estimated overall bias was 0.70 indicating that the hole surface area estimated by image analysis was on average 0.70 times the hole surface area of the ground truth. The limits of agreement estimate that 95% of the individual ratios lay between 0.35 and 1.38 (Fig. 5A). In the comparison between the WHO method and the ground truth, the bias and variability were observed to vary with the ground truth. Using the Bland–Altman regression method [36], the estimated overall bias suggested a smaller estimated hole surface area with the WHO method compared to the ground truth, and the bias increased as the ground truth hole area increased (Fig. 5B).

Fig. 5

Bland–Altman-based plots showing agreement between the manually segmented images and the ground truth (A) and the WHO method and the ground truth (B). The solid lines (−) in the middle are the mean ratios of image analysis surface areas to the ground truth. The two dashed lines (–) (lower and upper) represent the 95% limits of agreement in which 95% of the ratios were expected to lie. The short dash line at a ratio of 1 represents perfect agreement

The cut-off of 1000 cm² is specific to the WHO method. It is likely that a different cut-off for deeming nets as serviceable or too torn would be needed in order for image analysis to achieve similar results to the WHO method. From our preliminary data, values between 2000 and 3000 cm² had over 80% sensitivity and specificity in classifying nets using the WHO method as the reference. However further work would be necessary to estimate this cut-off robustly.

Discussion

There was strong agreement between the hole area estimates from image analysis with manual segmentation and the ground truth for nets with a wide range of damage, from 10 cm2 to 10,000 cm². This suggests that images of the entire panel of ITNs provide sufficient information to estimate hole surface area and supports the feasibility of using automated hole detection from images in the future. In contrast, the estimated hole surface area using the WHO method was significantly lower than the ground truth. By assigning fixed sizes to broad hole categories, the WHO method tended to underestimate the hole area, particularly for the smallest and largest holes. However, since the WHO method classifies nets based solely on these estimates with a cut-off specific for the method, it remains effective for determining whether nets are serviceable or too torn, despite the potential inaccuracies in hole area estimation.

Image analysis could potentially allow faster, cheaper and more accurate estimates, and this study using manual segmentation shows that the images contain sufficient information as a proof-of-principle. Further work will enable automated image analysis and presentation of the results in the NMCP dashboard. It will also assess whether this leads to cost and time savings for routine surveys. Further testing with a larger sample size and a variety of image conditions such as sunlight dapples and different backgrounds will be carried out. The reproducibility of the method will also be assessed.

Other publications have explored the use of image analysis for assessing ITN damage [25, 27]. These observed that the WHO method may overestimate the hole surface area. These results contrast with this study findings. Three possible explanations are: (i) The majority of the nets which had been in use for one year in the study by Vanden Eng et al. [25], showed low levels of damage with many small holes, which have been documented to overestimate of hole area. This study found that the level of bias depended on the extent of net damage. (ii) The way that the hole surface area was calculated for the WHO method differed between studies. In Vanden Eng et al. the size 2 hole category, with diameters ranging between 2.0 cm to 10 cm corresponded to hole surface areas ranging from 3.1cm² to 78.5 cm² and holes were ellipse shaped [27], but the present study assumed that holes were circular with an average diameter of 6 cm as per WHO Guidelines [26]. (iii) Differences between the studies can also be attributed to the different image analysis algorithms used in this study compared to Vanden Eng et al. [25, 27] which used ImageJ software [29]. This underscores the importance of investigating the comparability of the image analysis methods.

There are several limitations of this study. One limitation is that there were relatively few nets with a ground truth total hole surface area between 1 and 100 cm². This might have affected the ability of the test methods in comparison to the ground truth when the hole surface area estimated by the ground truth is less than 100 cm² but not zero. The counting of large holes that were at the corner of the frame and thus appeared on two side panels of the net frame probably led to the underestimation of the hole surface area by the WHO method since they were only counted as one hole depending on which side the largest portion of the hole was observed, while using image analysis such holes were counted as a different hole in each panel. When the nets were draped over the metal frame, it is likely that some of the holes that were present at the hanging loop were obscured and not captured in the picture and hence not visible in the image analysis, leading to fewer holes being counted in the image analysis methods. In this study the use of black cloth helped to ensure all holes were clearly visible and the nets were hung carefully to reduce folding and wrinkles however this might not fully reflect field settings where the net material may have wrinkling or overlapping. In some cases, it was not clear whether to allow flaps of material caused by damage to hang down or be pinned up and this affects the estimated hole size. Different sized nets may have different tensions in the material when hung on the frame. It is possible that this may affect the image analysis estimates a little, but would not affect the ground truth since each hole was photographed separately with the net surface perfectly smooth.

Conclusion

Images coupled with manual segmentation contain sufficient information to accurately capture the surface area of holes in ITNs. This provides a basis for the next step of automating hole segmentation using deep learning models. This will potentially provide a fast, objective, and low-cost method for routinely assessing the physical durability of ITNs as part of post-market surveillance. While the WHO method underestimated the hole surface area, it remains useful in classifying nets as either serviceable or too torn because the cut-off is specific to this method.