Abstract.
Although light microscopy is the reference standard for diagnosing malaria, maintaining skills over time can be challenging. Between 2015 and 2017, the U.S. President’s Malaria Initiative–funded MalariaCare project supported outreach training and supportive supervision (OTSS) visits at 1,037 health facilities in seven African countries to improve performance in microscopy slide preparation, staining, and reading. During these visits, supervisors observed and provided feedback to health-care workers (HCWs) performing malaria microscopy using a 30-step checklist. Of the steps observed in facilities with at least three visits, the proportion of HCWs that performed each step correctly at baseline ranged from 63.2% to 94.2%. The change in the proportion of HCWs performing steps correctly by the third visit ranged from 16.7 to 23.6 percentage points (n = 916 observations). To assess the overall improvement, facility scores were calculated based on the steps performed correctly during each visit. The mean score at baseline was 85.7%, demonstrating a high level of performance before OTSS. Regression analysis predicted an improvement in facility scores of 3.6 percentage points (P < 0.001) after three visits across all countries. In reference-level facilities with consistently high performance on microscopy procedures and parasite detection, quality assurance (QA) mechanisms could prioritize more advanced skills, such as proficiency testing for parasite counting and species identification. However, in settings with high staff turnover and declining use of microscopy in favor of rapid diagnostic tests, additional supervision visits and/or additional QA measures may be required to improve and maintain performance.
INTRODUCTION
Prompt and accurate diagnosis is an essential component of malaria case management. Since 2010, the World Health Organization (WHO) has recommended parasite-based diagnostic testing by microscopy or rapid diagnostic test (RDT) for all patients suspected of having malaria before treatment with artemisinin-based combination therapy.1 Quality-assured light microscopy serves as the reference standard for diagnosing malaria and holds a number of advantages over RDTs that make it particularly useful in an inpatient or reference-level setting. Microscopists are able to visualize parasites on peripheral blood smears, allowing for species identification, quantification, tracking of response to treatment, and identification of other common causes of illness.2–6
Maintaining malaria microscopy skills over time can be challenging, and the variable and often poor quality of microscopy is well documented.2–4 Laboratory systems and infrastructure, as well as supply chain management, are often inadequate to ensure high-quality microscopy.5 With the expanded use of RDTs to diagnose malaria, laboratory staff have increasingly less opportunity to examine blood smears. High caseloads at busy facilities also pose a challenge, resulting in staff feeling pressured to provide clinicians with a reading in far less time than is recommended to examine a blood smear for parasites.
MalariaCare was a 5-year U.S. President’s Malaria Initiative–funded project that worked in 17 countries supporting national malaria control programs to strengthen capacity in malaria and febrile case management, including the diagnosis of malaria using light microscopy. Its approach for ensuring quality malaria microscopy included training laboratory staff in malaria microscopy, establishing national assessment competency programs, and supporting microscopists for international accreditation and laboratory supportive supervision, all of which were components of a malaria microscopy quality assurance (QA) system adopted as recommended by the Global Malaria Program at WHO.1
Studies have demonstrated that supportive supervision can improve laboratory staff’s competency in parasite detection in pilot or randomized control trial settings.6,7 However, equally important steps in the process are the preparation and staining of slides. During supportive supervision visits conducted by MalariaCare (henceforth referred to as Outreach Training and Supportive Supervision [OTSS]), trained supervisors used a standardized checklist to observe laboratory staff performing these steps, provided individualized feedback on steps carried out correctly and incorrectly, and developed action plans with staff to address any gaps in microscopy competencies and broader laboratory diagnostics issues.8 Using MalariaCare program data, we evaluated the impact of OTSS on laboratory staff competencies.
METHODS
Program setting and population.
Of the 15 countries in sub-Saharan Africa that were supported by the MalariaCare project, nine conducted OTSS for microscopy. Within each country, ministries of health selected both public and private health facilities with laboratories that performed malaria microscopy, as well as stand-alone laboratories (Democratic Republic of the Congo [DRC] only) within certain regions or provinces that were mutually agreed on between the ministry and United States Agency for International Development (USAID) mission for OTSS visits. Within facilities, laboratory staff who were responsible for conducting any malaria-related slide preparation, staining, and reading tasks were eligible for observation and feedback.
Program description.
Between 2015 and 2017, trained ministry of health laboratory supervisors observed eligible staff performing malaria microscopy during routine OTSS visits. Some facilities received visits before 2015, but with a different checklist. Before visits, supervisors received a minimum of 3 days of training in supervision skills and use of the checklist.
Laboratory supervisors conducted up to three observations of staff preparing, staining, and reading microscopy thick smears, depending on staff availability, time, and the number of malaria microscopy tests ordered during the visit. Observed staff were evaluated using a checklist that included 20–23 key steps of 30 steps total in conducting malaria microscopy, depending on whether the staff used venipuncture or a finger prick to collect blood, and Giemsa or Field stain. The steps included in the checklist were adapted from WHO guidelines and modified by MalariaCare’s technical team, including both headquarters and field-based technical staff (Table 1). National malaria control programs from each country reviewed and approved the checklist for use during OTSS. Each item on the checklist was phrased as a question, and observers were prompted to check “yes” or “no” as to whether the health worker being observed correctly performed the step. Five of the 20–23 items were considered “minimum standard” steps (i.e., they are the most essential for proper performance of microscopy testing); these are highlighted in bold font in Table 1. The score for each observation was weighted so that slide preparation, staining, and supervisor/staff agreement on slide positivity each counted for one-third of the score. Within the slide preparation and staining categories, the minimum standard steps counted for two-thirds of the sub-score. The checklist also captured certain aspects of the laboratory outside of the observation, such as the presence of malaria microscopy reference materials and persistent stock-outs of supplies required for microscopy. The amount of time between visits for each health facility depended on a number of factors including ministry of health schedules and project budgets, but typically was 3–6 months in length.
Table 1.
Steps in malaria microscopy performance evaluated on outreach training and supportive supervision checklist
| Procedure* | Score |
|---|---|
| Identifies patient and records patient information in register | ☐Yes ☐No |
| Wears gloves (count as yes if not available) | ☐Yes ☐No |
| Cleans slide | ☐Yes ☐No |
| Labels slide with date and patient’s name and/or number | ☐Yes ☐No |
| Used finger prick to collect blood | |
| Cleans finger with alcohol and allows to dry | ☐Yes ☐No |
| Pricks finger, wipes off the first drop of blood, and places subsequent drop on slide w/o touching finger | ☐Yes ☐No |
| Used venipuncture to collect blood | ☐Yes ☐No |
| Labels ethylenediaminetetraacetic acid collection tube with date and patient’s name and/or number | ☐Yes ☐No |
| Cleans puncture site with alcohol and allows to dry | ☐Yes ☐No |
| Assembles sterile needle and syringe or vacutainer | ☐Yes ☐No |
| Collects and gently mixes venous blood sample successfully | ☐Yes ☐No |
| Applies blood from only one patient to each thick film slide | ☐Yes ☐No |
| Spreads thick film into 1–2 cm diameter circle; can read print placed under the slide | ☐Yes ☐No |
| Air-dries thick film slide before staining | ☐Yes ☐No |
| Filters stain before use | ☐Yes ☐No |
| Used Giemsa stain | |
| Uses standard 10% Giemsa solution† | ☐Yes ☐No |
| Prepares fresh Giemsa staining solution properly or on the day of observation | ☐Yes ☐No |
| pH of the staining solution between 7.2 and 7.4 (count as yes if unable to check) | ☐Yes ☐No |
| Immerses thick film slide in 10% Giemsa stain for 10–15 minutes | ☐Yes ☐No |
| Rinses thick film slide carefully with water | ☐Yes ☐No |
| Used Field stain | |
| Prepares fresh Field stain A solution (methylene blue) properly or on the day of observation | ☐Yes ☐No |
| Prepares fresh Field stain B solution (eosin) properly or on the day of observation | ☐Yes ☐No |
| Immerses slide in stain A for 3 seconds | ☐Yes ☐No |
| Rinses slide carefully with water after using stain A | ☐Yes ☐No |
| Immerses slide in stain B for 5 seconds | ☐Yes ☐No |
| Rinses slide carefully with water after using stain B | ☐Yes ☐No |
| Drains and air-dries slide (avoiding application of external heat) | ☐Yes ☐No |
| Segregates and safely disposes off sharp waste in a safety box | ☐Yes ☐No |
| Disposes off infectious waste in appropriate waste containers | ☐Yes ☐No |
| Washes off/disinfects liquid waste appropriately | ☐Yes ☐No |
| Supervisor agrees with staff on slide result | ☐Yes ☐No |
* Checklist steps in bold (“minimum standard” steps) are considered more important and are collectively weighted twice as much as the other steps when calculating scores.
† Health facilities preferred to use the 10% Giemsa solution because of its rapid turnaround when compared with the 3% Giemsa solution.
Analysis of implementation data.
Results gathered during observation of on-duty health-care workers (HCWs) performing malaria microscopy on the day of an OTSS visit were either captured on a paper-based checklist and subsequently entered into a Microsoft Access database or entered directly by supervisors into MalariaCare’s Electronic Data System (EDS), a system using District Health Information System version 2 (DHIS2) (Oslo, Norway) software to store and analyze data.9 Data from both Access and EDS databases were imported into Stata 14 (StataCorp, 2015. Stata Statistical Software: Release 14.1. College Station, TX: StataCorp LP) for data cleaning and analysis.
Of the nine countries visited for OTSS, seven were included in the analysis: DRC, Kenya, Malawi, Mali, Mozambique, Tanzania, and Zambia. One country, Ghana, was excluded because the national laboratory authorities chose to use a different checklist; and another country, Madagascar, was excluded because intervention was limited to one city, Antananarivo, and thus too dissimilar to the other countries in the analysis. In these countries, any health facility that received at least two MalariaCare-supported OTSS visits between 2015 and 2017 and had complete data collected using a standardized health facility checklist was included. We analyzed results at two levels: 1) at the observation level for individual HCW performance on individual checklist steps and 2) at the facility level for overall facility improvement on malaria microscopy competencies over time. We reported performance on each of the individual 23 steps in the checklist as the proportion of HCWs observed who performed the step correctly during the first, second, and third visits. We included observations with no steps missing, and facilities with at least three visits and one complete observation at each time point. If a HCW was observed more than once during a particular visit, we included only the results for the HCW’s first observation.
To estimate the impact of OTSS on facility microscopy performance by individual visit and over time, we calculated health facility scores for the first, second, and third visits, using the updated, standardized checklist, by averaging up to three scores of the complete observations included in the individual step analysis at each visit. We reported descriptive statistics for the average facility performance by country and visit number for the subset of facilities with scores for all three visits to show trends among facilities consistently visited over time. In addition, we conducted a multilevel mixed-effects linear regression (clustered at the health facility level) to estimate the independent effects of OTSS and other health facility characteristics that could potentially affect scores; in this analysis, any complete observation occurring at a facility with at least two visits was included. Health facility characteristics collected as part of the broader OTSS checklist and included as covariates in the adjusted regression were as follows: whether the facility received a previous OTSS visit before the first visit used in the analysis; whether the laboratory had at least one staff that received formal training in malaria microscopy; whether the facility had no stock-outs of slides, stains, or lancets lasting more than 7 days in the past 3 months; whether the laboratory had a functional internal QA (IQA) program; whether microscopy guidelines, standard operating procedures (SOPs), and bench aids were available in the laboratory; whether the facility was a hospital; and country.
RESULTS
A total of 1,037 health facilities performing malaria microscopy received at least two OTSS visits, with 475 receiving two visits, 377 receiving three visits, and 185 receiving four visits or more. Of the 1,037 health facilities visited, 779 had a complete observation for any visit. Earlier versions of the checklist did not ask the supervisor why observations were not complete. Of the later version of the checklist that did ask a reason for not completing an observation (n = 679 observations), 23% reported that no workers who performed malaria microscopy were present at the time of the visit; 12% reported a microscopy supply stock-out; 4% reported that no malaria microscopy test was ordered; and 4% reported a power outage. The rest (56%) did not give a reason, perhaps because they skipped the question or because they thought they had completed the checklist.
Observation performance on individual steps.
Of the 562 health facilities visited at least three times, 191 had at least one observation completed during each of the first three visits. A total of 916 observations across these 191 health facilities were analyzed (Table 2). Table 3 reports the percentage of HCWs observed at these facilities who performed each checklist step at each visit correctly.
Table 2.
Number of facilities and observations eligible for analysis on individual steps and facility performance, and numbers with complete observations
| Observations | Facilities | |
|---|---|---|
| Number eligible* | 2,425 | 562 |
| Number included in analysis† | 916 | 191 |
| Percentage of total | 37% | 34% |
* Observations occurred at facilities that received at least three outreach training and supportive supervision visits.
† Observations were complete, and facilities had at least one complete observation at each of the first three visits.
Table 3.
Proportion of health workers who performed microscopy checklist steps correctly: first, second, and third visits; and change in score (n = 916)
| Microscopy observation checklist step | Visit number | Percentage point change in score | ||||
|---|---|---|---|---|---|---|
| First | Second | Third | First to second | Second to third | First to third | |
| Number of observations | 293 | 300 | 323 | – | – | – |
| Identifies patient and records patient information in register | 92.2% | 95.3% | 92.0% | +3.1% | −3.3% | −0.2% |
| Wears gloves (count as yes if not available) | 91.8% | 96.7% | 94.1% | +4.9% | −2.6% | +2.3% |
| Cleans slide | 77.5% | 77.7% | 79.3% | +0.2% | +1.6% | +1.8% |
| Labels slide with date and patient’s name and/or number | 79.5% | 83.0% | 80.5% | +3.5% | −2.5% | +1.0% |
| Used finger prick to collect blood: N | 234 | 230 | 260 | – | – | – |
| Cleans finger with alcohol and allows to dry | 90.6% | 92.2% | 89.3% | +1.6% | −2.9% | −1.3% |
| Pricks finger, wipes off first drop of blood, and places subsequent drop on slide w/o touching finger | 66.9% | 70.4% | 75.9% | +3.5% | +5.5% | +9.0% |
| Used venipuncture to collect blood: N | 59 | 70 | 63 | – | – | – |
| Labels ethylenediaminetetraacetic acid collection tube with date and patient’s name and/or number | 65.3% | 85.7% | 78.0% | +20.4% | −7.7% | +12.7% |
| Cleans puncture site with alcohol and allows to dry | 72.6% | 94.0% | 85.7% | +21.4% | −8.3% | +13.1% |
| Assembles sterile needle and syringe or vacutainer | 63.2% | 95.2% | 86.8% | +32.0% | −8.4% | +23.6% |
| Collects and gently mixes venous blood sample successfully | 71.6% | 95.2% | 84.8% | +23.6% | −10.4% | +13.2% |
| Applies blood from only one patient to each thick film slide | 91.1% | 95.7% | 94.7% | +4.6% | −1.0% | +3.6% |
| Spreads thick film into 1–2 cm diameter circle; can read print placed under the slide | 73.7% | 72.0% | 82.7% | −1.7% | +10.7% | +9.0% |
| Air-dries thick film slide before staining | 88.1% | 90.7% | 91.6% | +2.6% | +0.9% | +3.5% |
| Filters stain before use | 62.8% | 78.0% | 76.2% | +15.2% | −1.8% | +13.4% |
| Used Giemsa stain: N | 244 | 237 | 259 | – | – | – |
| Uses standard 10% Giemsa solution | 89.8% | 96.2% | 96.5% | +6.4% | +0.3% | +6.7% |
| Prepares fresh Giemsa staining solution properly or on day of observation | 86.4% | 94.1% | 93.4% | +7.7% | −0.7% | +7.0% |
| pH of the staining solution between 7.2 and 7.4 (count as yes if unable to check) | 91.8% | 96.6% | 87.3% | +4.8% | −9.3% | −4.5% |
| Immerses thick film slide in 10% Giemsa stain for 10–15 minutes | 79.9% | 90.3% | 96.9% | +10.4% | +6.6% | +17.0% |
| Rinses thick film slide carefully with water | 84.8% | 96.2% | 96.9% | +11.4% | +0.7% | +12.1% |
| Used Field stain: N | 49 | 63 | 64 | – | – | – |
| Prepares fresh Field stain A solution (methylene blue) properly or on the day of observation | 72.1% | 51.6% | 70.1% | −20.5% | +18.5% | −2.0% |
| Prepares fresh Field stain B solution (eosin) properly or on the day of observation | 72.1% | 51.6% | 73.1% | −20.5% | +21.5% | +1.0% |
| Immerses slide in stain A for 3 seconds | 77.6% | 59.4% | 61.2% | −18.2% | +1.8% | −16.4% |
| Rinses slide carefully with water after using stain A | 85.1% | 100.0% | 97.0% | +14.9% | −3.0% | +11.9% |
| Immerses slide in stain B for 5 seconds | 79.1% | 57.8% | 64.2% | −21.3% | +6.4% | −14.9% |
| Rinses slide carefully with water after using stain B | 83.6% | 100.0% | 98.5% | +16.4% | −1.5% | +14.9% |
| Drains and air-dries slide (avoiding application of external heat) | 75.4% | 78.3% | 86.1% | +2.9% | +7.8% | +10.7% |
| Segregates and safely disposes off sharps waste in a safety box | 94.2% | 97.0% | 95.4% | +2.8% | −1.6% | +1.2% |
| Disposes off infectious waste in appropriate waste containers | 88.1% | 95.0% | 90.1% | +6.9% | −4.9% | +2.0% |
| Washes off/disinfects liquid waste appropriately | 89.8% | 90.7% | 89.2% | +0.9% | −1.5% | −0.6% |
| Supervisor agrees with staff on slide positivity | 91.8% | 93.7% | 92.3% | +1.9% | −1.4% | +0.5% |
* Checklist steps in bold (“minimum standard” steps) are considered more important and are collectively weighted twice as much as the other steps when calculating scores.
The majority of the HCWs observed collected blood using the finger prick method rather than venipuncture (79% versus 21%) and Giemsa stain rather than Field stain (81% versus 19%). For 20 of the 30 steps in the checklist, the proportion of HCWs who performed each step correctly improved between the first and the third visit. The most commonly missed steps across all observations—filtering the stain before use, spreading the thick film to an appropriate diameter and thickness, and draining and air-drying the slide appropriately—improved by 9.0–13.4 percentage points.
However, compliance with the checklist fluctuated over the three visits, with 15 steps improving by more than one percentage point between the first and second visit, and then declining by more than a percentage point between the second and third. Fluctuations were particularly large for the subset of observations that used venipuncture to collect blood (n = 192): all four steps in this group improved by more than 20 percentage points from the first to the second visit and declined by more than seven percentage points from the second to the third visit. For the subset of observations that used Field stain (n = 176), the rate of compliance with preparation of Field stain and immersion of the slide for the recommended time actually decreased by more than 18 percentage points between the first and second visit and improved to varying degrees (6.4–21.8 percentage points) from the second to third visit. Of all Field stain observations, 89% came from one country (country 3).
Facility performance.
The 916 observations for the 191 health facilities that had a complete score for the first three visits were averaged to produce a facility score for each visit. Figure 1 reports the average facility score and 95 percent confidence intervals (CIs) at each visit by country. Among these facilities, average microscopy scores improved from 87% (95% CI: 85–89%) during the first visit to 90% (95% CI: 88–92%) during the last. The proportion of facilities with scores for every visit differed widely among countries, from 13% for country 4 to 72% for country 1. For country 4, only four facilities had scores for the first visit; supervisors commonly did not complete the checklist because of a formatting issue with the paper copies.
Figure 1.
Average facility performance on microscopy performance over consecutive rounds of outreach training and supportive supervision, by country.
Regression analysis.
Of the 2,657 OTSS visits conducted, 1,497 OTSS visits (56%), covering 779 health facilities, had at least one complete observation and were included in the unadjusted regression model; 1,296 OTSS visits (49%), representing 725 health facilities, had complete observation scores and data on all covariates and were included in the adjusted regression model (Table 4).
Table 4.
Number of visits and health facilities eligible for, and included in, regression analyses
| Visits | Facilities | |
|---|---|---|
| Number eligible* | 2,657 | 1,037 |
| Number included in unadjusted regression† (% of eligible) | 1,491 (56%) | 779 (79%) |
| Number included in adjusted regression‡ (% of eligible) | 1,296 (49%) | 725 (70%) |
* Visits that occurred at facilities that received at least two outreach training and supportive supervision visits.
† Visits had at least one complete observation.
‡ Visits had at least one complete observation and had data for all covariates included in the adjusted regression.
Table 5 presents the facility characteristics of the visit sample included in the adjusted regression model. In addition, it also presents the mean microscopy facility score for each covariate. Scores at the 25th and 75th percentile are also given, indicating the score at which the bottom 25% of health facilities scored at or below, and the score at which the top 25% scored at or above, respectively. Of the 1,296 visits included in the adjusted regression, 30.7% were a first visit, 42.8% were a second visit, and 26.5% were a third visit. Mean scores ranged from 85.7% (among facilities during the first visit) to 91.6% (among facilities during the third visit). The majority of visits occurred at facilities that did not have any previous OTSS visit (81.5%), had a functioning microscope and no stock-outs of supplies (slides, stains, or lancets) lasting more than 7 days in the past 3 months (61.3%), and had microscopy SOPs (71.9%) and bench aids (69.1%). Roughly half of the visits occurred at laboratories with at least one staff member formally trained in malaria microscopy (52.2%), with IQA programs for microscopy (50.6%), and with the most recent microscopy guidelines for the country (52.2%). In addition, 49.7% of the visits were conducted in country 1.
Table 5.
Health facility characteristics based on OTSS visits included in regression analysis, mean facility microscopy scores, and scores at the 25th and 75th percentile, by facility characteristic (N = 1,296)
| Characteristic | Visits | Microscopy score | |||
|---|---|---|---|---|---|
| % | N | Mean | 25th percentile | 75th percentile | |
| Visit number | |||||
| 1 | 30.7 | 398 | 85.7 | 80.9 | 97.4 |
| 2 | 42.8 | 555 | 87.9 | 83.1 | 98.8 |
| 3 | 26.5 | 343 | 91.6 | 87.5 | 99.4 |
| Across all visits: | |||||
| Facility had prior OTSS visits | |||||
| No | 81.5 | 1,056 | 89.6 | 85.0 | 98.8 |
| Yes | 18.5 | 240 | 82.1 | 72.5 | 97.2 |
| Facility has at least one worker formally trained in malaria microscopy in the past 2 years | |||||
| No | 49.4 | 640 | 86.9 | 82.4 | 98.6 |
| Yes | 50.6 | 656 | 89.4 | 84.0 | 98.8 |
| Facility has no stock-outs of slides/stain/lancets | |||||
| No | 38.7 | 501 | 86.5 | 80.7 | 98.6 |
| Yes | 61.3 | 795 | 89.2 | 85.0 | 98.8 |
| Laboratory has internal quality assurance program | |||||
| No | 49.4 | 640 | 84.8 | 78.6 | 97.4 |
| Yes | 50.6 | 656 | 91.5 | 87.5 | 100.0 |
| Facility has latest microscopy guidelines | |||||
| No | 47.8 | 620 | 85.7 | 80.5 | 98.6 |
| Yes | 52.2 | 676 | 90.4 | 86.1 | 99.3 |
| Facility has microscopy standard operating procedures | |||||
| No | 28.1 | 364 | 82.8 | 75.0 | 96.4 |
| Yes | 71.9 | 932 | 90.3 | 86.1 | 99.3 |
| Facility has microscopy bench aids | |||||
| No | 30.9 | 400 | 84.4 | 77.1 | 97.4 |
| Yes | 69.1 | 896 | 89.9 | 85.1 | 98.8 |
| Facility is a hospital | |||||
| No | 58.1 | 753 | 87.8 | 82.8 | 98.6 |
| Yes | 41.9 | 543 | 88.7 | 83.8 | 98.8 |
| Country | |||||
| Country 1 | 49.7 | 644 | 88.7 | 83.6 | 98.6 |
| Country 2 | 7.3 | 95 | 86.3 | 83.5 | 98.6 |
| Country 3 | 14.5 | 188 | 79.1 | 71.1 | 94.0 |
| Country 4 | 5.1 | 66 | 90.8 | 86.3 | 97.4 |
| Country 5 | 6.7 | 87 | 92.6 | 87.7 | 100.0 |
| Country 6 | 13.0 | 168 | 92.9 | 92.2 | 100.0 |
| Country 7 | 3.7 | 48 | 92.6 | 89.5 | 99.1 |
OTSS = outreach training and supportive supervision. This table indicates the percentage and mean scores of all visits included in the regression.
In the unadjusted regression, a facility’s microscopy score was estimated as 2.1 percentage points higher during the second OTSS visit (P = 0.01) and 6.2 percentage points higher during the third OTSS visit (P < 0.001) when compared with the first. After adjusting for facility characteristics, a facility’s microscopy score was an estimated 0.7 percentage points higher during the second OTSS visit (P = 0.45) and 3.6 percentage points higher during the third OTSS visit (P < 0.001) when compared with the first (Table 6). Thus, the predicted improvement from the first to the second visit was not found to be statistically significant at the 5% level, but was statistically significant between the first and the third visit.
Table 6.
Regression results for health facility characteristics associated with percentage point improvement in scores: all countries, excluding country 3
| Characteristic | All countries | Excluding country 3 | ||
|---|---|---|---|---|
| Coefficient | 95% CI | Coefficient | 95% CI | |
| Visit number (ref: 1 visit) | ||||
| 2 | 0.7 | [−1.1, 2.4] | 3.4 | [1.6, 5.1] |
| 3 | 3.6 | [1.5, 5.6] | 5.3 | [3.1, 7.4] |
| Had prior outreach training and supportive supervision visits | −5.4 | [−10.0, −0.7] | −6.9 | [−13.0, −0.7] |
| Has at least one worker formally trained in malaria microscopy | 0.2 | [−1.5, 1.8] | 0.3 | [−1.4, 2.0] |
| Has no stock-outs of slides/stain/lancets | 1.2 | [−0.4, 2.8] | 1.3 | [−0.3, 2.9] |
| Has internal quality assurance program | 4.8 | [3.2, 6.5] | 4.9 | [3.2, 6.6] |
| Has latest microscopy guidelines | 0.3 | [−1.4, 2.0] | 0 | [−1.7, 1.7] |
| Has microscopy standard operating procedures | 4 | [2.0, 6.0] | 3.9 | [1.8, 5.9] |
| Has microscopy bench aids | 2.3 | [0.5, 4.1] | 2.4 | [0.6, 4.2] |
| Is a hospital | −0.4 | [−2.3, 1.5] | −0.5 | [−2.5, 1.5] |
| Constant | 80.7 | [78.7, 82.7] | 79.4 | [77.4, 81.3] |
| Observations | 1,296 | – | 1,108 | – |
CI = confidence interval. Note: Regressions included a control variable for the country and are not reported here.
Because it was observed that one country (country 3) had a relatively large number of facilities with a significant decline between the first and the second visit, as well as lower predicted scores than the rest of the countries, another regression excluding this country was run to assess the sensitivity of the regression model. Field staff attributed these lower scores to the increased use of RDTs in recent years; however, we were unable to verify this as we did not have access to national records on microscopy and RDT frequency. In addition, they felt that the decrease between the first and the second visit was attributed to a programmatic shift, where supervisors assessed laboratory staff in their own district during the first visit and an adjacent district in the following visits. When country 3 was excluded, the estimated improvement in microscopy score during the second visit was 3.4 percentage points (P < 0.001) and 5.3 percentage points during the third visit (P < 0.001), as compared with the first visit. The 1.9 percentage point estimated improvement between the second and third visit was statistically significant at the 5% level (P = 0.044).
Aside from OTSS visits, the factor associated with the largest improvement in microscopy performance was having an IQA program for microscopy (4.8–4.9 percentage points, P < 0.001). Having a prior OTSS visit with a different checklist was negatively associated with microscopy scores (P = 0.024–0.029). Having microscopy SOPs and bench aids were associated with a 3.9–4.0 percentage point (P < 0.001) and a 2.3–2.4 percentage point (P = 0.011) improvement in microscopy scores, respectively. All other factors included in the model were not found to be statistically significant at the 5% level.
DISCUSSION
We used programmatic data to assess the effect of OTSS on health facility performance on microscopic diagnosis of malaria in several sub-Saharan African settings. Our analysis indicates that OTSS results in improvements on the steps required in adequately preparing and accurately reading a blood slide for malaria diagnosis, and modestly improves facility performance within two to three visits. The limitation of this analysis is that we worked with data that were collected during program implementation, a proportion of which were incomplete and excluded from the analysis. Within this context, the data did not allow for a randomized selection of facilities.
Among facilities visited three times, improvements were witnessed for most of the steps recorded in the checklist after the first visit. This was likely due to the one-on-one mentoring targeting individual providers and the areas of weakness observed, and to the general feedback that OTSS supervisors provided to laboratory staff at the end of the visit, leading to the development of a negotiated action plan with laboratory staff and facility managers to address key issues. However, a closer look at the progress per individual step at each visit shows fluctuations for several of these steps. This underscores the complexity of malaria microscopy procedures, composed of several steps which may sometimes be performed by different individuals, and the difficulty of consistently following each step and each individual every time. In particular, performance for venipuncture improved during the second visit, and then declined significantly during the third visit. Finger-prick–collected blood is preferable for detecting low-density parasitemia, but venipuncture blood may have been used among a portion of HCWs because they were collecting blood for conducting other tests. At all times, providers should be trained and encouraged to use a finger prick for malaria testing. Sustained improvement in venipuncture techniques, a less common method of collecting blood, may require more OTSS visits than other steps or a review of malaria microscopy reference materials to ensure that they include a sufficient description of venipuncture. Significant performance fluctuations in Field stain steps were also found to occur over the course of three OTSS visits. However, because most of the observations of Field staining were performed in country 3 (where performance on all other steps fluctuated significantly as well), the results may be more indicative of a country-specific issue rather than the ability of supervisors to provide mentorship with regard to Field staining.
Supervisor agreement on whether a slide was positive for malaria parasites across all seven countries was high. This must be interpreted with some caution, given that this indicator largely depends on the supervisor’s level of expertise compared with that of the observed provider. Although the project attempted to only permit highly skilled laboratory technicians to become supervisors, this was not always possible because of the scarcity of accredited expert laboratory technicians in the countries.8
Among health facilities receiving three visits and with at least one complete observation for each visit, average microscopy performance improved between the first and the third visit for most of the countries supported, albeit modestly. Of the three countries that witnessed a decline, one of the countries (country 4) had only four facilities with sufficient data to track progress over three visits. Among health facilities in this country with scores for the second and the third visit (n = 31), average scores increased slightly from 92% to 94%. For the other two countries, program staff cited a general shift from microscopy to RDTs for malaria testing as a possible factor in diminishing skills, even though we were not able to independently verify this assertion.
Other possible reasons for decreased performance could include laboratory staff not using their skills frequently enough because of lower caseloads, complacency in mentoring during OTSS as supervisors get more acquainted with HCWs, or staff turnover. In a survey of laboratory technicians in seven sub-Saharan countries, it was found that more than half had switched jobs at least once in the 5 years prior.10 Although MalariaCare only started collecting data on staff turnover in the last year of the project, HCWs observed for microscopy reported lower rates of previous OTSS mentorship than those observed for clinical or RDT performance. Additional research linking staff turnover and other factors with performance, as well as studying performance over a longer period of time and including more facilities, may be needed.
Although the proportion of facilities with three visits that had complete data was low, the regression analysis—which included most of the laboratories visited in the OTSS program—did support a positive effect of OTSS between the first and the third visit even with the inclusion of country 3 (which saw the greatest decline), and a statistically significant increase between the first and the second visit when that country was removed from analysis. With these statistically significant results, it is unclear why receiving an OTSS visit before the analysis period was negatively associated with microscopy scores’ improvement. It is possible that the minority of facilities with prior OTSS visits (representing only 18.5% of the sample and three countries) were, by chance, less amenable to improvement through OTSS because of the previously mentioned factors not captured by the checklist and thus not included in the regression. Thus, the estimated effect of a prior OTSS visit may have suffered from omitted variable bias. In our sample, 90% of facilities with a prior visit were in countries that declined in performance. Another possibility is that since the introduction of the new checklist, training of supervisors put additional emphasis on mentorship. Thus, supervisors may have been more likely to provide feedback on individual missed steps in later rounds. It must also be noted that as shown in the regression analysis, OTSS could be most effective in improving performance when associated with a strong IQA program. Hence, one important function of OTSS should be strengthening the IQA mechanisms within health facilities to sustain the positive effects of OTSS.
Unlike RDT performance (discussed by Eliades et al.11), ensuring that HCWs follow proper procedures for malaria microscopy at low-performing laboratories, and in areas where malaria microscopy is performed with decreasing frequency, is likely to require a more concerted effort over a longer period of time. At the same time and as mentioned before, average microscopy scores were high for several countries at baseline, indicating that supportive supervision that includes extensive observation of HCWs conducting malaria microscopy on a routine basis may not be necessary for all health facilities, and a more targeted approach based on specific needs could result in greater efficiencies. However, our analysis of program implementation data did not allow for a comparison of results with similar facilities receiving no supportive supervision. Given the high level of performance, even at baseline, it is possible that facilities that did not receive any OTSS would have presented a greater decrease in performance and/or a decrease in performance among facilities across all countries. Further operations research should be conducted to determine the optimal frequency of OTSS visits based on health facility proficiency levels. Although quantification of malaria parasites is important for identifying and managing severe malaria cases, reading slides for quantification and species identification was not included in the microscopy score. During OTSS, we found that these two skills are often not performed, similar to findings in other studies.3,5 Because quantification and speciation of malaria parasites are competencies that are difficult to maintain over time, at lower level facilities, resources could focus on parasite detection, whereas quantification and speciation could be emphasized at reference-level facilities through additional activities such as proficiency testing. We recommend that these areas be included in scoring when evaluating malaria microscopy skills at higher level facilities in the future.
Among facilities that received three visits, only one-third had scores for each visit. As a rough comparison, a similar analysis of OTSS and RDT performance found that complete RDT observations over three sequential OTSS visits were available for 67% of health facilities visited (n = 2,994).11 Although incomplete data may have partially been due to supervisor adherence to filling out the checklist or issues with data entry in the database, investigation of recent checklist data indicates that many laboratories also continue to suffer from gaps in the supplies and equipment maintenance required to perform malaria microscopy. Thus, the results presented here should be interpreted with some caution and are not meant to represent the performance of all health facilities receiving OTSS in countries supported by MalariaCare.
Moreover, the lack of supplies and equipment diminishes the opportunity for mentorship, and thus should be addressed before, or in collaboration with, the implementation of supportive supervision. Although the regression did not find that having a sustained stock-out had a statistically significant effect on microscopy scores, the regression was not able to measure the long-term effects of stock-outs because the sample only included those facilities where supervisors were able to conduct observations.
Our analysis also shows a positive correlation between the availability of SOPs and bench aids and improvement in malaria microscopy performance. However, because the materials were not randomly distributed to facilities, it is unclear whether the estimated improved performance is due to the mere presence of these reference materials, or simply signifies a better-resourced and organized laboratory. The difference in performance between hospital and non-hospital facilities was not statistically significant. It is possible that some health centers—particularly those that have laboratories and perform microscopy—effectively act as the reference facility in rural areas and skill levels do not differ significantly from those of hospital personnel. Alternatively, busy hospital laboratory staff may encounter greater pressure to prepare and stain slides quickly. Regardless, our analysis indicates that both hospitals and smaller health facilities benefit from mentorship during OTSS.
The covariate most strongly associated with improved performance between OTSS visits was the facility having an IQA mechanism in place. The WHO defines laboratory IQA as “the daily control and monitoring of each stage of testing by laboratory personnel to ensure that all tests are performed accurately and precisely.” Recommended steps include daily checks of the quality and performance of the stain, weekly slide rechecking, evaluation of the performance of equipment and quality of supplies, and accurate recording in the register.1 The positive association of having an IQA mechanism and microscopy competencies shown in this analysis suggests that programs should further investigate the potential for supporting the implementation of IQA measures on a broad scale. The appropriate balance of IQA and external QA measures (such as OTSS) may depend on the individual context, but programs should strongly consider strengthening both to build robust microscopy competencies in a cost-effective manner.
While demonstrating that OTSS was able to modestly improve microscopy performance overall, we found that three countries still declined in performance over time. We recommend that further research be conducted to explore the relationship between declining microscopy use and microscopy skill maintenance. If skills decrease with fewer tests, as we suspect, then to maintain microscopy quality, programs may consider restricting microscopy use to higher level facilities, where adequate volumes of testing can be conducted, and providing more intensive support to these fewer facilities through both OTSS and IQA. Facilities with low frequency of microscopy could be transitioned to only using RDTs for malaria diagnosis, which are easier to use and easier to maintain high quality in malaria diagnosis.11 Further research should then be performed to determine whether such an approach is effective in maintaining accurate malaria diagnosis.
CONCLUSION
Data collected during OTSS indicate that supportive supervision improves the performance on the preparation, staining, and detection of malaria parasites in several contexts. In settings with high laboratory staff turnover and declining use of malaria microscopy in favor of RDTs, maintaining microscopy skills in certain laboratories and country contexts may require a more sustained and rigorous QA mechanism than is needed to maintain diagnostics skills using malaria RDTs. Program managers can learn from this analysis and further refine their QA systems to effect change.
Acknowledgments:
We acknowledge the contributions of the government and project staff in all seven countries who were the principal actors implementing the project and collecting the data.
REFERENCES
- 1.World Health Organization , 2016. Malaria Microscopy Quality Assurance Manual–Ver. 2. Available at: http://www.who.int/malaria/publications/atoz/9789241549394/en/. Accessed January 9, 2018.
- 2.Reyburn H, Mbakilwa H, Mwangi R, Mwerinde O, Olomi R, Drakeley C, Whitty CJM, 2007. Rapid diagnostic tests compared with malaria microscopy for guiding outpatient treatment of febrile illness in Tanzania: randomised trial. BMJ 334: 403. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Zurovac D, Midia B, Ochola SA, English M, Snow RW, 2006. Microscopy and outpatient malaria case management among older children and adults in Kenya. Trop Med Int Health 11: 432–440. [DOI] [PubMed] [Google Scholar]
- 4.Abreha T, et al. 2014. Malaria diagnostic capacity in health facilities in Ethiopia. Malar J 13: 292. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Petti CA, Polage CR, Quinn TC, Ronald AR, Sande MA, 2006. Laboratory medicine in Africa: a barrier to effective health care. Clin Infect Dis 42: 377–382. [DOI] [PubMed] [Google Scholar]
- 6.Odhiambo F, et al. 2017. Factors associated with malaria microscopy diagnostic performance following a pilot quality-assurance programme in health facilities in malaria low-transmission areas of Kenya, 2014. Malar J 16: 1–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Burnett SM, Mbonye MK, Martin R, Ronald A, Zawedde-Muyanja S, Willis KS, Colebunders R, Manabe YC, Weaver MR, 2016. Effect of on-site support on laboratory practice for human immunodeficiency virus, tuberculosis, and malaria testing. Am J Clin Pathol 146: 469–477. [DOI] [PubMed] [Google Scholar]
- 8.Eliades MJ, Alombah F, Wun J, Burnett S, Martin T, Kutumbakana S, Dena R, Saye R, Lim P, Hamilton P, 2019. Operational considerations and costs of malaria case management supportive supervision. Am J Trop Med Hyg 100: 861–867. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Burnett S, Wun J, Evance I, Davis K, Smith G, Lussiana C, Tesha G, Quao A, Robertson M, Hamilton P, 2019. Introduction of an electronic tool for improved data quality and data use during malaria case management supportive supervision. Am J Trop Med Hyg 100: 889–898. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Marinucci F, Majigo M, Wattleworth M, Paterniti AD, Hossain MB, Redfield R, 2013. Factors affecting job satisfaction and retention of medical laboratory professionals in seven countries of sub-Saharan Africa. Hum Resour Health 11: 38. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Eliades MJ, Wun J, Burnett S, Alombah F, Amoo-Sakyi F, Chirambo P, Tesha G, Davis K, Hamilton P, 2019. Effect of supportive supervision on performance of malaria rapid diagnostic tests in sub-Saharan Africa. Am J Trop Med Hyg 100: 876–881. [DOI] [PMC free article] [PubMed] [Google Scholar]