Challenges in Routine Implementation and Quality Control of Rapid Diagnostic Tests for Malaria–Rufiji District, Tanzania

Abstract

Rapid diagnostic tests (RDTs) represent an alternative to microscopy for malaria diagnosis and have shown high sensitivity and specificity in a variety of study settings. Current World Health Organization (WHO) guidelines for quality control of RDTs provide detailed instructions on pre-field testing, but offer little guidance for quality assurance once RDTs are deployed in health facilities. From September 2006 to April 2007, we introduced a histidine-rich protein II (HRP2)-based RDT (Paracheck) for suspected malaria cases five years of age and older in nine health facilities in Rufiji District, Tanzania, to assess sensitivity and specificity of RDTs in routine use at rural health facilities. Thick blood smears were collected for all patients tested with RDTs and stained and read by laboratory personnel in each facility. Thick smears were subsequently reviewed by a reference microscopist to determine RDT sensitivity and specificity. In all nine health facilities, there were significant problems with the quality of staining and microscopy. Sensitivity and specificity of RDTs were difficult to assess given the poor quality of routine blood smear staining. Mean operational sensitivity of RDTs based on reference microscopy was 64.8%, but varied greatly by health facility, range 18.8–85.9%. Sensitivity of RDTs increased with increasing parasite density. Specificity remained high at 87.8% despite relatively poor slide quality. Institution of quality control of RDTs based on poor quality blood smear staining may impede reliable measurement of sensitivity and specificity and undermine confidence in the new diagnostic. There is an urgent need for the development of alternative quality control procedures for rapid diagnostic tests that can be performed at the facility level.

INTRODUCTION

Prompt, reliable diagnosis is essential to the effective management of malaria. Clinical diagnosis alone is not specific, and results in inappropriate use of antimalarial drugs.^1–5 When inexpensive drugs (such as chloroquine and sulfadoxine-pyrimethamine) were effective, presumptive treatment of all febrile patients was thought to be cost-effective. However, increasing drug resistance to these compounds and the higher cost of alternative medications has led to an increased focus on malaria diagnosis. The introduction of rapid diagnostic tests (RDTs) for malaria in the 1990s offered hope for a simple, accurate diagnostic test that could be performed in health facilities without other diagnostic modalities. Field trials of histidine-rich protein II (HRP2)-based RDTs showed sensitivity and specificity of over 90% for Plasmodium falciparum malaria at parasite densities of > 200 parasites/µL.^6,7 However, few trials have examined healthcare worker behavior and RDT performance during routine implementation to determine whether sensitivity and specificity remain high and whether use of the test reduces over-treatment of malaria. One recent clinical trial in Tanzania did report 95.4% sensitivity and 95.9% specificity of RDTs performed by healthcare workers, but 30–63% of RDT negative patients were still treated with an antimalarial drug.⁸

The World Health Organization (WHO) recommends that RDTs be implemented with a comprehensive quality control strategy.⁹ First, RDTs should be purchased from a manufacturer that follows good manufacturing practices (GMP). Second, each lot of RDTs should be tested on arrival in the country of use to ensure that the tests weren’t exposed to extreme temperatures or other conditions that may affect RDT performance. RDT performance is measured by testing known dilutions of parasites (typically 200 and 5,000 parasites/µL) and a negative control.¹⁰ WHO also recommends post-deployment testing at the health facility level, but these recommendations are less developed. Suggested mechanisms include sentinel site monitoring, increased training and supervision, and teaching healthcare workers problem-solving skills when RDTs are not performing well. Because of the logistical difficulties of collecting and staining blood smears from remote sites, microscopy is not listed among potential post-deployment quality control mechanisms.⁹

In September 2006, the Tanzanian National Malaria Control Program introduced artemether-lumefantrine (AL) for the treatment of uncomplicated malaria because of increasing resistance to sulfadoxine-pyrimethamine (SP). Quinine remains the drug of choice for the treatment of severe malaria, malaria in the first trimester of pregnancy, and malaria in children weighing < 5 kg.¹¹ Malaria diagnosis is primarily based on clinical symptoms and provider experience. Microscopy is available at most hospitals and health centers, but not at the most peripheral health facilities. ParaHIT-f (Span Diagnostics, Surat, India), ICT Malaria Pf (ICT Diagnostics, Cape Town, South Africa), and Paracheck (Orchid Biomedical Systems, Mumbai, India) are registered for routine use in Tanzania. RDTs have been introduced in operational research on the mainland and on the island of Zanzibar. National Integrated Management of Childhood Illness (IMCI) guidelines state that all febrile children < 5 years of age should be treated for malaria. National guidelines for malaria treatment and diagnosis provide an inconsistent message regarding the use of diagnostics—microscopy or RDT—particularly for children < 5 years of age. Current guidelines include a treatment algorithm that recommends treating children < 5 years of age for malaria, even when they have tested negative by RDT or blood smear. Because of the lack of local experience in using RDTs and the possible confusion to the healthcare worker about treating a patient who tests negative, we provided training in RDT use and introduced them in nine health facilities in a rural district in Tanzania to gain experience that would help inform national policy regarding the routine use of RDTs in health facilities.

MATERIALS AND METHODS

Location

From September 2006 to April 2007, we introduced an HRP2-based RDT (Paracheck) for suspected malaria cases in nine health facilities with existing microscopy services in Rufiji District, Tanzania. Rufiji District is a rural setting with holoendemic malaria transmission located 178 km south of Dar es Salaam on the Indian Ocean. The district’s > 200,000 inhabitants are served by 59 health facilities, including two hospitals. An estimated 89% of the district’s population lives within 5 km of a health facility and acute febrile illness, including malaria, is one of the leading causes of death in the district.¹² There is considerable variation in the transmission intensity in Rufiji District from year to year, but the predominate species is P. falciparum with an average entomologic inoculation rate (EIR) of several hundred infectious bites per year. The majority of residents are subsistence farmers.

Training

There were 105 healthcare workers trained to perform RDTs at the nine health facilities during the first month of implementation. Training was conducted on site a half-day at each health facility, and all healthcare workers present on the training day were included. Training included RDT job aides, a dosing chart for AL, a treatment algorithm for incorporating test results, and practice in performing the rapid diagnostic test and thick blood smear. Because of conflicting messages regarding testing in children < 5 years of age and national IMCI guidelines, we instructed healthcare workers not to test children < 5 years of age and to instead treat febrile episodes according to national IMCI guidelines for this age group. Healthcare workers were instructed to perform RDTs and blood smears on all patients 5 years of age and older with fever or in whom they suspected malaria.

Laboratory procedures

Healthcare workers were trained to perform Paracheck RDTs according to the manufacturer’s instructions. Thick blood smears were collected for all patients tested with RDTs and stained and read by laboratory personnel in each facility. Field stain used in many facilities was replaced with 10% Giemsa for quality control purposes. Laboratory technicians were trained to stain blood smears for 30 minutes, although actual practice may have varied. Thick smears were subsequently reviewed by an experienced reference microscopist blinded to initial reading and RDT results to determine RDT sensitivity and specificity. The reference microscopist counted parasites against 200 white blood cells (WBC) and examined 100 fields before declaring slides negative. The expectation from prior field studies was that RDTs would maintain 80% or greater sensitivity and specificity throughout the evaluation period.

Ethical review

The implementation protocol was reviewed and determined exempt by the institutional review boards of the U.S. Centers for Disease Control and Prevention (CDC) and the Ifakara Health Research and Development Center.

Data analysis

Data and blood smears from each facility were collected weekly. Data collectors summarized the number of patients seen, the number of positive and negative RDTs, and the treatments received. Individual RDT results were compared with the reference microscopist’s reading of the thick blood smear to determine RDT sensitivity and specificity. Supervisory visits included a checklist that evaluated healthcare worker RDT performance. Data were entered into an EpiInfo version 3.3.2 (Centers for Disease Control and Prevention, Atlanta, GA) database for descriptive analysis. The SAS 9.1 (SAS Institute, Cary, NC) PROC GENMOD was used to perform log-binomial regression to model the change in RDT sensitivity with increasing parasite density.

RESULTS

During the seven months of implementation, 58,685 patients were seen at the nine health facilities. There were 16,779 (28.6%) patients diagnosed with malaria, and at least 13,178 (78.5%) patients diagnosed with malaria received AL (Figure 1). Only 3,650 (21.8%) of patients diagnosed with malaria had a positive RDT performed at the health facility, the remaining 13,129 (78.2%) were clinically diagnosed. Among patients diagnosed with malaria, 8,121 (48.4%) were children < 5 years of age. Despite training that promoted IMCI and requested that the healthcare worker not perform RDTs in infants and young children, 1,866 RDTs were performed on children < 5 years of age. Among adults and children five years of age and older diagnosed with malaria, 2,874 (33.2%) had a positive RDT and 5,784 (66.8%) were clinically diagnosed. Healthcare workers treated patients of all ages with positive RDTs and only treated 183 (1.5%) patients with negative RDTs.

Figure 1.

Patients diagnosed with malaria and treatment provided in nine health facilities in Rufiji District, Tanzania, October 2006–April 2007.

There were 15,661 RDTs and 12,539 blood smears performed during the implementation period, and 3,650 (23.3%) of RDTs were read as positive by the health facility staff. Only 10,765 (85.9%) blood smears were of sufficient quality to be read by the reference microscopist, and 2,134 (19.8%) of the blood smears read by the reference microscopist were positive. 20.8% (2873/13,795) of RDTs performed in older children and adults were positive; and 41.6% (777/1,866) of RDTs performed in children < 5 years of age were positive.

In all nine health facilities there were significant problems with the quality of staining and microscopy. Slides and stain had many bacterial contaminants. Health facilities were using poor quality immersion oil that caused slides to adhere to the wooden slide box and damaged the microscope stage. Supervisory visits frequently found that laboratory technicians did not read blood smears, but recorded the RDT result for both tests. Laboratory technicians from all nine health facilities were sent to a training laboratory in Bagamoyo, Tanzania for a five day “refresher” training course in January 2007 covering blood smear preparation, parasite identification, and how to perform an RDT. Intensive refresher training did not result in substantial improvements in the quality of slide preparation and had no measurable effect on RDT sensitivity and specificity.

There were also problems noted with the healthcare workers’ performance of the rapid diagnostic test. Supervisors found that timing devices weren’t used consistently to ensure that results were read 15 minutes after adding the buffer solution, as directed. In 42 supervisory visits, 8 (19%) healthcare workers did not read the RDT results at the appropriate time. The loop device included in the Paracheck RDT was difficult for healthcare workers to use. The loop should be dipped in a large drop of blood collected on the finger, but frequently blood was dropped onto the loop resulting in too much blood transferred to the RDT. Furthermore, the size of the loop made it difficult to transfer directly to the nitrocellulose paper often resulting in the blood being deposited on the side of the plastic test cassette. Healthcare workers occasionally deposited blood into the wrong hole (noted on 2 [4.8%] of 42 supervisory visits). Job aides were inconsistently used in 17 (40.5%) of 42 supervisory visits, and there were some initial problems with cassette labeling.

Sensitivity and specificity of RDTs were difficult to assess given the poor quality of routine blood smear staining. Mean operational sensitivity of RDTs based on reference slide reading was 64.8%, but varied greatly by health facility (range 18.8–85.9%). Health facility performance of RDTs, as measured by sensitivity, was consistent across the implementation period—whether performance was consistently good or bad. Overall sensitivity remained relatively constant during the seven months of implementation but never reached the 80% implementation goal. Figure 2 illustrates the sensitivity and specificity of RDTs by month of implementation. The figure also illustrates there was a seasonal increase in the percentage of positive blood smears after a short rainy season from October to December. Specificity of RDTs remained high at 87.8% despite relatively poor slide quality.

Figure 2.

Sensitivity and specificity of rapid diagnostic tests (RDTs) by implementation month, Rufiji District, Tanzania, October 2006–April 2007.

Sensitivity of RDTs increased with increasing parasite density. During the initial months of implementation positive blood smears were given a qualitative reading (e.g., +, ++, +++). After identifying problems with RDT sensitivity, blood smears were reviewed for quantitative reading (parasites per 200 WBC). Among 471 positive blood smears reviewed from mid-January to March 2007, overall RDT sensitivity was 65.2%, as shown in Table 1. Excluding the 15 patients with 1–5 parasites per 200 WBC (< 200 parasites/µL), which is below the reported detection limits of the RDT, overall sensitivity remained at 65.6%. Sensitivity of RDTs remained below 50% for detecting parasitemia of 6–50 parasites per 200 WBC (240–2,000 parasites/µL). RDT sensitivity increased with increasing parasite density and this trend was statistically significant (P < 0.001).

Table 1.

Rapid diagnostic test (RDT) sensitivity by parasite density, Rufiji District, Tanzania, January–March 2007 (2,311 blood smears reviewed, 471 positive blood smears)

N	Parasite density per 200 WBC	Parasite density per µL	Sensitivity (%)
15	1–5	40–200	53.3
71	6–20	240–800	47.9
91	21–50	840–2,000	42.9
71	51–100	2,040–4,000	69
108	101–500	4,040–20,000	72.2
115	> 500	> 20,000	86.1
471	–	–	65.2

WBC = white blood cells.

Paracheck RDTs were lot quality tested by WHO laboratories in Manila, the Ifakara Center’s Bagamoyo Research Trials Unit in Bagamoyo, Tanzania, and in Atlanta by CDC. All RDTs tested in Manila passed quality assurance testing at 200 and 5,000 parasites/µL. In Bagamoyo, one RDT failed to detect a high-density parasitemia (5,000 parasites/µL), and one test cassette in Atlanta had a manufacturing defect that prevented buffer from moving up the nitrocellulose paper rendering the test invalid. Scientists in Atlanta also noted that samples with low-density parasitemia (200–500 parasites/µL) produced faint positives that were difficult to read and the monoclonal antibody line was much thinner than the control line.

DISCUSSION

There are several possible reasons for the low sensitivity measured during this implementation period. First, blood smears stained in health facilities are an imperfect “gold standard”. Poor staining technique or categorization errors by the reference microscopist may have led to over reporting of positive blood smears, reducing RDT sensitivity. Similarly, healthcare worker performance of the RDT may have affected measured sensitivity. The loop blood transfer device included with the Paracheck RDT was difficult for healthcare workers to use properly and may have resulted in too little or too much blood transferred to the test well. Adding too little blood may have prevented the test line from appearing, while adding too much blood may have decreased result readability because of hemolyzed blood covering the test paper. Healthcare workers were frequently unable to locate the timing device provided, and may not have adhered to the manufacturers’ instructions to read the test after fifteen minutes. Reading the test too soon may prevent detection of faint positives because of the continued presence of hemolyzed blood. Problems with RDT manufacturing may have also contributed to poor measured sensitivity. One potential manufacturing problem is that the monoclonal antibody test line is much narrower than the control line. This may have led some healthcare workers to incorrectly read a weakly positive test as negative. A final possible explanation of the low measured sensitivity is variation in the HRP2 genotype of the P. falciparum parasites in Rufiji District. Low numbers of repeats of specific gene sequences have been associated with lower RDT sensitivity.¹³ However, this was only noted at parasite densities of 200–1,000 parasites/µL, and would not account for poor sensitivity measured at higher parasite densities. Furthermore, these HRP2 gene variations have been more frequently described in Southeast Asia, and not in Africa.^13,14

We experienced many technical problems with implementation of RDTs in rural Tanzania that had not been previously documented. Despite supervision and “refresher” training, performance of blood smears remained poor, and RDT sensitivity was highly variable. Supervisory assessments of healthcare workers’ performance of each step of the RDT process continually reported adequate performance despite poor sensitivity. Healthcare workers trusted RDT results and did not read all blood smears performed. Continued reliance on clinical diagnosis of malaria was evident in that only one-third of the older children and adults diagnosed with malaria had a positive RDT. The requirement of performing both an RDT and a blood smear may have overburdened laboratory staff and led to decreased compliance with diagnostic testing. High healthcare worker adherence to RDT results in this implementation was an unexpected finding. Previous social science research has shown a strong preference for laboratory diagnosis prior to treatment in this area of Tanzania (Interdisciplinary Monitoring Project for Antimalarial Combination Therapy in Tanzania (IMPACT-Tz), unpublished data). Healthcare worker confidence in RDT results remained high despite poor measured sensitivity. Furthermore, given the low sensitivity of RDTs, adherence to RDT results may have reduced the frequency of appropriate treatment of patients with malaria. This is of particular concern for the more than 1,000 children < 5 years of age who tested RDT negative and were not treated with an antimalarial, although they should have been per IMCI guidelines.

During this implementation, we used blood smears to conduct quality control of RDTs during routine implementation. Institution of quality control of RDTs based on poor quality blood smear staining may impede reliable measurement of sensitivity and specificity and undermine confidence in the new diagnostic. Because of initially poor RDT sensitivity, we continued to collect blood smears throughout the implementation period. Collecting blood smears for 1–2 days per month at health facilities performing RDTs might provide sufficient quality control, but remains labor intensive. A district health management team may not be capable of devoting a reference microscopist or supervisor to monitor RDT performance this intensively. Therefore, what constitutes an appropriate quality control system for RDTs used in rural areas of Tanzania? Previous suggestions that frequent supervision with evaluation of RDT performance could serve as quality control proved insufficient to detect problems with RDT sensitivity during this implementation. Additional training also had little impact on RDT performance. Further WHO recommendations for post-deployment quality control include sentinel site surveillance and teaching problem solving skills to healthcare workers when RDTs aren’t performing well. Sentinel sites surveillance would only provide an assessment of RDT quality at a few centers, which might be improved by staff that have higher levels of training or additional supports. Problem-solving skills for when RDTs are not performing well would first require that the healthcare worker be able to identify poor-performing RDTs—which healthcare workers did not do during our implementation.

There is an urgent need for the development of alternative quality control procedures for rapid diagnostic tests at the facility level. Local microscopy is frequently poorly sensitive and specific and unsuitable for quality control. Likewise, supervision and lot sampling appear insufficient to detect problems with RDT performance. One potential solution that is under development is a recombinant protein positive control. A positive control would allow the healthcare worker to test RDTs on-site and frequently monitor RDT performance. Such innovations should be prioritized and thoroughly evaluated in routine implementation sites to ensure that healthcare workers are able to identify problems with RDT performance using this tool. In the meantime, periodic supervision and comparison to reference microscopy may be the best currently available option for quality control at the health facility level.

The annual procurement of RDTs has increased exponentially in the past 5 years from 16,700 in 2000 to over 11 million in 2005 because of increasing support from the Global Fund to Fight AIDS, Tuberculosis and Malaria, the U.S. President’s Malaria Initiative, and other donors.¹⁵ Without appropriate quality control mechanisms in place, pre- and post-deployment, RDTs may inadvertently reduce appropriate treatment of patients with malaria while also reducing overtreatment. Reduction in appropriate treatment of malaria in vulnerable populations—young children, pregnant women, and the immunocompromised—could have serious consequences and ultimately undermine confidence in RDTs. Donors purchasing RDTs should collaborate with National Malaria Control Programs to ensure that adequate quality control procedures are implemented at all levels of the healthcare system using RDTs and must be willing to provide funding to support these essential functions.