Perspectives on Implementation Considerations and Costs of Malaria Case Management Supportive Supervision

Abstract.

Between 2012 and 2017, the U.S. President’s Malaria Initiative–funded MalariaCare project supported national malaria control programs in sub-Saharan Africa to implement a case management quality assurance (QA) system for malaria and other febrile illnesses. A major component of the system was outreach training and supportive supervision (OTSS), whereby trained government health personnel visited health facilities to observe health-care practices using a standard checklist, to provide individualized feedback to staff, and to develop health facility–wide action plans based on observation and review of facility registers. Based on MalariaCare’s experience, facilitating visits to more than 5,600 health facilities in nine countries, we found that programs seeking to implement similar supportive supervision schemes should consider ensuring the following: 1) develop a practical checklist that balances information gathering and mentorship; 2) establish basic competency criteria for supervisors and periodically assess supervisor performance in the field; 3) conduct both technical skills training and supervision skills training; 4) establish criteria for selecting facilities to conduct OTSS and determine the appropriate frequency of visits; and 5) use electronic data collection systems where possible. Cost will also be a significant consideration: the average cost per OTSS visit ranged from $44 to $333. Significant variation in costs was due to factors such as travel time, allowances for government personnel, length of the visit, and involvement of central level officials. Because the cost of conducting supportive supervision prohibits regularly visiting all health facilities, internal QA measures could also be considered as alternative or complementary activities to supportive supervision.

INTRODUCTION

Case management of malaria, including early and accurate diagnosis, and effective treatment adherent to diagnostic test results, is an essential component of a malaria control and elimination strategy. Successful delivery of case management services requires several components. Guidelines and policies must be up-to-date to reflect World Health Organization (WHO)–recommended standards and must be available to providers and through training institutions. Care must be accessible and affordable. A quality assurance (QA) system must be established to build and maintain quality and to ensure effective referral and follow-up. Drugs, supplies, and equipment must be available. Finally, effective communication mechanisms must be in place to make patients and caretakers aware of the symptoms and to know when and where to seek care.

Health-care worker (HCW) knowledge and practices are key components of the case management continuum. Clinical, laboratory, and pharmacy staff must be trained to diagnose and treat effectively. A key component of most QA systems is supervision. Under traditional supervision models, supervisors are tasked with inspection of health facilities, primarily by assessing overall facility supplies and operations. In such situations, supervisors rarely focus on competency-based indicators.¹ Supportive supervision, a more recent model, emphasizes the importance of joint problem-solving, mentoring, and two-way communication between the supervisor and those being supervised,² and has demonstrated positive impacts on HCW performance at both the facility and community levels.^3–7 Unfortunately, few publications have examined, on a multicountry level, the operational considerations and challenges, or the costs, of implementing a supportive supervision system.^2,8,9

Between 2012 and 2017, the U. S. President’s Malaria Initiative (PMI)– funded MalariaCare project supported national malaria control programs (NMCPs) in designing and implementing a case management QA system for malaria and other febrile illnesses, or components thereof, based on NMCP needs and requests. MalariaCare’s comprehensive QA approach is detailed in Figure 1.

Figure 1.

MalariaCare’s QA approach. Reference numbers signify steps in the QA approach. QA = quality assurance.

In some countries, per NMCP requests, MalariaCare provided technical assistance to ensure that policies and guidelines were updated to align with global standards (Figure 1, Step 1). To disseminate the guidelines and build skills, MalariaCare also supported skills-based training for both malaria diagnostics and clinical case management (Step 2). For malaria microscopy, the project supported the development of an expert laboratory cadre to serve as trainers over the long term. To further disseminate knowledge and reinforce competencies on-the-job, trained supervisors conducted outreach training and supportive supervision (OTSS) visits; these served as the core component of MalariaCare’s QA approach (Step 3). Outreach training and supportive supervision visits enabled supervisors to use supportive supervision to work collaboratively with health facility staff and enabled on-site training and mentorship through structured observation of HCW and health facility practices.

Outreach training and supportive supervision supervisor teams consisted of supervisors recruited from referral hospitals and district and regional health offices; they generally included a laboratory and a clinical expert. Depending on country-specific policies or needs, the teams also included a malaria focal point, a pharmacist and a health information officer. The teams typically visited one to two facilities a day and were guided by a structured case management checklist to observe staff performing diagnostic testing, including use of rapid diagnostic tests (RDTs) and microscopy, and clinical consultations. The supervisors provided both positive and critical feedback to staff. At the end of a visit, supervisors prioritized the top problems found, reviewed whether problems found during a previous visit were resolved, and developed, together with facility managers and staff, an action plan to address recurring and new problems.

Between OTSS visits, supervisors and subnational and national officials convened during lessons learned workshops (LLWs) to review OTSS data, identify common high and low performance areas, and to agree on action to address systemic issues such as widespread stock-outs or lack of training, which affected the facility’s ability to maintain or improve quality case management (Step 4). To address persistent gaps not resolved by OTSS or formal training, MalariaCare organized additional capacity-building activities in some countries (Step 5).

This study examines operational aspects related to supportive supervision, reflecting on MalariaCare’s experiences in supporting OTSS visits in nine countries. The review is not meant to be exhaustive, but rather to focus on selected areas for national programs to consider when implementing a QA system that includes supportive supervision, including checklist content; supervisor selection, evaluation, and training; facility selection; measurement of progress; and cost. Other critical operational challenges—such as stock-outs—also exist but require government or donor intervention beyond the supervision programs.

MATERIALS AND METHODS

Based on the needs and requests of the NMCPs of these countries, nine of the 17 countries that were supported by MalariaCare (Democratic Republic of the Congo [DRC], Ghana, Kenya, Madagascar, Malawi, Mali, Mozambique, Tanzania, and Zambia) received OTSS visits between 2012 and 2017. Health facilities selected for OTSS visits varied over the course of the project, due to shift in the strategic regions within each country that PMI supported, logistical issues, and progress or decline in regional performance. In addition, hospitals and other high-volume facilities were targeted in each country to specifically assess competencies in malaria microscopy and management of severe malaria. By the end of the project, more than 5,600 health facilities had received between one and 10 or more OTSS visits.

Perspectives on the implementation of OTSS were elicited during debriefing meetings with MalariaCare staff after each round of visits, and during LLWs with government health department staff and OTSS supervisors. Supervision quality and progress toward project objectives were assessed using data from the OTSS checklists that supervisors filled out (either on paper or in a mobile application). Finally, cost data were collected from project expenditure reports for the most recent round of OTSS visits.

DISCUSSION

The following aspects of planning a supportive supervision scheme were found to be important to ensure quality.

Checklist development and implementation.

A checklist used during supportive supervision generally provides the framework for conducting a visit. Under MalariaCare, the checklist was meant to be not simply a data collection tool to ensure that supervisors assessed all relevant facility practices, but also a mentoring aid, in line with the overall objective of the supportive supervision program. The standard MalariaCare OTSS checklist included questions on human resources, availability of malaria reference material, drugs, supplies, and equipment; a register review to measure adherence to testing and treating protocols over the previous 3 months; and up to three observations each of various staff performing a malaria RDT, performing malaria microscopy, or conducting a clinical consultation of a febrile patient (see Supplemental File 1). Designing a supportive supervision checklist that would provide an accurate picture of health facility performance was challenging. Longer checklists reduce time available for actual mentoring, increase data entry costs, and require significant time to analyze the data. National programs generally do not have the resources to accommodate those needs. Thus, the contents of a checklist should not be viewed as immutable, but rather should be evaluated and refined over time as NMCP needs evolve. As checklists further develop, any proposed changes should be assessed during actual supportive supervision visits to ensure that time available for mentorship increases, and that data and information generated are useful to facility staff, supervisors, and the national program. As an example, the MalariaCare OTSS checklist included a register review module, which required a supervisor to match patient data across a facility’s laboratory, outpatient, and/or pharmacy registers to assess adherence to test results and whether recipients of antimalarials had previously received a positive malaria test result. This was a time-consuming activity, particularly at large health facilities with multiple registers which, generally, were poorly filled out. Ultimately, the register review was retained as part of the OTSS checklist because it provided an opportunity to discuss testing and treatment protocols with health facility staff. However, supportive supervision schemes aiming to assess similar indicators should consider triangulating or replacing the collection of such data with that from other sources, such as national health management information systems, or conducting the register review section less frequently.

All checklist questions were developed primarily to facilitate mentorship and on-site training, but analyzing the results for all questions was not found to be an efficient use of program managers’ time. In the case of MalariaCare, six key indicators were developed so that the focus of attention would be on areas that were most influenced by OTSS (adherence to proper diagnostic and clinical evaluation practices). These key scores helped program managers and health authorities not conducting the visits themselves to take necessary action between OTSS visits.

Like the development of the checklist, the development of a scoring system that accurately captures progress also requires testing and adjustment over time. For example, MalariaCare moved from calculating the percentage of steps in a given set of observations to a weighted score, where certain steps that were considered more important for accurate diagnosis and treatment counted for a higher proportion of the score.

Supervisor selection and ongoing evaluation.

Throughout the project, MalariaCare staff recommended that OTSS supervisors be selected based on posttest performance and assessments during supervisor training. Ultimately, however, supervisors were selected by national and subnational health departments. Selection also was influenced by the position within the government system held by the supervisor and based on the availability of staff to conduct supervision during a specific time period. For example, many supervisors conducting OTSS under MalariaCare were unable to dedicate more than 2 weeks at a time to conducting OTSS because of their clinical or laboratory responsibilities. Furthermore, in the project’s experience, the most capable supervisors were more likely to be called away to other duties or transferred to new regions, decreasing the chance for consistent, high-quality supervision at a particular facility. National programs could, therefore, consider establishing competency criteria that are realistic for the country, and ensuring that local health authorities (who are typically responsible for selecting supervisors) understand and buy into the selection process. As an example, MalariaCare initially aimed to have all laboratory supervisors attain WHO Level 1 or 2 equivalence on parasite detection and quantification during malaria diagnostic refresher trainings (MDRTs). This objective, however, proved difficult to achieve in most countries. As an alternative, the project recommended that candidates with the highest scores and with more bench practice be selected as supervisors.

Once supervisors are selected and trained, supportive supervision programs should consider periodically assessing and reinforcing supervisor competencies in the field, first by assessing as many supervisors as possible during their initial visits, then progressively decreasing assessments once the supervisors appear to follow protocol consistently. Under MalariaCare, program and government staff visited a sample of health facilities during each round of OTSS visits to ensure that supervisors fulfilled expectations and were able to use the checklist effectively. In the process they provided feedback to supervisors to improve their performance. In the final year of the project, a checklist was developed to evaluate supervisors’ preparation, their conduct during observations, the accuracy and completeness of the supervision checklist, and their ability to develop and follow up on appropriate action plans (see Supplemental File 1). Such a system for evaluating supervisors can help supportive supervision programs identify common supervisor deficiencies and emphasize certain points during future supervisor training. The results from the initial evaluation of supervisors indicated that overall performance was high, but that development and follow-up on action plans was inconsistent.

Supervisor training.

The availability of qualified supervisors is a precondition for a functional supportive supervision model. Competent supervisors, with a combination of expertise and knowledge transfer skills, have the ability to influence HCW behavior in a positive way. Previous studies on supervision note that supervisors who are perceived by staff to act in a judgmental, rather than a supportive, fashion are often ineffective in changing HCW behavior and looked upon by facility staff as a distraction rather than a help.^1,5,9,10

Supervisors must not only demonstrate technical competence within their field, but also have the ability to foster a positive learning environment. Training supervisors on both sets of skills before conducting supervision visits is crucial. Under MalariaCare, laboratory supervisors participated in MDRT courses to improve microscopy and RDT performance. The course included daily skills assessments over a 5-day period. Clinical supervisors attended clinical case management training (CCMRT) focusing on patient assessment, the importance of appropriately testing for malaria before treatment, recognizing the signs of severe disease, administering correct treatment, and counseling and referring as needed. This typically was covered over a 5-day period. Training to improve mentoring skills and familiarize supervisors with the checklist was conducted over a 3-day period. MalariaCare supervisor training included didactic sessions, skills-based practical sessions (microscopy and RDT), discussion, and role-playing. Based on MalariaCare’s experience in training more than 3,500 participants in more than 100 supervisor training sessions, a minimum of one half day devoted to role-playing, effective communication skills, and identifying common root causes of problems was required to transform existing supervisors’ understanding of supervision from a traditional model to a supportive one. One half to a full day was assigned to field visits to a nearby health facility—this was useful for helping supervisors understand the structure of the checklist and visit. Over the course of the 5-year project, most supervisors received one technical skills training and one supervision skills training. Generally, we found that supervisors who were supervised in the field became good mentors without the need for additional training and that the knowledge of national malaria case management guidelines gained from CCMRTs appeared to last over several supervision rounds. On the other hand, MalariaCare field staff felt that microscopy skills (particularly for quantification) generally deteriorated over time. Thus, in an ideal situation, laboratory supervisors should receive periodic MDRT. Given the low proportion of slides read correctly during MDRT pretests in MalariaCare supported-countries (67.3% for parasite detection, 30.2% for species identification, and 15.5% for parasite quantification), and subsequent improvement following the training, this seems warranted. The WHO Malaria Microscopy QA Manual–Version 2 also recommends refresher training for malaria microscopy every year.¹⁰

Initial facility selection and visit frequency.

Even though all health facilities would benefit from supportive supervision to varying degrees, resource constraints generally make it necessary to target and prioritize this support to those most in need. The cost and human resource demands of running and sustaining a supportive supervision program for all facilities in a given country is often prohibitive: in the nine countries where MalariaCare provided OTSS support, the total number of health facilities ranged from 1,060 (Malawi) to 8,943 (Kenya) (see Table 1). In addition, routinely conducting supportive supervision visits at all facilities in a country would not be cost-effective, as health facility performance will vary and improve at different rates for different facilities. Developing robust criteria for initial facility selection, and to determine the optimal frequency of visits over time, is essential for an effective supportive supervision program.

Table 1.

Average cost of daily allowance and transport cost per facility visit

Country*	Average inputs						Daily allowance and transport cost per visit ($)
	Supervisors per visit	Days per facility (visit plus travel time)	Daily allowance ($)	Lodging included in daily allowance†	Transport cost per day ($)	Vehicle origin
Ghana	4	1.14	$12.00	No	$110.50	Hire	$180.38
Kenya‡	2	1.25	$7.50	No	$10.00	N/A–Stipend	$43.64
Malawi	2	0.84	$12.90	Some	$124.00	Hire	$125.99
Mali	2.1§	1.03	$15.00	Some	$45.00	Government	$78.80
Mozambique	2	1.56	$32.00	Yes	$150.00	Hire	$332.89
Tanzania	3	0.74	$55.00	No	$79.00	Government	$181.64
Zambia	2	1.48	$15.67	Some	$48.84	Government	$119.01

Costs for Democratic Republic of the Congo are not shown because their most recent outreach training and supportive supervision rounds followed an intensive mentorship model, whereby supervisors spent additional days at each facility.

* Madagascar is not included, as only a small number of health facilities were visited near the capital.

† Lodging was included for a proportion of supervisors, depending on whether they had to travel far from their home base. The daily allowance reported is the average across all supervisors.

‡ In Kenya, transport costs are distributed as travel allowances to each supervisor. To calculate the transport cost per visit, the transport cost per day must be multiplied by the number of supervisors per visit, as well as days spent per facility.

§ In Mali, one of every nine supervision teams had three people; the rest had two.

National malaria control programs, in collaboration with the PMI mission and MalariaCare, followed one of two general strategies to initially select facilities for OTSS in each focus country. One approach, as implemented in Ghana and Tanzania, was to visit most or all facilities in a defined area at least once; this allowed for a comprehensive understanding of performance throughout the health system. In other countries, however, facility selection targeted high-burden or high-volume facilities to maximize impact. For example, in Zambia, districts were selected based on the highest burden of malaria cases according to the national health information system; but in other countries (DRC, Mali, Mozambique), reference hospitals for the project’s designated subregions were prioritized. Within these overarching national guidelines, local health authorities selected facilities based on their personal knowledge of the health facilities in their area.

Criteria must also be established to determine how frequently facilities receive supportive supervision visits. If, for example, one selection criterion is to support facilities with microscopy, specific performance indicators for microscopy (as measured from checklist data collected during each supportive supervision visit) can be used to assess progress and determine how often a facility should continue to receive OTSS. Once facilities reach a defined level of performance, the frequency of OTSS visits could be adjusted downward, and/or other QA measures could be applied to maintain performance. If performance were to decrease, facilities could revert to a more intensive OTSS schedule. The frequency of the visits, or the specific combination of OTSS with other QA measures, would need to be determined on a case by case basis, but we recommend that programs consider a minimum of one OTSS visit per facility per year.

In Tanzania, MalariaCare worked with the NMCP to implement such a facility planning strategy. District hospitals were prioritized and visited quarterly regardless of performance. However, other facilities were targeted based on performance. We selected facilities visited in the previous round that failed to meet a minimum standard on key indicators or that were missing checklist data. If there were more low-performing facilities to visit than resources allowed, priority was given to those with low microscopy scores. If resources were available for additional facilities after applying the criteria, the average score of the facilities visited in the first three rounds of OTSS guided selection.

Failure to establish standard criteria for determining which facilities should receive additional visits could mean that supportive supervision resources are focused on facilities selected for convenience rather than greatest need or that new facilities might be visited without returning to poor-performing facilities for follow-up. MalariaCare frequently found the latter to be true, as health department authorities wanted to cover as many health facilities as possible. Determining visit frequency is thus an exercise that requires national-level standards and guidance to ensure the efficacy of supportive supervision across multiple regions, reflecting the cost-effective use of scarce resources to strengthen service delivery at a group of facilities whose composition changes continuously due to performance fluctuations.

Checklist data management.

The MalariaCare QA approach relied on accurate and complete checklist data not only for facility selection, but also to assess supportive supervision impact and for highlighting focus areas for training and mentorship. Although the checklist had a large number of questions, MalariaCare focused on the results from a subset of them to track progress over time. In the earlier years of the project, tracking for even this limited set of indicators proved challenging: supervisors filled out hard copies of the checklist and often failed to input answers to key questions. Moreover, the hard copy checklists were sent to a central location for data entry, which in some cases added weeks or months to the timeline. Analysis of individual health facility progress over time was either time consuming (depending on the number of facilities visited) or impossible because of slightly different health facility names being written or entered for each round. To address these challenges later on, in all countries where large-scale OTSS was carried out (seven of nine countries), paper checklists were replaced with MalariaCare’s electronic data system (EDS), an Android application that links to District Health Information System version 2 (DHIS2) (University of Oslo, Oslo, Norway) for data storage and visualization.¹¹

Supervisors using EDS were equipped with handheld tablets to enter and upload checklist data over a mobile network. In the first rounds of OTSS using EDS, 89% of health facilities (n = 1,556) had at least one complete clinical observation, compared with only 23% during the last set of visits using the paper checklist (n = 1,462). Similar increases were seen for RDT and microscopy observations and for register review. We believe that this dramatic improvement was because of a combination of 1) supervisor preference for tablets over paper checklists (based on feedback during training), 2) a feature within the tablet itself that calculated the percentage of the checklist that was complete, and 3) the emphasis during supervisor EDS training on the importance of accurately completing the checklist. Because of this experience, we recommend that an electronic system of data collection be used to enter and analyze supportive supervision data when resources permit. Although using an electronic system requires higher upfront costs and specialized training to manage it, it significantly improves the completeness of data and the ability to prioritize facilities for follow-up in a timely fashion. Based on a cost analysis of paper checklist data entry compared with EDS, electronic data capture tends to be costlier than paper checklists per visit. However, the difference between electronic data capture and paper checklist costs decreases when supportive supervision is conducted on a large scale. For example, EDS was $17.78 more expensive per visit than paper checklists in one country where 120 facilities were visited but dropped to $0.16 more per visit in another country, where 782 facilities were visited.¹¹ For supportive supervision programs that are unable to implement electronic systems but still want to be able to use data for decision-making, extra emphasis during supervisor training on checklist completeness, plus follow-up with supervisors after data entry on key missing areas, should improve data quality. However, timeliness, completeness, and accuracy is not likely to reach the performance levels seen when using an electronic system.

Operational costs of OTSS.

Table 1 presents average costs per visit for daily allowances and transport for supervisors for the most recent rounds of OTSS visits, as these are applicable to all intervention countries and to all supervision structures. Democratic Republic of the Congo was not included because their OTSS model followed an intensive mentorship model, where more days were spent at each facility. The total cost per visit for these two cost drivers ranged from $44 in Kenya to $333 in Mozambique. Daily allowance costs included supervisor meals and, if supervisors could not return to their homes at the end of the day, lodging. Transport inputs included vehicle hire, transportation stipends to supervisors, fuel for government vehicles, and/or daily allowances for hired drivers.

The average number of days per facility equals the amount of time spent at the facility for the actual OTSS visit plus travel time to move between facilities. Travel time was influenced by distance between facilities and accessibility. For example, travel time between facilities in Mozambique took more than half a day because of both geographic distance and infrastructure challenges; by contrast, facilities in Tanzania were close enough to each other that two facilities could be visited in 1 day.

Travel time to facilities substantially increased both transport and daily allowance costs by increasing the number of days needed to complete supervision visits. Costs were also higher for those countries where supervisors had to travel farther from their home to conduct OTSS, as this not only increased total travel days, but also required overnight stays. For example, in Kenya and Ghana, supervisors conducted OTSS in areas close to their homes and so were only provided a stipend for lunch; in countries where supervisors and their drivers had to travel significant distances from their home, lodging and full meals had to be provided. In Malawi, average daily allowance costs were kept low by paying supervisors based on facility distance—supervisors were only paid lodging if the distance to the facility required an overnight stay; otherwise, supervisors were only paid a lunch stipend.

Transport from facility to facility took one of three forms. Three countries used government vehicles, and transport costs included driver daily allowances and fuel. In two countries, government vehicles were not available and hired vehicles were used instead, thus, increasing transport costs substantially. In the case of DRC and Kenya, supervisors did not have to travel far from their home base and instead were provided a travel stipend to use local public transport rather than hiring vehicles, which kept transportation costs relatively low.

Beyond the OTSS visit itself supervisor training and monitoring adds to the cost of a supportive supervision program. The supervisors were mentored on identified problems during such visits, but not tracked over time, so the project was unable to document if “supervision of supervisors” visits improved long-term quality. Finally, using data from supportive supervision visits to make programmatic decisions also requires financial resources, whether for entering paper checklists in a database or using an electronic system, which may include the cost of tablets, airtime, and server space as compared with printing and data entry for paper checklists. The cost of both approaches is discussed in more detail by Burnett et al.¹¹

Supplementing supportive supervision with other QA activities.

Because the cost of supportive supervision prohibits the possibility of regularly visiting every facility, complementary or alternative methods of ensuring quality should be factored into an optimal QA system that includes supportive supervision. Instituting internal QA (IQA) measures within facilities, for example, has the potential to improve the quality of case management in a less costly and more continuous manner. In Kenya and Mozambique, MalariaCare helped revive or create committees at the facility level that meet regularly to address specific deficiencies related to malaria case management. Although MalariaCare was not able to measure improvement in these facilities compared with those without committees, feedback from facilities where such committees were implemented has been positive. Internal QA measures conducted within the laboratory (such as routinely checking the quality of slide stains, staff review of problematic slides, and monitoring standard operating procedures) could also supplement, or in some cases replace, the need for external supervisors to observe procedures. In an analysis of microscopy performance the factor associated with the largest improvement (aside from receiving OTSS visits) was having IQA procedures.¹² We believe that IQA approaches deserve additional attention and should be evaluated to determine their benefits over a longer period of time.

CONCLUSION

MalariaCare’s experience has indicated that supportive supervision, in the context of a comprehensive QA system, builds, improves, and maintains performance over time, as demonstrated in other studies from the project.^12–14 For a well-structured OTSS to be effective and efficient, host governments and other implementing partners should develop the following when implementing supportive supervision: 1) a checklist that balances data collection with the primary objective of mentoring; 2) criteria for supervisor selection coupled with ongoing evaluation; 3) a dynamic strategy for facility selection and criteria to determine the continued need for support and the type of support; 4) criteria for selecting facilities to conduct OTSS and determine the appropriate frequency of visits; and 5) a system that enables the analysis of supportive supervision data for decision-making. Ensuring the quality of supervision schemes through these methods can be cost- and human resource–intensive. Thus, national programs should consider supportive supervision as part of a broader, dynamic QA strategy that allows for adjustments to be made based on performance data and which uses an optimal combination of external quality assurance and IQA measures.

Note: Supplemental file appears at www.ajtmh.org.