Improving the ability of districts in Uganda to monitor their HIV programs

Abstract

BACKGROUND

Although district health teams (DHT) in Uganda are supposed to monitor and support facilities to ensure quality HIV data collection, reporting and use, they are often ill-equipped to do so. We implemented a program designed to build the capacity of districts to manage and use their own HIV-related program data and to assist facilities to collect and evaluate their own data.

METHODS

We conducted a baseline assessment of the monitoring and evaluation (M&E) capacity of 38 districts. In the 10 worst-performing districts, we identified and trained district-level staff to become M&E mentors who in turn trained and supervised facility-level staff. We collected information on action plans developed by facilities to address major issues of concern. Following the intervention, we reassessed M&E capacity of the 10 targeted districts.

RESULTS

Among the 38 districts assessed, one-half did not have a biostatistician, less than one-quarter had staff trained in the basics of M&E or data analysis, and less than one-quarter had an M&E plan. The main concerns of facilities included lack of updated data collection tools, lack of supervision, inaccurate data recording, and limited ability to analyze and use data. In the 10 targeted districts, comparison before and after the intervention showed that the number of districts with trained M&E staff increased (4 to 9), the number of M&E plans increased (3 to 6), and the number using data for programming increased (4 to 8). Implementation of action plans by facilities successfully addressed many issues and led to improved programming.

CONCLUSION

Challenges of district M&E in Uganda mainly result from a lack of skilled human resources. On-the-job training and direct involvement of district staff to provide support to facilities can lead to improvements in data quality and use.

BACKGROUND

Routine monitoring and evaluation (M&E) are required to ensure that health care programs act responsibly to fulfill their mandates and that they are effective. Uganda has made an effort to harmonize M&E of HIV care and prevention programs at the national level (Uganda MOH, 2010). However, to be most effective, M&E also needs to happen on the ground where programs are implemented and data are collected. In Uganda, the primary monitoring of HIV programming is supposed to be undertaken at the level of its 112 districts which oversee local government, administration, and health services. Although regions exist in Uganda, they are geographic designations and do not have administrative functions.

To date, district-level M&E of health programs in Uganda has primarily focused on collecting and aggregating data from facilities and reporting this information to the national level through an electronic system called DHIS2 (https://www.dhis2.org/overview). Aggregate data are used to determine the overall reach of HIV programming and for reporting to funding agencies. In Uganda, patient-level data are collected at health care facilities on standardized paper-based Health Management Information System (HMIS) forms, aggregated, and then sent to the district health team (DHT). The DHTs are expected to analyze and interpret these data to determine how well HIV programming is functioning in their districts and to identify which facilities need better support. The district-based M&E unit is in charge of these functions and is comprised of the HMIS ‘focal person’ responsible for collecting aggregate data from facilities and reporting it to the national system; a biostatistician tasked with analyzing the district-level data; and a surveillance officer who ensures collection of information on reportable diseases through the Integrated Disease Surveillance Response system. The M&E team is also supposed to supervise and support facilities to ensure the completeness and accuracy of their data and the timeliness of reporting to the district.

Despite this system, many challenges exist in carrying out effective M&E sub-nationally (Uganda MOH, 2011; Uganda MOH, 2012). Basic infrastructure and human resource challenges prevent the DHT from working with facilities. Challenges include lack of transportation for supervisory visits, limited staffing, poor understanding of what is required for effective M&E, and low skill levels of staff. Inadequate funding at the national level precludes regular performance reviews of districts. Thus, even though districts are mandated to analyze and act on their own data, they rarely do so.

Inadequate human resources and low skill levels have been found to negatively affect data quality and use in other low- and middle-income country settings. The Performance of Routine System Management (PRISM) tool was developed to systematically identify behavioral and other factors associated with poor data quality in health information systems (Nicol et al., 2013; Nicol et al., 2016; Hotchkiss et al., 2010). Results of PRISM assessments in low- and middle-income countries identified multiple barriers to using data and being invested in its quality, including limited competencies in data manipulation and evaluation, as well as low levels of supervision.

Several approaches to addressing these challenges have been carried out in sub-Saharan Africa. These include task shifting, in which designated and trained staff members are put in charge of data management and M&E so that clinical health workers can be relieved of the burden of reporting (Fulton et al., 2011). This approach was tried in Botswana where recently graduated health care workers with no prior experience in data management or use were given on-the-job training in M&E (Mpofu et al., 2014). These health care workers then became district M&E officers, a new cadre in the country. This approach resulted in improved data quality, ownership, and use. To encourage evidence-based health programming and policy, public health students and practitioners in South Africa received training in how to analyze simple data sets so that they had the basic skills required for data use before they graduated (Williams et al., 2010). Others have focused on the importance of mentoring and site-based training for those already in district-level positions (Belrhiti et al., 2016, Edwards et al., 2015). However, a Cochrane review of interventions for hiring, training and retaining district-level managers could find only two studies that fulfilled their inclusion criteria and concluded that evidence for the importance of on-the-job training and other professional support programs was lacking (Rockers et al., 2013). This points to the fact that few evaluations of interventions designed to improve district-level data comply with criteria for scientific rigor.

To improve district–level M&E of HIV programs in Uganda, we developed, implemented and evaluated a model of assisting districts to analyze their data and to support facilities, based on the assumption that on-the-job training, mentoring, and support could improve data quality and use. The program was carried out by the Monitoring and Evaluation Technical Assistance (META) project, and continues to date with support from the Monitoring and Evaluation Technical Support (METS) project. Both are collaborations between the Makerere University School of Public Health and the University of California, San Francisco and have been supported by the U.S. Centers for Disease Control (CDC)-Uganda/President’s Emergency Plan for AIDS Relief. We describe the program and an initial evaluation of its effectiveness.

METHODS

Selection of districts and assessment of M&E capacity

CDC-Uganda supported META to assist districts with their HIV M&E capacity. Uganda has 112 districts which are ranked yearly by the Ministry of Health (MOH) based on a combination of seven health measures including immunization coverage, the proportion of pregnant women receiving antenatal care, and the timeliness and completeness of data reporting, among other indicators. Based on the 2012 MOH ranking, we identified the 38 districts in the country with the lowest rankings. Early in 2013, we conducted a baseline assessment of the M&E capacity of these districts. The assessment was based on a 20-item checklist on staffing, M&E training and skills, and infrastructure; it was completed by directly interviewing district-level staff. On the basis of these assessments, we prioritized 10 of the 38 worst-performing districts to receive an intervention designed to improve their M&E capacity; the intervention was conducted from 2013-2014. These 10 districts were chosen because they had the lowest scores on META’s baseline assessment, many of their staff had never received training in M&E, and/or because the MOH wanted them to be the first to receive assistance due to their poor performance on health indicators.

In 2015, we repeated the assessment of the 10 districts that received targeted support and mentorship from META. We report here on the process of improving the capacity of districts to monitor their programs, the results of the initial assessment of the 38 districts, and changes in M&E among the 10 targeted districts. METS, the follow-on project to META, has since provided support to the remaining 28 districts.

M&E training and support intervention

META developed a plan to support districts to take charge of their M&E activities based on meetings and collaboration with district-level staff (Figure 1). We held an initial meeting with the health and political leadership in each of the 10 targeted districts to share the results of the baseline assessment and to discuss the gaps that had been identified. The meetings were also used to underscore the need for M&E of local health care programs and to gain support from district leadership. During the same meeting, priority areas for training of staff of the DHT and facilities were identified. DHTs consist of the District Health Officer (DHO); assistant DHOs in charge of maternal, child, and environmental health; ‘focal persons’ for pharmacy and laboratory services, and the M&E unit.

Figure 1:

Approach to improving the capacity of districts to monitor and evaluate their HIV programs, Uganda, 2013-2015.

Following the initial meeting, a three-day workshop was held for the same group of health and government leaders in each of the 10 districts. The purpose was to provide more information about the importance of M&E and to equip district leaders with the supervisory skills necessary to support M&E in their respective districts. During the workshop, each district identified five members of their DHT to become district ‘M&E mentors.’ Each district was also asked to identify 10 health facilities within their district that had the most difficulty in completing HMIS forms in an accurate and timely manner. Thus, 100 facilities in total were identified.

Three staff members from each of the 100 targeted facilities were asked to participate in data management and use training. Typically, these staff included those in charge of the health facility, record assistants who entered data and compiled reports, and health care workers in charge of HIV clinics who were usually nurses. With support from META, the district mentors facilitated workshops in their district for these facility staff. Topics included data quality, completeness, and timeliness of reporting; the importance of completing key facility registers; the value of reporting to the district level; and how to make use of data. Facility staff also developed action plans to address the most problematic issues at their sites. Approximately three to five months after these workshops for facility staff, district mentors and META staff visited the targeted facilities to provide support and to discuss how well action plans had been executed. Revised action plans were developed based on whether the initial problems had been addressed or whether new issues could be tackled, with a repeat follow-up of action plans three months later.

District mentors also conducted monthly reviews of data from health facilities. They were supposed to follow-up in person with those facilities that appeared to be having problems with data timeliness, completeness or consistency and to provide support-supervision. Follow-up was to occur on a monthly basis. In some districts, mentors agreed to divide responsibility for certain facilities among themselves, and, in others, the mentors decided to visit each health facility as a team. Incentives for mentors to carry out these reviews and supervision were largely indirect. For example, we engaged the district political and health leadership to develop expectations of the DHT and accountability to the district as a whole. METS also provided some funding for mentors to help cover their transportation and out-of-station expenses incurred while visiting each health facility. In addition, META staff often accompanied mentors on supervision visits to facilities.

Evaluation of timeliness of reporting

META evaluated the timeliness of reporting to the district of two types of reports: a monthly report on HIV and prevention of maternal-to-child HIV transmission indicators and a quarterly report on HIV programming. We evaluated dates when reports were received from facilities at the district level as recorded in the DHIS2 system.

Analysis

Data from baseline and follow-up assessments of the district M&E capacity were entered and cleaned in Excel 2013 (Microsoft Corporation, USA). We computed descriptive statistics consisting of proportions. This assessment was conducted as part of routine programming, and no individual identifiers were collected. Therefore, human subjects’ ethics review was not required.

RESULTS

Table 1 shows the results of the baseline assessment of M&E capacity in the 38 districts identified as performing poorly on MOH indicators. One-half of the districts (50.0%) had a biostatistician, and only slightly more (60.5%) had an HMIS focal person. Very few of the districts (37.4%) had staff members who were fully trained in any of the primary areas necessary for M&E, although a large proportion (81.6%) had staff members who had been trained in how to complete HMIS tools. Only 15.8% of districts had an M&E plan in place, and only 26.3% reported allocating a specific portion of their budget to M&E activities. Slightly more than half (55.5%) were using data for planning or improving service delivery. All 38 districts had dedicated and functional computers for entering data into the DHIS2 national system. Districts indicated that the main challenge they faced was lack of training in data management, analysis and use.

Table 1.

Baseline assessment of M&E capacity of 38 districts in Uganda, 2013-2014

Assessment Area	Number of districts (N)	Percentage (%)
M&E staff currently employed
Biostastician	19	50.0
HMIS focal person	23	60.5
Surveillance focal person	30	78.9
M&E staff trained in the following
Basic M&E	8	21.1
Data management	11	28.9
Data analysis	6	15.8
Data use	15	39.5
HMIS*	31	81.6
M&E staff have documented roles	25	65.8
District M&E plans exists	6	15.8
Implementing Partner provides M&E Support	38	100.0
Data are used for planning and improvements	21	55.3
SOP* for data management exists	17	44.7
Dedicated computer for data management exists	38	100.0
M&E related challenges reported
Insufficient number of staff	15	39.5
Limited or no budget to support M&E	31	81.6
Insufficient storage space for records	12	31.6
Lack of data analysis capacity	24	63.2
Limited or no office space for M&E staff	12	31.6
Poor Internet connectivity	13	34.2
Staff lacking training in data management and use	30	79.0

^*HMIS: Health management information system; SOP: Standard operating procedure

Differences between the baseline and follow-up assessments in the 10 districts that received intensive training and support are shown in Table 2. At follow-up, more districts reported having staff trained in key aspects of M&E, likely a result of the META training. Despite the training, however, only six out of 10 felt their staff members were adequately trained in data analysis. At follow-up, more districts reported that roles of M&E staff were clear and that they had developed M&E plans and budgets. Many facilities began generating and posting bar charts and line graphs of the HIV-related program indicators in which they were most interested; many also developed and posted annual targets for service delivery. This enabled facilities and districts to improve service delivery based on the data. For example, some districts realized that the proportion of HIV-infected clients who were initiated on ART was low, and as a result they worked with facilities to ensure that they initiated eligible patients on treatment promptly. Other districts realized they had not been monitoring adherence to ART and that strategies to improve adherence to medications and to clinic visits needed to be put into place. In addition, screening of HIV patients for tuberculosis was occurring infrequently; supervision and training about the importance of screening resulted in improved tuberculosis case finding as well as treatment outcomes.

Table 2.

Baseline (2013) and post-intervention follow-up (2015) assessment of M&E capacity in 10 districts, Uganda

Assessment Area	Number of districts (N)
	Baseline	Follow-up
Presence of key M&E staff
Biostatistician	5	9
HMIS focal person	9	9
Surveillance focal person	10	10
District staff trained in M&E staff
Basic M&E	3	10
Data management	4	10
Data analysis	2	6
Data use	3	10
HMIS*	6	10
M&E staff have documented roles	4	9
District M&E plans exists	3	6
Specific budgets for M&E activities	2	7
Implementing Partner provides M&E Support	10	10
Data are used for planning and improvements	4	8
SOP* for data management exists	4	9
Dedicated computer for data management exists	10	10
M&E related challenges reported - limited funds	6	7

^*HMIS: Health management information system; SOP: Standard operating procedure

Among the 100 facilities that were supported with mentorship, 49% had sent prior monthly and quarterly reports on time to their districts during the initial follow-up period compared to 93% after on-site mentorship (data not shown). Action plans developed by health facilities are listed in Table 3. Key challenges mainly revolved around data completeness, timeliness and quality. Many facilities experienced regular stock-outs of standard data collection and reporting tools and had limited knowledge of how to complete them; poor understanding of how to perform data analysis was also identified as a problem.

Table 3.

Action plans developed by facilities and progress achieved.

(N=number of facilities out of 100 who identified the issue as important)

Issue identified	Action taken	Progress
Late submission of data reports to the district; insufficient number of staff compiling reports (N=51)	Staff from each department trained to summarize data	More than 90% of health facilities submitted reports on time
Data collection tools incomplete; registers not updated on a monthly basis and staff not familiar with how to complete them (N=88)	Training provided in how to complete registers. Supervisors required to check tools and registers on a regular basis.	Based on mentorship team’s evaluation of data quality, registers were completed and accurate; quality of reports to district improved as a result
Lack of QI committee to evaluate data and interpret it for use in programming (N=79)	Mentorship teams worked to set up QI committee; training in how to review specific indicators to evaluate service provision	Projects developed to improve programming weaknesses, such as ART adherence assessments, TB screening of HIV patients, and ongoing review of TB and HIV records to improve TB case-finding
Limited data analysis and use (N=84)	Hold facility data review meetings quarterly; performance graphs developed after data analysis	Graphs of MOH performance indicators compared to MOH targets displayed in areas where the staff and the public could see them*
Poor record keeping and use of outdated data tools (N=65)	Organize space for record keeping and discard outdated registers; provision of updated tools by META and MOH	Improved filling of clients’ files in the HIV clinics so they could be retrieved; data collected on updated forms
Lack of knowledge about data management by some health care workers (N=92)	Oriented health facility staff in data management and reporting with support from META staff	Data requests from different departments increased; different departments became involved in compiling reports
Limited or no health facility Supervision of records assistants; high levels of absenteeism (N=95)	Staff in charge of facilities began regular supervision; data committees with representatives from departments were created, and performed monthly reviews of data quality	Reduced absenteeism of records assistants and increased involvement by individual departments in data management
Inaccurate data reported to the district and thus also to the MOH (N=74)	Conduct data quality assessments routinely to check accuracy and completeness of data	Improvement in accuracy of reporting for key HIV indicators.

^*MOH-specified indicators that were required to be monitored included immunization coverage, numbers and proportions of HIV infected clients enrolled in care, ante-natal clinic attendance of pregnant women, among others.

DISCUSSION

We have described a process for building the capacity of District Health Teams in Uganda to monitor their own HIV programs. A follow-up evaluation of the 10 districts that were targeted for training and mentorship indicated that the program’s key achievements were an increase in the number of districts that reported being able to use data, that developed an M&E plan and that had specified roles for their staff. In addition, more districts had earmarked a portion of their budgets for M&E activities, although three of the 10 continued to feel the amount of funding was inadequate. Specified and adequate funding are critical to the functioning of district M&E because these monies pay for transportation to facilities so staff can provide supervision and review the quality and completeness of data. Among a new cadre of M&E officers in Botswana, 65% of those who were surveyed said that lack of monies for transportation was a major impediment to providing oversight (Ledikwe et al., 2013).

We found that the primary barrier to district level M&E was the limited skill level of staff, not only within the district M&E team but also at facilities. The scarcity of trained human resources has been widely acknowledged as a major barrier to the effective use of public health data in less-resourced countries (Hongoro et al., 2004; Kimaro et al., 2005; Makombe et al., 2008; Mpofu et al., 2014). Various approaches have been implemented to address this problem including task shifting, on-the-job training, and mentoring (Fulton et al., 2011; Lehmann et al., 2009). We found mentoring and on-the-job-supervision to be helpful in improving district and facility-level data quality and use despite a Cochrane review stating that strong evidence about the utility of such an approach was lacking (Rockers, P., 2013). Other efforts in Uganda to improve district leadership are ongoing including a two-year leadership in public health fellowship program open to District Health Officers and sponsored by the Makerere University School of Public Health (Matovu et al., 2011).

Recruiting and retaining skilled staff to work in district health offices, which are often remote, can be difficult; once staff members are trained, they often leave for better-compensated positions. Even though M&E staff positions are specified within the Uganda health care sector, the income of district health workers is low, and motivation can lag when little or no supervision or encouragement come from the national level. Providing ongoing courses for professional development and support from higher levels of public health would be helpful.

Many of the same difficulties with staffing, skills and training that exist at the district and facility level in Uganda are also present at the national level. Very rarely, if at all, does information return to the district health offices from national evaluation of data, or are supervision and support provided (Mbonye et al., 2013). As a result, we initially found that few districts had any interest in evaluating or using their data, not only because of their limited ability to do so but also because its collection was mainly viewed as fulfilling a reporting requirement that had little local relevance. The lack of direct oversight by higher levels of health administration was evident by the widespread unavailability of updated HMIS tools. In 2014, all Ugandan HMIS forms were updated, but distribution was sporadic, and little training in their use was provided. None of the 100 facilities that we evaluated had updated HMIS forms, and even within the same district, facilities used different versions. To help remedy this situation, METS is now in charge of printing and distributing updated HMIS forms to all 112 districts of the country and also of providing training in their use.

Uganda is currently rolling out an electronic medical record system, OpenMRS, and as of mid-2017, nearly 1000 facilities were equipped with it. OpenMRS can be used to report directly to the national DHIS2 system, to report electronically to the district level, and to perform automatic facility-level analysis. Use of electronic data capture has been shown to improve data quality and improve the efficiency of reporting by reducing the burden of completing paper-based tools, all of which can lead to improved data use (Amoroso et al., 2010; Ekouevi et al., 2011; Garrib et al., 2008). Even with electronic data, however, regular data quality assessments need to be performed to ensure the accuracy of information collected (Mphatswe et al., 2012; Xiao et al., 2017). It is expected that the use of OpenMRS in Uganda will facilitate regular data analysis, particularly of cohorts used to determine retention on ART, which is one of the primary measures of HIV program effectiveness.

Our model was designed to promote local district ownership and use of data and to enhance supervision of facilities. To ensure that data use was valued, we found that first engaging the political and administrative leadership of the district who have purview over the district health office was critical. Our strategy also involved identifying mentors chosen by the districts who could help facility staff to perform basic M&E. However, we found that the majority of the mentors themselves lacked a basic understanding of M&E and so were unable to help others. As a result, we have changed our model and now first train mentors in M&E principles and in how to provide effective oversight and support. Different methods of training and supporting district-level staff to improve the quality and use of health care data in decision-making in less-resourced settings have been tried, but which methods are most effective has not been systematically studied (Nutley et al., 2013; Vasan et al., 2017). Some evaluations have shown that on-the-job mentorship of clinicians and district staff can improve data quality and use (Edwards et al. 2015; Workneh et al., 2013). Overall, strong management, leadership, and supervision at multiple levels are likely to be important in enhancing M&E.

Our study had a number of limitations. The intervention focused largely on data quality rather than its interpretation and use; as a result, districts continued to feel they lacked skills in data analysis. Although we have provided examples of how districts and facilities changed programming in response to better monitoring, we did not collect quantitative or systematic measures of improvements in programming that would help determine the effectiveness of the district-level intervention. Secondly, the initiation of mentorship at facilities was often delayed because of competing activities by the districts. These delays may have contributed to a decline in enthusiasm among staff, who may also have forgotten some of the training they received. In addition, the tools we used to assess M&E capacity were limited in scope and were not designed to obtain detailed information about barriers to M&E and individuals’ responses to the mentorship. We present findings on only 10 targeted districts. Since completing the intervention described here, however, we have provided support to the remaining 28 districts initially identified as having poor performance on MOH indicators. Finally, we do not have data on sustainability and long-term outcomes.

Although most districts in Uganda experience challenges with staffing levels and skills, simple but tailored interventions can lead to improvements in M&E capacity. The strategy of enabling the district health team to provide facility-level support and oversight can be an effective method for strengthening M&E systems and improving data quality.