The feasibility of automating audit and feedback for ART guideline adherence in Malawi

Abstract

Objective

To determine the feasibility of using electronic medical record (EMR) data to provide audit and feedback of antiretroviral therapy (ART) clinical guideline adherence to healthcare workers (HCWs) in Malawi.

Materials and methods

We evaluated recommendations from Malawi's ART guidelines using GuideLine Implementability Appraisal criteria. Recommendations that passed selected criteria were converted into ratio-based performance measures. We queried representative EMR data to determine the feasibility of generating feedback for each performance measure, summed clinical encounters representing each performance measure's denominator, and then measured the distribution of encounter frequency for individual HCWs across nurse and clinical officer groups.

Results

We analyzed 423 831 encounters in the EMR data and generated automated feedback for 21 recommendations (12%) from Malawi's ART guidelines. We identified 11 nurse recommendations and eight clinical officer recommendations. Individual nurses and clinical officers had an average of 45 and 59 encounters per month, per recommendation, respectively. Another 37 recommendations (21%) would support audit and feedback if additional routine EMR data are captured and temporal constraints are modeled.

Discussion

It appears feasible to implement automated guideline adherence feedback that could potentially improve HCW performance and supervision. Feedback reports may support workplace learning by increasing HCWs' opportunities to reflect on their performance.

Conclusion

A moderate number of recommendations from Malawi's ART guidelines can be used to generate automated guideline adherence feedback using existing EMR data. Further study is needed to determine the receptivity of HCWs to peer comparison feedback and barriers to implementation of automated audit and feedback in low-resource settings.

Objective

The objective of this study is to measure the degree to which guideline adherence feedback could be automatically generated using electronic medical records (EMRs) and a national guideline for delivery of antiretroviral therapy (ART) in Malawi, Africa. We also sought to characterize and measure the factors that limit automated feedback in this setting.

Background and significance

Healthcare workers (HCWs) are defined as ‘all people engaged in actions whose primary intent is to enhance health.’¹ HCW performance in low-resource settings is frequently below the standards of care recommended in clinical practice guidelines (CPGs), leading to millions of unnecessary deaths each year.² Clinical performance, defined as adherence to a CPG, depends upon appropriate training and continuous learning. Training interventions to improve HCW performance are based on CPGs that standardize the delivery of healthcare. Once training is complete, HCWs in low-resource settings have limited opportunities to improve their knowledge through continuous learning by, for example, receiving feedback that reinforces new guideline-based knowledge. A major barrier to the provision of feedback in these settings is human resource shortages. However, the increased use of electronic health information systems is creating new opportunities to generate automated feedback to support learning, potentially improving HCW performance. In this study, we evaluated the feasibility of measuring HCW performance and generating guideline-based feedback in a low-resource setting using EMR data and a CPG.

Imbalance in the global health workforce and task-shifting

The global distribution of HCWs is skewed away from low-income countries that hold the greatest proportion of the global disease burden, resulting in a critical shortage of HCWs.¹ For example, the WHO estimates that the disease burden in African countries represents 24% of the global disease burden, while the region has approximately 3% of the world's HCWs.³ A primary strategy to improve the quality of healthcare in the midst of human resource shortage in low-resource settings is through task-shifting. Healthcare organizations engage in task-shifting activities by training less-specialized HCWs to perform the work of more specialized HCWs. In a typical task-shifted scenario, tasks performed by physicians such as initial clinical evaluation and prescribing medication are transferred to clinical officers (also called medical officers) who are mid-level clinicians with 3–4 years of post-secondary medical training. Nurses are assigned the tasks traditionally performed by clinical officers, such as prescription refills and consultation for stable patients. A new class of health assistant called a peer educator is created to perform the nursing tasks that require the least amount of professional training.⁴ One of the primary activities in any task-shifting intervention is the development of a CPG for use by task-shifted HCWs.⁵

Clinical practice guidelines in low-resource settings

The Institute of Medicine defines CPGs as ‘systematically developed statements to assist practitioner and patient decisions about appropriate healthcare for specific clinical circumstances.’⁶ In a task-shifted context, public health organizations create CPGs to assist HCWs to target the highest causes of patient disability and mortality, leading to a higher sensitivity in diagnostic processes at the cost of a lower specificity.⁷ An example of a CPG developed for use in a task-shifted healthcare environment is Treatment of AIDS: Guidelines for the Use of Antiretroviral Therapy in Malawi developed by Malawi's Ministry of Health.⁸ Treatment of AIDS, a 114-page document, provides a comprehensive set of clinical recommendations for the management of ART that the Ministry of Health mandates all HCWs to follow. Representations that allow practitioners with limited training to effectively manage the most prevalent manifestations of disease are essential for national treatment programs like those in Malawi. For example, nurses in ART clinics in Malawi use a checklist to determine if a patient is stable enough to continue their ART medication, or if the patient needs to be evaluated by a clinical officer to re-assess treatment (table 1).

Table 1.

Checklist of symptoms for patients attending the clinic from Malawi's national guidelines for the treatment of AIDS⁸

Did you experience any new or worsening symptoms since your last visit such as:
Fever	Yes	No
Abdominal pain	Yes	No
Vomiting	Yes	No
Diarrhoea	Yes	No
Weight loss	Yes	No
Rash	Yes	No
Pain or numbness in your legs	Yes	No
Cough	Yes	No
Yellow eyes	Yes	No
Any unwanted changes in body shape	Yes	No
Any other new symptoms	Yes	No

If any of the symptoms are recorded as Yes, then refer to a clinician. If all symptoms are recorded as No then the patient can be dispensed antiretroviral drugs.

A major challenge to the operationalization of CPGs is that HCWs are unlikely to use them in daily practice.⁹ Barriers to guideline adherence in low-resource settings are varied and include disagreement with recommendations in a guideline as well as lack of incentive.¹⁰ Additionally, HCWs do not receive adequate feedback about adherence to CPGs, according to recent studies in African countries that identify improved supervision and feedback as an important incentive for HCW motivation.^{11 12} A widely-used intervention to improve guideline adherence in low-resource settings is audit and feedback.^13–15

Audit and feedback

Audit and feedback is the process of summarizing clinical performance and providing feedback to HCWs. During the audit process, auditors collect data either prospectively using paper-based or electronic documentation, or retrospectively by analyzing paper-based medical charts, medical registries, or EMR data. Feedback is delivered to HCWs outside of the clinical setting as a written, verbal, or electronic performance summary over a pre-determined time period. Performance summaries used in studies of audit and feedback can range in duration from 1 week to 6 months or more. CPG implementers have increasingly used audit and feedback in multi-faceted implementation strategies to change provider behavior that may also include educational meetings, peer feedback, or electronic reminders and alerts.¹⁶ In low-resource settings, audit and feedback has been extensively used without the use of an EMR.^{17 18}

HCWs require feedback to gain insight into their understanding of guidelines and thereby improve and maintain the quality of care, but not all forms of feedback are sufficient for enabling improved performance.^19–21 Systematic literature reviews have found that audit and feedback can improve adherence to recommended clinical practice with small to moderate effects.^{16 22} However, taken as a whole, studies of audit and feedback diminish its potential to impact clinician performance by overlooking features of feedback delivery that can improve its effectiveness.²³ Actionable feedback, defined as feedback that is timely, individualized, non-punitive, and customizable, can significantly improve the effect of feedback on performance.²⁴ Furthermore, when baseline adherence is low, as is common in low-resource settings, audit and feedback is likely to be more effective.¹⁶

Audit and feedback performed manually can improve HCW performance in low-resource settings, but it requires allocation of human resources already in critical shortage. Automated audit and feedback generated using electronic data presents an opportunity to leverage existing resources to improve HCW performance without requiring additional effort from HCWs to collect or record performance data.

Electronic health information systems in low-resource settings

The presence of electronic health information systems in developing countries is expanding. A primary motivating factor for the development and implementation of electronic health information systems is to improve efficiency of data management at the patient and population levels. Disease epidemics in sub-Saharan Africa like HIV and multi-drug resistant tuberculosis have elicited large-scale public health campaigns that introduce clinical information systems to manage data for monitoring treatment outcomes and forecasting drug demand.^25–27 Public health data that are aggregated from patient-level records are gaining recognition as an efficient approach to monitoring and evaluating disease treatment programs. Other factors contributing to the growth of electronic health information systems are their potential to improve the quality of healthcare and support expansion of health services to a national scale in developing countries.^{28 29}

Baobab Health Trust and the Malawi Ministry of Health have implemented the Baobab Anti-Retroviral Therapy (BART) EMR in 10 ART clinics in the central and southern region of Malawi.²⁵ The BART EMR was designed to collect a minimal dataset that supports monitoring and evaluation of Malawi's national ART program. HCWs, including doctors, clinical officers, and nurses, use BART to enter routine patient encounter data such as history of present illness, vital signs, symptoms, and prescription information at the point of care. At the end of December, 2010, there were 93 883 patients registered in the ART system of whom 52 906 were alive and on ART.

This study explores the feasibility of using EMR data and a CPG to generate performance feedback for HCWs in Malawi, in support of guideline-based training on the treatment of AIDS. The objectives of this research are (1) to determine how many recommendations from Treatment of AIDS can be used to generate performance feedback in ART clinics where an EMR is used routinely at the point of care, (2) to measure the distribution of feedback across HCWs, and (3) to describe the characteristics of available EMR data that could be re-used to generate performance feedback for HCWs in a low-resource setting.

Materials and methods

We created a CPG and EMR evaluation process to determine the feasibility of generating HCW performance feedback. For this evaluation we used only CPG statements that directly addressed clinical management of adult ART patients from Treatment of AIDS. We began by identifying candidate recommendations from the CPG. We then identified each candidate recommendation's components and evaluated them using criteria for decidability, executability, measurability, computability, and auditability. Each stage of the process identified a subset of the preceding stage to arrive at a final set of recommendations that were suitable for generating feedback using EMR data from Malawi (figure 1).

Figure 1.

Methods for identification of candidate, decidable, executable, measurable, computable, and auditable recommendations.

Identification of candidate recommendations using the Guideline Elements Model (GEM)

Candidate recommendations are any statements from the guideline that describe one or more recommended actions to be taken under conditional clinical circumstances. To represent medical knowledge contained in candidate recommendations we used the Guideline Elements Model (GEM), a document-based knowledge representation model for CPGs.³⁰ We selected GEM for its ability to preserve direct linkages between the CPG text from Treatment of AIDS, which the Ministry of Health mandates all HCWs to follow, and the resulting feedback. We used GEM Cutter II, an XML editor that facilitates the mark-up of guideline text to structure recommendations and their components within the GEM hierarchy. For each recommendation, we identified the following components: decision variable, value of decision variable, and action (figure 1).

Decidable, executable, and measurable recommendations

To evaluate candidate recommendations for decidability, executability, and measurability, we used a subset of the criteria from the GuideLine Implementability Appraisal (GLIA) v 1.0 instrument (box 1).³¹ Two judges, one physician (MG) and one clinical officer (RD), independently judged the candidate recommendations using GLIA criteria by scoring them as ‘Yes,’ ‘No,’ or ‘Unknown.’ Where the judges gave discordant scores, they discussed each recommendation to arrive at a final agreed score. Recommendations having any criteria answered as ‘No’ or ‘Unknown’ by both judges were designated as not being decidable, executable, or measurable, and the corresponding barrier(s) to satisfying the criteria were documented. Recommendations having all criteria answered as ‘Yes’ by both judges were assessed for computability.

Box 1. Selected criteria from the GuideLine Implementability Appraisal (GLIA) instrument³¹.

Decidability (precisely under what circumstances to do something)

10. If there are more than one condition in the recommendation, is the logical relationship among all conditions (ANDs and ORs) clear?

Executability (exactly what to do under the circumstances defined)

11. Is the recommended action (what to do) stated specifically and unambiguously?

Measurable outcomes (the degree to which the guideline identifies markers or endpoints to track the effects of implementation of this recommendation)

17. Can criteria be extracted from the guideline that will permit measurement of adherence to this recommendation?

Computability (the ease with which a recommendation can be operationalized in an electronic information system)

28. Are all patient data needed for this recommendation available electronically in the system in which it is to be implemented?

29. Is each condition of the recommendation defined at a level of specificity suitable for electronic implementation?

30. Is each recommended action defined at a level of specificity suitable for electronic implementation?

31. Is it clear by what means a recommended action can be executed in an electronic setting, for example, creating a prescription, medical order, or referral, creating an electronic mail notification, or displaying a dialog box?

Computable recommendations

To assess the computability of decidable, executable, and measurable recommendations, we used the GLIA criteria for computability (box 1).³¹ The first author, a developer of the EMR in Malawi, scored each recommendation as ‘Yes,’ ‘No,’ or ‘Unknown.’ We resolved recommendations that had any criteria answered as ‘Unknown’ by reviewing the recommendation components with another EMR developer (OJG). We designated all recommendations that had all criteria answered as ‘Yes’ as computable. Recommendations that did not meet computability criteria were designated as uncomputable, and the corresponding barrier to satisfying the criteria was documented.

In answering the computability criterion #28, ‘Are all patient data needed for this recommendation available electronically in the system in which it is to be implemented?,’ we documented the required data elements from the EMR that represented each recommendation's components. We interpreted GLIA criterion #28 to include only data elements that the EMR is potentially capable of collecting, and distinguish these from the smaller set of data elements that are collected in practice. We created a criterion for ‘auditability’ that specifically measures the set of data elements used in practice in the clinics.

Auditable recommendations

Auditability is defined as the availability of representative EMR data for each component of a recommendation. To assess the auditability of each computable recommendation, we created a ratio-based performance measure using the recommendation's components. The performance measure's denominator is the number of clinical encounters where specific clinical circumstances were met (the condition). The performance measure's numerator is the number of clinical encounters where specific clinical circumstances were met (the condition) and the recommended action was taken.

$$\text{Performance}=\frac{\text{Condition(s) and Action}}{\text{Condition(s)}}$$

We selected the clinical encounter instead of the HCW's patients as the unit of measure because of a lack of continuity of care that permits a patient to present the same problems to multiple HCWs over a series of encounters. We created an SQL query representing the denominator of the performance measure for all HCWs in the EMR data. We designated recommendations whose associated denominator query returned zero encounters as being not auditable. For each associated denominator query that returned one or more encounters, we queried the EMR data representing the numerator for an individual HCW to create a proof-of-concept feedback report for the recommendation. We designated recommendations having an associated numerator query returning zero encounters as being not auditable at the cost of excluding any recommendations that were in fact auditable but were universally not adhered to by HCWs during the data collection period. We designated recommendations that had one or more encounters representing the denominator and one or more encounters representing the numerator of the performance measure as being auditable.

Distribution of encounter frequency for individual HCWs

For each auditable recommendation, we measured the distribution of average monthly encounters for individual HCWs. We measured average monthly encounters over the entire period of available EMR data for all HCWs who used the EMR for more than 1 month. We calculated frequency of encounters at a monthly interval based on Jamtvedt et al's designation of a monthly interval as a moderate frequency.¹⁶ We analyzed the frequency of recommendations to identify recommendations for which we could more adequately assess HCW adherence, as more frequently occurring recommendations can provide a more reliable indication of HCWs' adherence. However, frequency is not necessarily the only determinant of significance of a recommendation because some rare events may be highly significant.

Results

EMR data

We analyzed de-identified, structured EMR data from four ART clinics in Malawi over a 2½-year period from April, 2008 to October, 2010, when the ART guidelines were in effect. During the 2½-year period a total of 117 HCWs (62 clinical officers, 55 nurses) used the EMR at the point of care to create patient records for 27 528 individual patients. The HCWs recorded a combined 423 831 encounters.

Identification and assessment of recommendations

We identified 174 candidate recommendations from Treatment of AIDS. Of these, 152 recommendations met the GLIA criteria for decidability, executability, and measurable outcomes. Candidate recommendations that failed to meet the criteria for decidability often related to possible differential diagnoses. An example is the recommendation ‘Pancreatitis should be suspected if the patient develops severe upper abdominal pain, nausea and vomiting.’ This statement failed the criteria for having a measurable outcome because of the difficulty of measuring HCW suspicion.

Out of the 152 recommendations that were decidable and executable with measurable outcomes, 58 recommendations met the GLIA criteria for computability. From the 58 computable recommendations, 22 did not have associated EMR data resulting from HCWs' routine use of the system. Another 15 of the computable recommendations were not assessed for auditability due to complexity in representing temporal constraints. Of the 58 computable recommendations, 21 met the criteria for auditability (figure 2; totals for the recommendations not meeting GLIA criteria are not mutually exclusive and do not sum up to the n displayed in each box). Proof-of-concept feedback for the 21 auditable recommendations, using EMR data from the month of April, 2010, for one nurse and one clinical officer are displayed in table 2.

Figure 2.

Identification and assessment of recommendations.

Table 2.

Auditable recommendations and proof-of-concept adherence feedback for one ART clinic nurse and one clinical officer from the month of April, 2010

Auditable recommendation	Sample adherence for April, 2010
	Individual		Clinic
	%	Ratio	%	Ratio
Nurse
1. If any symptoms are recorded as YES, then refer to a clinician	33.3	28/84	40.4	415/1027
2. If fever is recorded as YES, then refer to a clinician	–	0/0	62.5	40/64
3. If abdominal pain is recorded as YES, then refer to a clinician	75.0	3/4	46.2	43/93
4. If vomiting is recorded as YES, then refer to a clinician	–	0/0	83.3	15/18
5. If diarrhoea is recorded as YES, then refer to a clinician	33.3	1/3	32.4	12/37
6. If weight loss is recorded as YES, then refer to a clinician	–	0/0	85.7	6/7
7. If rash is recorded as YES, then refer to a clinician	11.1	1/9	60.5	49/81
8. If leg pain or numbness in your legs is recorded as YES, then refer to a clinician	25.0	8/32	37.2	105/282
9. If cough is recorded as YES, then refer to a clinician	21.1	4/19	29.1	60/206
10. If yellow eyes is recorded as YES, then refer to a clinician	–	0/0	–	0/0
11. If any unwanted changes in body shape is recorded as YES, then refer to a clinician	–	0/0	100	10/10
12. If any other new symptom is … YES, then refer to a clinician	58.8	20/34	46.5	276/593
13. If all symptoms are recorded as NO then the patient can be dispensed ARVs	92.2	329/357	84.7	2757/3256
Clinical officer
14. CPT-eligible patients on CPT	77.3	92/119	89.6	1206/1346
15. Adult CPT prescription: one tablet (480 mg) twice a day	100	92/92	100	1206/1206
16. HIV-positive TB patients will start on cotrimoxazole according to the current CPT policy	16.7	1/6	17.8	8/45
17. Stavudine should not be combined with zidovudine (AZT) due to pharmacologic antagonism	100	85/85	100	1114/1114
18. All adults will now receive the stavudine-30 mg regimen	100	85/85	100	1114/1114
19. The drug (lamivudine) should never be given as monotherapy	100	92/92	100	1236/1236
20. Patients with acute hepatitis (manifested by jaundice) should not be given d4T/3TC/NVP	–	0/0	–	0/0
21. In the case of jaundice or high suspicion of hepatitis with impending liver failure, d4T/3TC/NVP should be stopped	–	0/0	–	0/0

ART, antiretroviral therapy; ARVs, antiretroviral drugs; CPT, cotrimoxazole preventive therapy.

Distribution of encounter frequency for individual HCWs

The mean frequency for all nurses' auditable recommendations was 45.13, ranging from 0.03 to 580.44 encounters per month. The mean frequency for all clinical officers' auditable recommendations was 58.83, ranging from 0.37 to 341.75 encounters per month. Figure 3 shows the distribution of average encounter frequencies across auditable recommendations. Thirteen of the 21 recommendations (#1–3, 7–9, 12–15, and 17–19) have a mean frequency above four encounters per month, per HCW, making them suitable for providing individualized feedback for at least one encounter per week on average.

Figure 3.

Distribution of average monthly encounters for 21 auditable recommendations listed in table 2.

Discussion

The results of this feasibility study show that 21 (12%) of the recommendations in Malawi's ART guidelines can be audited using EMR data to generate automated performance feedback for an average of 45 clinical encounters per month, per individual HCW. An additional 37 (21%) of the ART recommendations could support audit and feedback without requiring alteration of the guideline text or EMR system design. To enable feedback in these cases, 15 out of 37 computable recommendations must be made auditable using representations of temporal constraints to measure HCW performance. The use of temporal reasoning algorithms would allow an automated feedback system to measure adherence to recommendations that are contingent on one or more prior treatment periods, or actions that must be executed within a given time period. For example, a recommendation may indicate that a higher-dose prescription should be given following a 1-month treatment of the same drug with a lower-dose prescription. The remaining 22 of 37 computable recommendations could not be audited because the data required to represent either the denominator or the numerator were not captured as a part of routine system use. For example, some non-ART prescriptions are routinely recorded on paper, excluding them from electronic audit.

Although 21 recommendations represent a small percentage of Malawi's ART guidelines, performance summaries from the set of recommendations could potentially have a large effect on HCW performance by increasing HCWs' opportunities to reflect on their individual and group performance. Performance summaries of the nurses' referral checklist (table 1) could provide nurses with new insight into their individual referral and treatment patterns of patients. This feedback can reveal gaps in nurses' understanding of the guideline or differences in beliefs about the efficacy of referral for each symptom. One important symptom is unexplained weight loss, which is associated with early mortality in ART treatment.³² Another example is patients' complaints of leg pain or numbness indicating peripheral neuropathy, which is highly prevalent but under-diagnosed in Malawi.³³ Clinical officers' performance summaries could similarly reveal prescribing patterns for cotrimoxazole preventive therapy, a priority for AIDS care that can significantly reduce mortality.³⁴

For ART clinic supervisors, performance summaries can potentially be used to enhance supportive supervision for HCWs. Clinic-level performance summaries have the potential benefit of facilitating group discussion about HCWs' knowledge of and attitudes toward the guideline. For individual HCWs, performance summaries may assist supervisors in targeting gaps in an HCW's understanding of the guideline. The availability of routine performance data can potentially assist ART clinic supervisors in identifying changing practice patterns over time.

Guideline developers in Malawi may benefit from automated audit and feedback data as a form of feedback about the utility of the guideline itself. Using aggregate performance summaries, guideline authors may be able to identify adherence barriers where adherence is uniformly low, or where contrasting levels of adherence occur between HCW groups across ART clinics. We anticipate that the implementation of an automated audit and feedback system will contribute value to the process of delivering ART that may in turn increase the demand for feedback data. Providing peer comparison feedback can itself potentially serve as a powerful motivator for HCWs to increase their use of the EMR, enabling a higher percentage of auditable recommendations. Thus there are potential incentives for both EMR developers and guideline authors to develop guidelines and EMR systems to support the delivery of automated audit and feedback.

A viable means for increasing the number of auditable recommendations would be a collaborative process for guideline authors and EMR developers to identify the most significant ART recommendations and the EMR data elements that can be collected to represent them using the methods outlined in this study. Using such a process, coordinated development of future versions of the guideline could align ART guideline implementation with automated audit and feedback that targets HCWs' lack of familiarity with guideline revisions to accelerate the uptake of new recommendations.

Limitations

We excluded recommendations with zero encounters in the performance measure's numerator from the set of auditable recommendations, (eg, recommendations that no health worker was found to have adhered to over the entire 2½-year period of data analyzed) because of the difficulty in distinguishing between true non-adherence and non-use of the EMR by HCWs. Therefore we may have underestimated the number of auditable recommendations. Another limitation of our approach is that the feasibility of providing automated performance summaries to HCWs is dependent upon social, cultural, and environmental constraints that were not evaluated in this study. Future work will evaluate the acceptability of routine performance feedback by HCWs in Malawi to better understand the barriers to providing automated audit and feedback in this setting.

Conclusion

A moderate number of recommendations from Malawi's ART guidelines can be used to generate automated guideline adherence feedback using existing EMR data. Coordinated development of guidelines and EMR systems in Malawi has the potential to increase the feedback that could be generated. Further study is needed to determine the receptivity of HCWs to peer comparison feedback and other barriers to the implementation of automated audit and feedback in low-resource settings.