Computer-Supported Feedback Message Tailoring for Healthcare Providers in Malawi: Proof-of-Concept

Abstract

Although performance feedback has the potential to help clinicians improve the quality and safety of care, healthcare organizations generally lack knowledge about how this guidance is best provided. In low-resource settings, tools for theory-informed feedback tailoring may enhance limited clinical supervision resources. Our objectives were to establish proof-of-concept for computer-supported feedback message tailoring in Malawi, Africa. We conducted this research in five stages: clinical performance measurement, modeling the influence of feedback on antiretroviral therapy (ART) performance, creating a rule-based message tailoring process, generating tailored messages for recipients, and finally analysis of performance and message tailoring data. We retrospectively generated tailored messages for 7,448 monthly performance reports from 11 ART clinics. We found that tailored feedback could be routinely generated for four guideline-based performance indicators, with 35% of reports having messages prioritized to optimize the effect of feedback. This research establishes proof-of-concept for a novel approach to improving the use of clinical performance feedback in low-resource settings and suggests possible directions for prospective evaluations comparing alternative designs of feedback messages.

Introduction

Globally there are significant gaps between best practices drawn from medical evidence and decisions made by healthcare professionals^1–3. Closing these gaps is increasingly difficult because of accelerating rates of the production of biomedical knowledge and the increasing complexity of healthcare systems⁴. The urgency of overcoming these challenges has motivated the development of a Learning Health System (LHS), a cyber-social infrastructure that enables both learning from data and the creation of feedback loops to improve the delivery of patient-centered care⁵.

Audit and feedback (AF), defined as the provision of clinical performance summaries to healthcare providers, teams, and organizations, is widely used to support learning and behavior change for healthcare quality improvement⁶. A recent Cochrane review including 140 clinical trials shows that AF can significantly improve compliance with desired practice, but that it is unclear how and when it is effective⁶. Seeking to improve the utility of AF evidence, researchers have called for a shift from overall effectiveness studies towards comparative effectiveness studies, evaluating how and when AF intervention components will work⁷.

To better understand AF components, researchers have argued for the explicit use of psychological theory that explains how AF operates to change behavior and support clinical learning^8,9. Frameworks that enable the use of theory in behavior change interventions include the Theoretical Domains Framework¹⁰ (TDF) and the Capability, Opportunity, Motivation and Behavior (COM-B) system¹¹. A mapping between the TDF and COM-B¹⁰ enables researchers to model determinants of behavior within COM-B, and to relate these determinants to a broader set of causal mechanisms within theoretical constructs in the TDF that may hold implications for the effective delivery of performance feedback in organizations.

We have proposed that a theory-informed feedback tailoring tool used by clinical supervisors could improve AF for individual healthcare provider recipients¹². Computer-supported feedback tailoring requires clinical supervisors to use their familiarity with the recipient and situation to select an optimal feedback message from a menu of potentially relevant messages. This approach may be especially useful in low-resource settings where factors such as staff turnover, disruptions to care, and minimal dataset collection create a need to handle high uncertainty and missing information in the analysis of clinical data¹³. Understanding the design space for individually-tailored feedback messages is a key step toward designing and implementing such a system. Our objectives for this research were to analyze clinical performance data to understand the requirements and potential impact of computer-supported feedback message tailoring in a low-resource setting.

Methods

To establish proof-of-concept for using computer-supported feedback message tailoring in low-resource settings, we used a five-stage process (Figure 1). First, we measured clinical performance using four guideline-based indicators. Second, we developed a preliminary model of the effect of feedback on performance for a range of barriers to performance improvement in antiretroviral therapy (ART) clinics. Third, we created a rule-based message tailoring process that used the model of the effect of feedback in our specific context. Fourth, we used the message tailoring process to retrospectively generate menus of tailored messages about each individual healthcare provider within the performance dataset. Finally, we analyzed the performance dataset and the resulting tailored message data to understand the potential impact for computer-supported feedback message tailoring in this context.

Figure 1.

Study design to establish proof-of-concept for a computer-supported feedback message tailoring system

Setting and data collection

We evaluated our approach in Malawi, where healthcare providers use a national electronic medical record system (EMR) in the provision of ART in public hospitals¹⁴. Malawi is a landlocked country in Sub-Saharan Africa with a population of close to 17 million people. The country has a largely agricultural economy, a highly rural population, and high rates of poverty, with approximately 74% of the population earning less than $1.25 per day. Like most low-income countries, Malawi has a significant shortage of healthcare providers. With a ratio of approximately one physician for every 50,000 inhabitants, Malawi and neighboring Tanzania have the lowest doctor-to-patient ratio in the world¹⁵. For this reason, care in ART clinics is primarily provided by clinical officers (non-physician clinicians with 3 to 4 years of post-secondary medical training) and nurses. We collected de-identified EMR data from ART clinics in public hospitals in Malawi. This research was approved by the University of Pittsburgh Institutional Review Board (IRB), protocol #PRO12100159 and the Malawi National Health Sciences Research Committee (NHSRC), protocol #1019.

Performance measurement

Performance indicators are commonly used to identify potential problems that may represent data quality problems or valid exceptions to recommended clinical practice¹⁶. In prior research we developed a method for identifying guideline-based performance measures that could be routinely evaluated within EMR data for ART in Malawi¹⁷. Using this approach, we identified four performance measures based on statements from Malawi’s national guideline for the clinical management of HIV, 2011 edition¹⁸ which have also been used for quality improvement purposes in multiple Sub-Saharan African countries¹⁹ (Table 1). We measured individual-level rather than clinic-level performance because each task is commonly performed independently and because performance may be influenced by individual differences in knowledge, skills and motivation.

Table 1.

Guideline-based performance indicators for ART in Malawi¹⁷,¹⁸,¹⁹

Performance indicator	Malawi ART guideline recommendation	Numerator	Denominator
Monitoring of nutritional status: Pediatric patient height	“Record length / height to the nearest cm at every visit (children)”(2011 edition, page 18)	Number of children with height recorded at least once during the review period	Number of children with at least one clinical visit during the review period
Monitoring of nutritional status: Weight	“Record weight in kg to the nearest 100g at every visit” (2011 edition, page 18)	Number of patients with weight recorded at least once during the review period	Number of patients with at least one clinical visit during the review period
Cotrimoxazole Preventative Therapy (CPT) prescribing	“Provide CPT to all patients in HCC and ART follow-up” (2011 edition, page 32)	Number of patients who were prescribed CPT	Number of patients with at least one clinical visit during the review period without CPT contraindications
WHO clinical staging	“WHO clinical staging is mandatory for all HIV patients” (2011 edition, page 12)	Number of patients with a WHO clinical stage at the time of ART initiation	Number of patients who were initiated on ART during the review period

The performance measures in Table 1 use ratios that accommodate both individual and team-based care. The denominator reflects the total number of opportunities a provider had to provide recommended care to each patient. For example, if a provider used the EMR to document an ART visit with a patient who was eligible to receive cotrimoxazole preventative therapy (CPT), this patient was counted towards the total number of patients in the provider’s denominator for the month of that visit. The numerator reflects the documented care received by the patients who were counted in the individual provider’s denominator, regardless of who provided the care to the patient. For example, in a scenario where a provider does not prescribe CPT to an eligible patient at the time of an ART visit, but the patient receives a prescription for CPT on another day during the review period from any other provider, the patient would still be counted in the provider’s numerator. One exception is for patients whose care happens to be provided adequately but over a time frame spanning review periods. We used the Ruby programming language, the MySQL database system, and R statistical analysis software to measure and graph performance for each of the four performance indicators. To validate the results we reviewed the scripts, queries, and graphs of the performance data with EMR developers. We conducted the review by discussing the approach for performance measurement, the structure of each SQL query, and the definitions of clinical concepts used by each query.

Modeling the effect of feedback on clinical behavior

We created a preliminary model of the influence of feedback on ART performance, adapted from an earlier model of the effect of feedback on clinical behavior¹² (Table 2). The use of this model requires several key assumptions to be met. First, we assume that performance is measured routinely at individual and clinic levels so that a recipient’s performance can be interpreted in relation to group performance. Second, we assume that a supervising clinician is charged with giving feedback and has a) some awareness of the events that have occurred in the clinic during the performance measurement interval and b) some awareness of or willingness to make estimations about the individual recipients’ determinants of behavior such as knowledge, skills, and motivation¹². We demonstrate how the tailoring approaches in the rightmost column of Table 2 could be applied in a prototype feedback planning menu that a clinical supervisor to could use to select tailored messages for a healthcare provider (Table 3).

Table 2.

A preliminary model of the influence of feedback on antiretroviral therapy (ART) performance

COM-B	TDF domain	TDF construct	Barrier to ART performance improvement	Hypothetical causal mechanism for individual feedback	Potential influence of feedback	Performance features	Tailoring approach
Capability	Knowledge	Knowledge	Awareness of guideline	Feedback can change awareness to impart new knowledge that leads to behavior change	High	Consistently low individual performance, no prior feedback provided	Current score: Prioritize individual feedback, include guideline recommendations
			Awareness of performance			No prior feedback provided	Peer comparison: Prioritize individual feedback, include peer comparison
		Procedural knowledge	Knowledge of how to use the EMR			Consistently low individual performance, no prior feedback provided	Current score: Prioritize individual feedback, recommend or provide EMR training
Opportunity	Context and resources	Material resources	Unavailable resources (drugs, scale, EMR, measuring rod)	None (no direct influence on availability of material resources)	Low	Low group performance	Withhold feedback: Withhold or deprioritize individual feedback
	Social influences	Social pressure	Peer pressure and social norms	None (no direct influence on social norms)
Motivation	Beliefs about capabilities	Self-efficacy	Beliefs about specific capabilities	Feedback can influence perceptions of ability that lead to behavior change	Conditions on situation	Low individual performance, improvement trend	Self-comparison: Prioritize individual feedback using a truncated scale graph when improvement trend is present

Table 3.

Example prototype feedback planning menu for ART weight recording performance

Beliefs about determinants of recipient’s weight recording behavior	Tailoring approach	Proposed design
Recipient may be unaware that the ART guidelines recommend recording a patient’s weight at each clinic visit.	Current score: Prioritize individual feedback, include guideline recommendations	In the last month, 22% of patients who you provided care for in the ART clinic had their weight recorded. Malawi’s national ART guideline recommends that all patients have their weight recorded at each ART visit (2011 edition, page 18).
Recipient may be unaware of how he or she is performing relative to peers.	Peer comparison: Prioritize individual feedback, include peer comparison	In the last month, your performance was more than 60% below your top-performing peers in the ART clinic.
Recipient may not know how to use the EMR to record a patient’s weight.	Current score: Prioritize individual feedback, recommend or provide EMR training	In the last month, 22% of patients who you provided care for in the ART clinic had their weight recorded. To record a patient’s weight using the EMR, select “Record vitals” from the ART visit menu…
Recipient does not have access to a functioning scale to record patient weight.	Withhold feedback: Withhold or deprioritize individual feedback	[None]
A common practice in the clinic is to deprioritize the recording of patient’s weight relative to other tasks, especially during times of high workload.	Withhold feedback: Withhold or deprioritize individual feedback	[None]
Recipient is aware of his or her past performance. Recipient may believe he or she is not capable of improving performance. Increased effort is likely to lead to improvement.	Self-comparison: Prioritize individual feedback using recent performance history and a truncated scale graph when improvement trend is present	In the last 2 months, your weight recording performance has improved by 7%

Message tailoring process

Using the performance features from Table 2, we developed a rule-based message tailoring process (Table 4) to identify and make inferences about observable features of clinical performance data. The purpose of the tailoring process is to infer when individualized feedback might be useful, in various formats, and to prioritize a set of tailored feedback messages within a menu (Table 3) for a clinical supervisor to use.

Table 4.

Feedback message tailoring process

	1. Identify performance features	2. Infer construct salience	3. Infer message component relevance	4. Prioritize messages
Method and inputs	Feature classification using performance data	Rule-based inference using performance features	Rule-based inference using performance features	Rule-based inference using construct salience and message relevance
Output data type	Binary (Present/absent)	Integer score	Integer score	Integer score
Example	Feature: consistently_low_performance Rule: If performance has remained below 50% during the performance interval then consistently_low_performance is present	Construct:self-efficacy Rule: If consistently_low_performance is present, increase the salience score for self-efficacy as a barrier to improvement.	Message component:peer_comparison Rule: If a 10% performance gap exists between the recipient and top-performing peers, increase the relevance score for messages containing peer_comparison	Rule: If self_efficacy is salient as a barrier, decrease the priority score for messages using peer_comparison

1. Identify performance features

The first step of the message tailoring process is to classify each known feature as present or absent for an individual’s performance. Performance features are the individual and situational characteristics associated with an individual provider, and his or her behavior that is targeted by an AF intervention. Our objective was to demonstrate the feasibility of using a range of performance features that could support inferences about barriers to behavior change and their associated theoretical constructs. We selected a preliminary set of 11 performance features such as “consistently low performance” and “consistently high performance” based on our understanding of the clinical setting gained from a qualitative study of performance feedback in ART clinics in Malawi¹³. We anticipate that this feature set is a small sample of the meaningful features that could be used.

2. Infer construct salience

Once performance features are identified, these can be used to make inferences about an individual’s potential barriers to behavior change and associated theoretical constructs. For example, when individuals demonstrate consistently high performance for a behavior, we may infer that barriers to behavior change are not likely to be present. Conversely, consistently low performance may indicate that barriers associated with capability constructs such as a lack of knowledge or skills are in effect, especially when accompanied by concurrent high performance of peers.

3. Infer message component relevance

Some components of potential messages will be more relevant to an individual and situation that others. For example, showing a graph of performance data that emphasizes a trend by using a truncated scale will only be relevant when the improvement trend is above some threshold of meaningful improvement. We propose that performance features can also be used to determine which elements of feedback messages are most relevant for a recipient. For example, performance features could be used to estimate the relevance of the following feedback message components from the tailored messages in Table 3:

• Scale truncation: The use of a truncated vertical axis to emphasize change in performance

• Self comparison: Comparing an individual’s past performance with current performance

• Peer comparison: Comparing an individual’s current performance with current peer performance.

4. Prioritize messages

Finally, we prioritized messages appearing in the menu to place messages that were most likely to address barriers to behavior change at the top of the menu. To prioritize the messages, we evaluated both the message component relevance scores and estimated construct salience scores for the provider’s current month of performance. We created preliminary rules based on hypothetical causal mechanisms from Table 2 to prioritize each of the tailoring approaches in Tables 2 and 3 (Current score, peer comparison, withhold feedback, and self-comparison). We created two additional prioritization categories: no prioritization for the cases where no messages could be prioritized, and prioritized combination, in the event that two or more messages were equally of highest priority.

Analysis of performance data and prioritized messages

We conducted two simple analyses to understand the potential impact of message tailoring in ART clinics in Malawi: identification of performance gaps, and analysis of variability in message prioritization. We first calculated the average number of performance gaps occurring each month to estimate how frequently peer comparison feedback could be provided. We defined a performance gap as a 10% or greater difference in performance between an individual healthcare provider and the average performance of two higher-performing peers working in the same clinic in the same month. We excluded providers who had seen 10 or fewer patients in a month.

Our second analysis concerned the variability of prioritized messages within a tailoring menu. The purpose of this analysis was to understand how often a standardized message format could be used for all providers as compared to the routine tailoring of messages. We anticipate that increased variability of message priorities indicates a greater potential impact of message tailoring because it reflects individual and situational differences among healthcare providers that standardized feedback formats are less likely to accommodate. If messages of the same format have the same priority for more than 95% of providers, it would suggest that message tailoring is not necessary. However, if the priority of messages is more evenly stratified across message type groups, and if the size of these groups changes over time, it would suggest a greater potential impact for message tailoring. We created a prioritized list of message types for each performance measure across all individuals and months. To assess the variability of message formats, we identified the top-priority message format only and did not consider the rank of lower-priority message formats. We calculated the percentage of individual performances that had each of the four types of message as the highest priority, plus the two additional prioritization categories.

Results

Data collection

Data collection for this study occurred in November, 2013. We collected de-identified EMR data from 11 ART clinics in Malawi that were using the National ART EMR (Software version BART 1), and which was recorded between October, 2011 and September, 2013.

Performance measurement

We retrospectively measured clinical performance from 372 healthcare providers in 11 hospital-based ART clinics over a two-year period. We measured individual performance for all four measures at a monthly frequency by individual provider for a total of 7,448 individual monthly performance reports having a denominator of five or more patients. Across all clinics, an average of seven automated monthly performance reports could be generated per month for each performance indicator.

Summary statistics for each of the four guideline-based performance measures for all healthcare providers grouped by ART clinic are shown in Table 5. The average monthly performance for weight recording and WHO clinical staging were consistently high (> 94%) for all but one clinic. The average monthly performance for pediatric height recording ranged from 2.3% to 98.4%, while the average monthly performance for CPT prescribing ranged from 48.9% to 87.6%. For CPT prescribing performance, there was a generalized decrease in performance in 2012 across clinics, with providers at most clinics having a wide range of performance during the period. The generalized decrease in performance is associated with a national shortage of CPT drugs that occurred in 2012.

Table 5.

Estimated clinical performance of 11 ART clinics in Malawi

	Pediatric height recording		Weight recording		ART staging		CPT prescribing
Clinic	M	SD	M	SD	M	SD	M	SD
1	91.0%	0.065	95.1%	0.031	98.2%	0.032	81.1%	0.294
2	48.3%	0.159	94.9%	0.035	96.5%	0.042	87.6%	0.220
3	98.3%	0.021	99.4%	0.008	98.3%	0.017	83.3%	0.303
4	96.3%	0.037	99.6%	0.003	99.3%	0.010	65.2%	0.424
5	23.7%	0.138	84.3%	0.077	98.1%	0.021	60.0%	0.417
6	82.3%	0.079	95.1%	0.018	99.4%	0.011	65.5%	0.415
7	92.5%	0.029	98.5%	0.003	89.4%	0.074	74.0%	0.270
8	37.7%	0.227	98.3%	0.008	98.2%	0.020	81.1%	0.328
9	2.3%	0.017	99.3%	0.006	95.5%	0.036	48.9%	0.309
10	91.0%	0.063	97.5%	0.015	98.3%	0.030	73.3%	0.390
11	98.4%	0.009	99.1%	0.005	97.1%	0.017	79.7%	0.263

Performance gaps of 10% or greater

Performance gaps that could be used for peer-comparison feedback occurred slightly more frequently than 1 gap per month on average across all clinics and measures. The mean monthly total of performance gaps for each indicator across sites between October, 2011 and September, 2013 is shown in Figure 2. The average number of performance gaps that could be used to give peer comparison feedback to a single provider for all 11 sites ranged from 0.32 to 2.45 gaps per month.

Figure 2.

Average monthly performance gap totals between October, 2011 and September, 2013

Message prioritization variability

Based on the preliminary rule set, the tailoring process resulted in 35% (2,624 / 7,448) of individual monthly reports being prioritized to optimize the effect of feedback on performance. We calculated the percentage of all messages that had each message type as the highest priority on an individual’s monthly report (Figure 3). No reports had peer comparison messages as the highest priority. Across all performance indicators, increased stratification of tailored message types appears to be associated with lower performance. For example, the indicators having higher performance, which are weight recording and WHO clinical staging, had a higher average percentage of messages that were not prioritized, at 75% for weight and 83% for WHO clinical staging. In contrast, pediatric height recording and CPT prescribing, which have lower overall performance, had increased stratification of highest priority percentages across message types.

Figure 3.

Variability in prioritization of feedback messages

Discussion

We found that computer-supported feedback message tailoring could be routinely used in ART clinics in Malawi. The results of this study answer several important questions about using EMR data to generate tailored performance feedback messages in a low-resource setting. Most significantly, we identified an opportunity to use existing EMR data to routinely monitor individual clinical performance and provide tailored feedback across a range of guideline-based performance indicators in a low-resource setting. This approach could be expected to yield individualized monthly reports for ART providers working at each site, with approximately 35% of reports being tailored to optimize the effect of feedback on performance. Although performance appears to allow limited room for improvement in some ART clinics, we found regular opportunities to provide individualized feedback to address performance gaps and potential performance or data quality problems. These findings are significant because the existing National EMR infrastructure in Malawi would allow these reports to be generated in every ART clinic using the EMR, totaling 66 clinics at the end of June, 2015. Moreover, such a system may enable feedback to be generated more rapidly than the current quarterly reporting schedule of the National ART monitoring and evaluation program.

We sought to understand if feedback message tailoring could potentially impact clinical performance by exploring differences in features of performance data. We found differences in performance features that appear to hold meaningful implications for the design of feedback messages. On average, based on a preliminary set of hypothetical causal mechanisms offered by behavioral and cognitive theories, more than 50% of feedback messages for pediatric height recording could be tailored for individual or situational differences in performance. Similarly, close 50% of feedback messages could be tailored for differences in performance with regard to CPT prescribing in this setting. Where performance is higher, there appear to be fewer opportunities to tailor feedback messages. However, even the indicators having higher performance allowed for routine tailoring for approximately 25% of messages for weight recording on average, and for an average of 16.3% of messages for WHO clinical staging.

These findings are significant because they represent the first evaluation, to our knowledge, that uses a model of clinical behavior and psychological theory for the purpose of feedback message tailoring. To our knowledge, this approach represents a novel contribution that holds implications for related research in biomedical informatics, implementation science, and global health. In the field of biomedical informatics, this work introduces a novel class of knowledge-based system to support evidence-based care and quality improvement. In implementation science, this work is the first demonstration of theory-informed tailoring of feedback messages for individual healthcare providers based on standardized measures of clinical performance. In the domain of global health, this work represents the first supervision tool of its kind for a setting where supervision resources are limited. The significance of these findings increases with the increasing availability of eHealth data that can be used to generate performance feedback.

This research has several limitations. A key limitation of this analysis is that the application of theory within tailoring rules was not rigorously validated. In a prior model formation study¹² we applied theory using knowledge gained from a review of the literature. Furthermore, the model we developed for the effect of feedback on ART performance was a general model that did not accommodate potential differences in barriers to behavior change between the four clinical behaviors. Developing behavior-specific models, which is an important next step, may yield different message tailoring approaches.

A potential limitation of this approach is the ability of a clinical supervisor to accurately perceive barriers to behavior change that inform the selection of feedback messages. We anticipate that supervisors who use a feedback message tailoring system could overcome this limitation by observing the effects of their selected feedback messages on clinical performance. These observations could enable a supervisor to learn how to improve the use of feedback for individual recipients.

Our assessment did not account for data quality problems, however we anticipate that performance feedback could also be used to target the improvement of clinical data quality where such problems exist. The message tailoring process may also provide a systematic approach for integrating routine data quality assessment into AF, to inform clinical supervisors when clinical data is not “fit for use” as performance feedback.

The classification thresholds that we used for this analysis were chosen based on our understanding of the clinical context rather than empirical research. For example, we classified low group performance as an average performance below 50%, but it is likely that the actual thresholds for low group performance may vary across ART clinics, and across performance indicators. In the case of WHO clinical staging, a threshold for low group performance might be set much higher for most clinics because there are no valid exceptions for this guideline recommendation. In the case of CPT prescribing there are exceptions for patients with CPT contraindications, therefore a lower threshold would be expected. To address this limitation, we chose classification thresholds that err on the side of a lower bound, meaning that thresholds that we validate are likely to lead to greater variability of tailored messages.

Finally, a limitation of this proof-of-concept approach is that the manual development and maintenance of a message tailoring process for each clinical context is not likely to be sustainable. We aim to address this and other limitations in future research by developing a publicly-maintained feedback message tailoring knowledge-base that builds on established frameworks for applying psychological theory to behavior change interventions^10,11, and which uses an argumentation model to represent dynamic and contextualized evidence²⁰.

Conclusion

Computer-supported feedback message tailoring is a promising approach for improving the use of AF within LHSs. In this research we explored the design space for individually tailored feedback messages, to establish proof-of-concept for computer-supported feedback tailoring in Malawi. We found that individually-tailored performance feedback can be generated using routinely collected EMR data in ART clinics in Malawi. This finding suggests that clinical supervisors could use feedback tailoring tools to improve the effect of feedback on clinical performance in low-resource settings. Future research should study the use of a feedback tailoring system and its impact on clinical performance.