Skip to main content
NCBI home page
Learning Health Systems logoLink to Learning Health Systems
. 2024 Apr 9;8(3):e10419. doi: 10.1002/lrh2.10419

Precision feedback: A conceptual model

Zach Landis‐Lewis 1,✉, Allison M Janda 2, Hana Chung 3, Patrick Galante 1, Yidan Cao 1, Andrew E Krumm 1,3,4
PMCID: PMC11257058  PMID: 39036537

Abstract

Introduction

When performance data are provided as feedback to healthcare professionals, they may use it to significantly improve care quality. However, the question of how to provide effective feedback remains unanswered, as decades of evidence have produced a consistent pattern of effects—with wide variation. From a coaching perspective, feedback is often based on a learner's objectives and goals. Furthermore, when coaches provide feedback, it is ideally informed by their understanding of the learner's needs and motivation. We anticipate that a “coaching”‐informed approach to feedback may improve its effectiveness in two ways. First, by aligning feedback with healthcare professionals' chosen goals and objectives, and second, by enabling large‐scale feedback systems to use new types of data to learn what kind of performance information is motivating in general. Our objective is to propose a conceptual model of precision feedback to support these anticipated enhancements to feedback interventions.

Methods

We iteratively represented models of feedback's influence from theories of motivation and behavior change, visualization, and human‐computer interaction. Through cycles of discussion and reflection, application to clinical examples, and software development, we implemented and refined the models in a software application to generate precision feedback messages from performance data for anesthesia providers.

Results

We propose that precision feedback is feedback that is prioritized according to its motivational potential for a specific recipient. We identified three factors that influence motivational potential: (1) the motivating information in a recipient's performance data, (2) the surprisingness of the motivating information, and (3) a recipient's preferences for motivating information and its visual display.

Conclusions

We propose a model of precision feedback that is aligned with leading theories of feedback interventions to support learning about the success of feedback interventions. We plan to evaluate this model in a randomized controlled trial of a precision feedback system that enhances feedback emails to anesthesia providers.

Keywords: audit and feedback, coaching, healthcare quality, learning, performance improvement

1. INTRODUCTION

When performance data are provided as feedback to healthcare professionals, they may use it to significantly improve care quality. 1 However, the question of how to provide effective feedback remains unanswered, as decades of evidence have produced a consistent pattern of effects with wide variation. 2 Efforts to increase feedback intervention effectiveness includes calls for better use of evidence, 3 , 4 , 5 better coordination and embeddedness of trials of feedback in implementation laboratories, 5 , 6 more use and development of theory, 7 , 8 , 9 , 10 and improving the design of feedback in dashboards, reports, and information systems. 11 , 12 , 13 , 14 , 15

Enduring challenges such as the rapid expansion of biomedical and health knowledge 16 that is concurrent with provider burnout 17 , 18 and information chaos, 19 suggest that fundamentally different approaches to the delivery of feedback are needed to improve its effectiveness. In the context of healthcare professional coaching, a focus on protecting autonomy leads to asking the learner to guide the process of prioritizing objectives and goal setting for learning. 20 , 21 , 22 , 23 , 24 Based on these objectives and goals, a coach supports the learner with feedback that is appropriate for their performance level, motivation, and other identified needs. To our knowledge, automated systems that deliver feedback, such as clinical quality dashboards and reporting systems, are largely missing such personalized functionality.

In the absence of coaching‐type feedback in clinical quality reporting systems, allowing clinicians to prioritize their own feedback could help address these gaps. However, clinicians may lack insight into learning opportunities and priorities that a more global analysis of their performance data could inform. In contrast to feedback studies that focus narrowly on improving one clinical practice, we propose to shift the question of “what works” to be anchored on the feedback recipient, such as individuals, teams, or organizations, across clinical or health‐related practices. Infrastructure for studies based on this paradigm may offer several benefits, including an ability to leverage more granular analyses of performance information, 25 , 26 n‐of‐1 studies, 27 , 28 computer‐actionable theories of feedback, 29 and the development of dynamic, continuous‐tuning feedback systems. 30

To create infrastructure for large‐scale study of feedback interventions using a “coaching”‐informed approach, new models and system architectures are required. We developed a precision feedback system and models of information for this purpose. The context of anesthesia care offers a large set of quality metrics with high‐quality clinical process data and attribution to individual providers who use an anesthesia machine during operative cases. Currently, a subset of anesthesia clinicians work at institutions that contribute data to a platform that returns monthly quality email reports containing their crude performance on these quality metrics. This platform can be employed to study feedback and a conceptual model of precision feedback.

2. OBJECTIVE

Our objective is to propose a conceptual model of precision feedback. To achieve this objective, we iteratively represented models of feedback's influence from theories of motivation and behavior change, visualization, and human‐computer interaction. Through cycles of discussion and reflection, application to clinical examples, and software development, we implemented and refined the models in a software application to generate precision feedback messages from performance data for anesthesia providers.

3. PRECISION FEEDBACK

We propose a model of precision feedback (Figure 1). Precision feedback is feedback that is prioritized based on its motivational potential for a specific recipient. We present this model and describe its elements, beginning with a foundational definition of feedback, and describing each element of the model (Table 1). The term feedback has been defined and used in various ways to refer to the delivery and influence of performance information and related processes. 7 , 10 , 31 , 32 , 33 , 34 We use the term feedback to mean information about performance that can guide future action. 35 This definition originates from the context of learning environments, in which feedback is primarily delivered by educators and coaches using framing and prioritization that aims to motivate learners.

FIGURE 1.

FIGURE 1

Open in a new tab

A conceptual model of precision feedback.

TABLE 1.

Glossary.

Term Description Source
Achievement Motivating performance information that is about a change from a negative comparison to a positive comparison. 25
Benchmark A comparator with a performance level that is calculated from the performance of other health professionals or peers. 25,43
Comparator Information that is used to identify a discrepancy with the performance level of a feedback recipient. 25
Comparison Motivating performance information that is about a discrepancy between the performance levels of a feedback recipient and a comparator. 25
Explicit target A comparator with a performance level that is explicitly expected. 25,43
Feedback Information about performance that can guide future action. 35
Feedback recipient A person, team, or organization to whom a feedback intervention is directed. 25
Loss Motivating performance information that is about a change from a positive comparison to a negative comparison. 25
Motivating performance information Performance information that holds motivational potential.
Motivational potential An ability to motivate.
Performance information Information about measures, levels, time intervals, comparators, and a feedback recipient. 25,26
Precision feedback Feedback that is prioritized according to its motivational potential for a specific recipient.
Trend Motivating performance information that is about a change in performance. 25
Open in a new tab

For our purposes, feedback refers to statements and quantitative information about past performance that are distinct from advice about the future. For example, feedback to a physician about antibiotic stewardship might include a statement about the proportion of appropriate prescriptions for patients in a previous month, such as “Your rate of appropriate prescribing of antibiotics was below the standard of care for September, 2023.” In contrast, advice would include guidance about how to improve, such as “Avoid sending a urine culture when the patient does not have any symptoms of catheter‐associated urinary tract infection.” We recognize that advice is sometimes referred to as “corrective feedback.” 22 , 33 We adopt a narrow meaning of feedback in our context to enable clarity about the delivery, use, and functions of performance information. Similarly, we use the term performance information narrowly to mean statements and quantitative data about performance levels of recipients and comparators (Table 1).

To recognize types of feedback, we use a typology of feedback based on its function for feedback recipients, which can include evaluation, coaching or appreciation (Figure 2). 36 Most feedback provided via audit and feedback can be recognized as evaluation feedback, that is necessary to inform feedback recipients about where they stand, relative to comparators, and their current performance levels, possibly with historical performance information to visualize performance changes. Evaluation feedback is needed to further recognize the other functions of feedback. Coaching feedback involves identifying learning opportunities to motivate improvement and see progress, whereas appreciation feedback involves recognizing accomplishments and motivating sustainment of performance. Feedback that is provided for any of these purposes can originate from multiple sources, including from patients, a recipient's team leader or supervisor, peers, or telemetry data from machines that may be summarized in a report. For our purposes, the source of the feedback is independent from the proposed model, which focuses on the information content that is related to motivation.

FIGURE 2.

FIGURE 2

Open in a new tab

Precision feedback message types and examples.

We developed the proposed model through iterative modeling and analyses of performance information from a wide range of clinical contexts. 26 However, our primary demonstration domain for this model is anesthesia care. We implemented the model in the context of an anesthesia care research and quality improvement consortium, the Multicenter Perioperative Outcomes Group (MPOG). 37 , 38 MPOG has developed and maintains a national‐scale infrastructure for perioperative quality improvement initiatives in more than 70 institutions and 23 US States. Each month, data from the electronic health record (EHR) and complementary data sources are sent from member institutions to the MPOG registry. Quality improvement measure performance is attributed to individual anesthesia providers, including attending anesthesiologists, resident anesthesiologists, and Certified Registered Nurse Anesthetists (CRNAs) based on their relationship to the case and process of care or outcome measured. Approximately 10 000 anesthesia providers receive a feedback email from MPOG each month about care quality and outcomes of their operative cases. MPOG has developed more than 70 quality improvement measures using EHR data and computed phenotypes that can be selected for inclusion in emails. A primary purpose of the provider feedback email is to support individual quality improvement and learning with data about that individual's clinical practice. These data are also available in a clinical quality dashboard for each provider to review, and in aggregate for quality champions at an institution to review, using standardized representations of operative case progression and clinical outcomes. To implement precision feedback within the MPOG infrastructure, we have enhanced emails with messages that appear at the top of the email template (Figure 3).

FIGURE 3.

FIGURE 3

Open in a new tab

Example precision‐feedback enhanced email to an anesthesia provider.

4. PERFORMANCE INFORMATION

Performance information can be understood to generally contain five data elements: measures, recipients, comparators, performance levels, and time intervals. 25 , 26 These elements can form a foundation for analyses to produce precision feedback and form the basis for elicitation of preferences about alternative types of performance information.

4.1. Measures

Measures are indicators or metrics used to calculate performance. 39 , 40 , 41 , 42 Measures in healthcare are widely used for the purposes of quantifying and improving healthcare quality, related to both care processes and outcomes. While these measures are used at all health system levels, not all measures are suitable for generating feedback to clinicians, especially at higher levels of scale. Measures that are focused at the organizational or team‐level may have little applicability for front‐line healthcare providers. In some cases, however, when performance is attributed to individuals, teams, and organizations, the data that are generated may be useful to clinicians.

Sets of measures may be developed that link processes and outcomes for better assessment of healthcare. For example, linked measures for prevention of postoperative nausea may include a process measure addressing appropriate prescribing of anti‐nausea medication, and a clinical outcome measure of observed post‐operative nausea and vomiting.

4.2. Recipients

When performance is measured, it must be attributed to some person or group by whom the information is intended to be received. The primary recipient of performance information is not necessarily the person who has provided health care directly. Performance information may be intended to be received by individuals or teams at all levels of health systems, from individual clinical team members to healthcare administrators. In some cases, recipients are specified at larger levels of scale, such as a learning community whose performance is measured as a whole, relative to other learning communities.

4.3. Comparators

Comparators are the goals or standards that a feedback recipient compares their performance to. 43 A key type of comparator is an explicit target that represents a desirable future state, such as a goal or standard. Explicit targets may be set by the feedback recipient as part of a learning goal, or by others, such as a quality improvement consortium. Another key type of comparator is a benchmark that represents a summary statistic for a population's performance, such as an average or a top‐performer percentile. Benchmarks are social comparators, derived from a population's performance to make a comparison. The terms benchmark and goal as comparators are sometimes used interchangeably, likely because feedback recipients may set goals based on the performance levels of benchmarks.

4.4. Performance levels

When performance is measured, data are produced that can be called performance levels. Performance levels are commonly represented as ratio‐scale values, such as counts and percentages. These performance levels are attributable to individuals and teams or may be attributed to a summary statistic for a population, such as the peer average or the achievable benchmark of care. 44 Measurement processes produce a performance level that is attributed to a recipient, and commonly to benchmark comparators. For explicit target comparators, a performance level may be chosen without a measurement process, for example when a healthcare professional or team sets an improvement goal based on their experience, without referencing others' performance.

Depending on the purpose of performance measurement, performance levels may be expressed in non‐ratio scale values, for instance by ordinal scale values (eg, “high” or “low” or red/green without a numerical level) or using interval scale values, such as grades that have an underlying percentage value which is not made explicit. Performance levels are commonly displayed in dashboards using multiple representations and scale types, including visualizations that complement numerical and text‐based representations. For example, bar charts use the length of a bar, which is a ratio‐scale attribute, to graphically represent performance levels such as counts or percentages. 45

4.5. Time intervals

Performance information may be about a single time interval (eg, FY 2023), or for multiple time intervals in series. Adding the dimension of time to performance information enables recipients to perceive rates of change to establish expectations for future performance. In healthcare organizations, time intervals included in performance information commonly range from monthly to annually. Visual displays that contain multiple time intervals, or time‐series displays, have the potential to show trends in performance data.

5. MOTIVATING PERFORMANCE INFORMATION

Not all performance information is motivating, but feedback intervention theories suggest that motivation is a foundational mechanism for feedback. 20 , 33 , 46 , 47 , 48 However, there are multiple types of motivation from feedback, and potential adverse consequences that include de‐motivating the recipient. 20 The foundational elements of performance information (measures, recipients, comparators, performance levels, and time intervals) can be used to understand the motivational potential of performance information. Motivating performance information includes comparisons, trends, achievement, and loss, each of which have potential to be motivating or demotivating. Feedback recipients may also have diverse preferences for motivating performance information, depending on their motivational orientation, organizational context, and information needs.

5.1. Comparisons

A performance comparison is a discrepancy between two performance levels within a single time interval. 25 Comparisons are typically made between the levels of a feedback recipient and a comparator. Feedback theories recognize that when a comparison is negative (ie, the recipient's performance level is worse than that of a comparator), the recipient may be motivated to increase effort to eliminate the discrepancy that is revealed by the comparison. 33 , 47 , 48 Conversely, when a recipient's performance is better than a comparator, they may be motivated to sustain performance (ie, maintain a positive comparison). The size of a performance comparison relates to its motivational potential, such that larger comparisons may have greater motivational potential than smaller comparisons, in cases where the delivery of this information changes the awareness of the recipient. When the recipient's performance equals that of a comparator, it can be considered as a comparison having a size equal to zero.

5.2. Trends

A trend is information about a change in performance. 25 Trends can show performance improving or worsening (ie, positive or negative trends), and this rate of change, also called performance velocity, 33 is commonly visualized as the slope of a trend line across time intervals. Feedback recipients use trends to establish expectations for future performance that can be motivating or demotivating, depending on the recipient's motivational orientation and contextual factors. 20 , 46 , 49 The slope of a trend line relates to its motivational potential, such that a greater slope indicates a greater performance velocity, whether positive or negative.

5.3. Achievement

Achievement, as represented in performance information, can be understood as a change from a negative performance comparison to a positive one. 25 For example, when a recipient's performance was worse than a top‐performer benchmark at a previous time interval (ie, generating a negative comparison), and has improved to equal or exceed the top‐performer benchmark at the current time interval (ie, generating a positive comparison), the recipient can be understood to have achieved the benchmark. As defined, achievement necessitates the existence of a positive trend, with a previous negative comparison and a current positive comparison. The motivational potential of achievement may depend on several factors, including the prior negative comparison sizes, the slope of the current positive trend, and prior achievement recency, if any, for a given performance measure. Achievement can be especially motivating in the context of learning and skill development, where performance improvement is desired.

5.4. Loss

Loss is the inverse of achievement, represented by performance changes from a positive to a negative comparison, 25 which also necessitates a negative trend. The motivational potential of loss, when delivered as performance information, depends on the size of prior positive comparisons and the slope of the current negative trend, as well as prior loss recency. Loss can be especially motivating when safety and avoidance of problems are prioritized, 46 , 49 and where performance sustainment is desired, rather than improvement.

6. SURPRISINGNESS

Surprisingness of feedback refers to the magnitude of characteristics that contribute to surprise, through the delivery of unexpected performance information. Feedback can be understood from the perspective of the recipient as something that has value in changing awareness and expectations. 29 For example, a feedback message with the same information, delivered in two consecutive time intervals provides little information to the recipient, and may be considered a waste of time to receive, thus such a message has little motivational potential. This perspective of feedback is consistent with information theory and communication models. 50 Variables related to surprisingness are comparison size, trend line slope, achievement and loss recency, and message recency.

6.1. Comparison size

The size of a comparison is a ratio‐scale value about the distance between the performance levels of a feedback recipient and a comparator. For example, if a recipient's performance level is 80% and the level of a top‐performer benchmark is 92%, the comparison size is −12% (80%‐92% = −12%). Comparison size influences the surprisingness of motivating information because larger sizes may be more motivating, while smaller sizes may be less motivating. It is possible that as performance improves, smaller comparison sizes could increase motivation as a function of the recipient's reinforced self‐efficacy and increased expectancy of goal attainment. 48 This type of motivation may be akin to receiving a motivational boost when nearing the finish line of an endurance race. The moderating influence of comparison size may differ based on the type of motivation that a recipient experiences and the sign of the performance level. 48 , 51 For example, the influence of comparison sizes on motivation may vary less for positive feedback than they do for negative feedback, although to our knowledge, this has not been tested in clinical audit and feedback.

6.2. Trend line slope

The slope of a trend line indicates a rate of change in performance level. As this rate increases, whether worsening or improving, the surprisingness of the information may increase. The absence of a trend indicates that a performance level has remained constant. Providing this information to a feedback recipient is unlikely to be motivating because it is less likely to change their awareness of how their clinical behaviors are reflected in their performance measurements. However, it is entirely possible that constant performance levels can provide motivational influences, such as in the case of activity streaks. 52

6.3. Achievement and loss recency

Recency is the duration between repeated performance events, such as recurring achievements or losses. Healthcare professionals may habituate to the repeated delivery of performance showing achievements or losses, reducing its motivational influence, but there is a lack of evidence about habituation to clinical audit and feedback. Perhaps the initial achievement of a goal is the most motivating in a learning environment. The same could be said for a loss, such as when performance falls below some standard for the first time. As the duration between the last occurrence and a new occurrence increases, the motivational influence of the event may also increase.

6.4. Message recency

Similar to achievement and loss recency, sending the same message repeatedly to a feedback recipient may reduce its motivational potential as healthcare professionals habituate to a specific message, but we lack evidence about these processes. The recency of messages and their performance measures previously delivered can be monitored to avoid producing low‐value feedback. However, in some cases, continuity of feedback may be desirable, such as during continued improvement towards a goal as a feedback recipient goes through a learning curve.

7. MOTIVATION FROM FEEDBACK

7.1. Motivational potential

We anticipate that the concept of motivational potential is needed to enable precision feedback that can improve the effectiveness of feedback interventions. We define motivational potential as the ability to motivate. Many factors may contribute to the motivational potential of feedback interventions, including characteristics of the recipient, the decision or behavior that feedback is about, and their setting.

Feedback interventions have potential to demotivate recipients, resulting in unintended consequences, such as goal abandonment and reduced self‐efficacy. 20 , 33 , 53 Regulatory Fit Theory describes mechanisms through which feedback can demotivate, such as when recipients who are oriented towards growth and improvement receive repeated negative feedback, which can lead to discouragement and eventual task or goal abandonment. 20 , 22 , 33 , 46 Moreover, when performance is stable, performance feedback may not be motivating because it does not provide much new information, especially when performance is high and prior feedback has made recipients aware of their continued high performance.

Beyond avoiding feedback that lacks motivational potential, to our knowledge, clinical quality dashboards do not adequately leverage the motivational potential of positive feedback, which can motivate recipients to increase their effort to improve. 22 Motivating information that results in positive feedback includes information about improving trends and positive comparisons, which may be related to the approach or achievement of goals or benchmarks.

We understand motivational potential as an important attribute of performance feedback messages that can be used to guide the prioritization of precision feedback messages. Factors affecting motivational potential include the motivating information that a feedback message contains and the corresponding surprisingness of these elements (Figure 1).

7.2. Precision feedback messages

Precision feedback messages are statements in natural language about motivating performance information. Feedback can be visualized in charts or displayed in a table that do not contain motivational messages, but using these messages may facilitate the interpretation of visualizations or may concisely describe performance when no visualization is needed. For example, the message “You are not a top performer” was used to motivate providers to reduce unnecessary prescribing of antibiotics in concise emails without a chart and with minimal information about performance. 54 Motivational messages frame performance information in ways that may affect motivational potential. For example, a negative performance comparison between the recipient and a comparator can be framed as an opportunity to improve, or as a risk for poor outcomes.

7.3. Precision feedback preferences

The motivational potential of feedback messages has a relationship with the recipients' preferences for motivating performance information. Preference for this information refers to the relative value of characteristics of motivating information that a recipient holds when alternatives are available. For example, ranking of individual peers as comparators may be desirable in some clinical specialties, and may be strongly dispreferred in others. Within a population, preferences for the visual display of motivating information may vary based on the purpose of the visualization, 55 graph literacy and numeracy, 56 and ease of cognitive processing of charts. 57

Preferences can be elicited for a population using various discrete‐choice experiment methods, including best‐worst scaling, conjoint analysis, or other discrete choice experiment approaches. 58 , 59 , 60 , 61 Cluster and subgroup analyses can be used to identify common preference groups in a population, and to develop profiles for feedback recipients within a region, profession, or organization. Challenges for preference elicitation include achieving sample size sufficiency as well as uncertainty about preference stability and completeness.

In any population, there is a “one‐size‐fits n” spectrum, such that “one size” fits a number of individuals ranging from the population total to a single individual (Figure 4). 62 Recognizing feedback preference clusters may help to inform the use of feedback messages to satisfy greater numbers of preferences in a diverse population. A recipient's preferences for positive feedback may be in tension with an organizational aim of improving low performance. To some extent, framing may be used to align feedback about low performance with an individuals' motivational orientation. 51 For example, it may be feasible to use a preferred framing about progress towards goals when current performance is below a desired level, but gradually improving. Nevertheless, we anticipate that it will be necessary to balance appreciation messages that recognize accomplishments with coaching messages that focus on areas for improvement, even when negative feedback is generally dispreferred by recipients. It may be feasible to identify an optimal ratio of positive to negative feedback for individuals, or for a population. 63

FIGURE 4.

FIGURE 4

Open in a new tab

A one‐size‐fits‐n spectrum.

8. DISCUSSION

We have proposed a model of precision feedback that recognizes motivating information and the motivational potential of feedback as being central to improving the value of feedback interventions to healthcare professionals and teams. Based on theories of motivation applied to feedback or developed for feedback interventions, this model incorporates elements of performance information, recipient preferences, and surprisingness variables for motivating information, all of which may be essential for improving the motivational potential of feedback.

Precision feedback builds upon several theories and relates closely to Clinical Performance Feedback Intervention Theory (CP‐FIT), a leading theory for implementation research in audit and feedback. 10 In relation to CP‐FIT, precision feedback can be understood to instantiate and extend the set of feedback variables for the purpose of increasing the successful completion of feedback cycles on the part of individuals and teams. For example, using a precision feedback approach, high‐priority feedback messages may be more likely to be interacted with, perceived, accepted, and used to form intentions to improve or sustain performance. Motivating information and motivational potential could be tested as theoretical constructs that can inform the prioritization of feedback messages, to contribute to future development of CP‐FIT.

The proposed model is developed primarily from behavior change theories 64 related to feedback and is aligned with work to develop a cumulative science of behavior change interventions. 65 , 66 , 67 The proposed model may contribute to further modeling of behavior change techniques related to the use of feedback, and to build on the findings of related studies. 68 , 69 This model relates to frameworks for behavior change theory, such as COM‐B, 70 focusing on capability and motivation as the primary types of theoretical mechanisms through which feedback influences behavior. 29

The proposed model of precision feedback may guide research about precision feedback messages that are both easier to cognitively process and more motivating to the recipient. Precision feedback systems may enable the provision of more effective feedback via the following mechanisms: (1) enabling feedback interventions to be prioritized and adapted for healthcare professionals' diverse needs and preferences, (2) enabling healthcare organizations to learn about the effectiveness of feedback through a new infrastructure and data sources about the value of feedback, and (3) enabling implementation researchers to study the influence of feedback information elements in the context of learning health systems.

As an untested model, the limitations for this work are not yet well‐defined. However, we anticipate that as health informatics and learning health system researchers from a single institution, there are many additional perspectives and contexts that could inform the further development and refinement of the model. We have high confidence in the relevance of these foundational constructs, but we anticipate that this model is not complete and that additional constructs may be essential. Nevertheless, our model was refined through substantial work from diverse team members and informed by our related work. 25 , 26 Furthermore, this model has been refined through the development of a precision feedback system that has been designed to prioritize feedback messages using these constructs at national scale for anesthesia providers. 62

To advance the research that this model enables, we recognize a need for future studies to develop measures of motivational potential and engagement with precision feedback messages. An implication of this model is that studies of feedback are needed to better understand moderating relationships between preferences and motivating information. Another area of research may be to assess influences of surprisingness variables on the effectiveness of feedback through retrospective analyses of performance data from feedback interventions. Finally, an implication of our model for future research is that surprisingness data could be collected in future trials of feedback interventions at large scale.

We plan to evaluate this model in a randomized controlled trial of a precision feedback system that enhances feedback emails to anesthesia providers. 62 As defined, we anticipate that precision feedback may be applied broadly to feedback about clinical performance, and potentially other health‐related contexts.

CONFLICT OF INTEREST STATEMENT

Zach Landis‐Lewis has received research support, paid to the University of Michigan, and related to this work, from the National Library of Medicine (K01 LM012528, R01LM013894). Allison M. Janda has received research support, paid to the University of Michigan and unrelated to this work, from Becton, Dickinson, and Company.

ACKNOWLEDGMENTS

The authors would like to acknowledge the National Library of Medicine for funding this research (K01LM012528, R01LM013894). The authors would also like to thank Rachel Richesson for her comments on a draft manuscript. Allison M. Janda received research support from the National Institute of General Medical Sciences under award T32GM103730. The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Funding for participation in the Multicenter Perioperative Outcomes Group and the Anesthesiology Performance Improvement and Reporting Exchange was provided by departmental and institutional resources at each contributing site. In addition, partial funding to support underlying electronic health record data collection into the Multicenter Perioperative Outcomes Group registry was provided by Blue Cross Blue Shield of Michigan/Blue Care Network as part of the Blue Cross Blue Shield of Michigan/Blue Care Network Value Partnerships program. Although Blue Cross Blue Shield of Michigan/Blue Care Network and Multicenter Perioperative Outcomes Group work collaboratively, the opinions, beliefs, and viewpoints expressed by the authors do not necessarily reflect the opinions, beliefs, and viewpoints of Blue Cross Blue Shield of Michigan/Blue Care Network or any of its employees.

Landis‐Lewis Z, Janda AM, Chung H, Galante P, Cao Y, Krumm AE. Precision feedback: A conceptual model. Learn Health Sys. 2024;8(3):e10419. doi: 10.1002/lrh2.10419

REFERENCES

  • 1. Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:CD000259. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2. Ivers NM, Grimshaw JM, Jamtvedt G, et al. Growing literature, stagnant science? Systematic review, meta‐regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med. 2014;29(11):1534‐1541. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3. Brehaut JC, Colquhoun HL, Eva KW, et al. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med. 2016;164(6):435‐441. [DOI] [PubMed] [Google Scholar]
  • 4. Ivers NM, Sales A, Colquhoun H, et al. No more “business as usual” with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci. 2014;9(1):14. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Foy R, Skrypak M, Alderson S, et al. Revitalising audit and feedback to improve patient care. BMJ. 2020;368:27. Available from: https://www.bmj.com/content/368/bmj.m213 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Grimshaw JM, Ivers N, Linklater S, et al. Reinvigorating stagnant science: implementation laboratories and a meta‐laboratory to efficiently advance the science of audit and feedback. BMJ Qual Saf. 2019;28(5):416‐423. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Colquhoun HL, Brehaut JC, Sales A, et al. A systematic review of the use of theory in randomized controlled trials of audit and feedback. Implement Sci. 2013;8(1):66. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. Colquhoun HL, Carroll K, Eva KW, et al. Advancing the literature on designing audit and feedback interventions: identifying theory‐informed hypotheses. Implement Sci. 2017;29(12):117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Brehaut JC, Eva KW. Building theories of knowledge translation interventions: use the entire menu of constructs. Implement Sci. 2012;7(1):114. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10. Brown B, Gude WT, Blakeman T, et al. Clinical performance feedback intervention theory (CP‐FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta‐synthesis of qualitative research. Implement Sci. 2019;14(1):40. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Colquhoun HL, Sattler D, Chan C, et al. Applying user‐centered design to develop an audit and feedback intervention for the home care sector. Home Health Care Manag Pract. 2017;29(3):148‐160. [Google Scholar]
  • 12. Bravo CA, Llovet D, Witteman HO, et al. Designing emails aimed at increasing family physicians' use of a web‐based audit and feedback tool to improve cancer screening rates: cocreation process. JMIR Hum Factors. 2018;5(3):e25. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Landis‐Lewis Z, Kononowech J, Scott WJ, et al. Designing clinical practice feedback reports: three steps illustrated in veterans health affairs long‐term care facilities and programs. Implement Sci. 2020;15(1):7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Brown B, Balatsoukas P, Williams R, Sperrin M, Buchan I. Interface design recommendations for computerised clinical audit and feedback: hybrid usability evidence from a research‐led system. Int J Med Inform. 2016;94:191‐206. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Brown B, Balatsoukas P, Williams R, Sperrin M, Buchan I. Multi‐method laboratory user evaluation of an actionable clinical performance information system: implications for usability and patient safety. J Biomed Inform. 2018;1(77):62‐80. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Stead WW, Searle JR, Fessler HE, Smith JW, Shortliffe EH. Biomedical informatics: changing what physicians need to know and how they learn. Acad Med. 2011;86(4):429‐434. [DOI] [PubMed] [Google Scholar]
  • 17. West CP, Dyrbye LN, Shanafelt TD. Physician burnout: contributors, consequences and solutions. J Intern Med. 2018;283(6):516‐529. [DOI] [PubMed] [Google Scholar]
  • 18. Shanafelt TD, Hasan O, Dyrbye LN, et al. Changes in burnout and satisfaction with work‐life balance in physicians and the general US working population between 2011 and 2014. Mayo Clin Proc. 2015;90(12):1600‐1613. [DOI] [PubMed] [Google Scholar]
  • 19. Beasley JW, Wetterneck TB, Temte J, et al. Information chaos in primary care: implications for physician performance and patient safety. J Am Board Fam Med. 2011;24(6):745‐751. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Kluger AN, van Dijk D. Feedback, the various tasks of the doctor, and the feedforward alternative. Med Educ. 2010;44(12):1166‐1174. [DOI] [PubMed] [Google Scholar]
  • 21. Sargeant J, Lockyer J, Mann K, et al. Facilitated reflective performance feedback: developing an evidence‐ and theory‐based model that builds relationship, explores reactions and content, and coaches for performance change (R2C2). Acad Med. 2015;90(12):1698‐1706. [DOI] [PubMed] [Google Scholar]
  • 22. Hattie J, Timperley H. The Power of feedback. Rev Educ Res. 2007;77(1):81‐112. [Google Scholar]
  • 23. Greenberg CC, Klingensmith ME. The continuum of coaching: opportunities for surgical improvement at all levels. Ann Surg. 2015;262(2):217‐219. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24. Roberts NK, Williams RG, Kim MJ, Dunnington GL. The briefing, intraoperative teaching, debriefing model for teaching in the operating room. J Am Coll Surg. 2009;208(2):299‐303. [DOI] [PubMed] [Google Scholar]
  • 25. Landis‐Lewis Z, Stansbury C, Rincon J, Gross C. Performance summary display ontology: Feedback intervention content, delivery, and interpreted information. Int Conf Biomed Ontol (ICBO 2022), Ann Arbor, MI. (pp. paper‐2172) 2022; Available from: https://icbo-conference.github.io/icbo2022/papers/ICBO-2022_paper_2172.pdf [Google Scholar]
  • 26. Lee D, Panicker V, Gross C, Zhang J, Landis‐Lewis Z. What was visualized? A method for describing content of performance summary displays in feedback interventions. BMC Med Res Methodol. 2020;20(1):90. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27. Hekler EB, Klasnja P, Chevance G, Golaszewski NM, Lewis D, Sim I. Why we need a small data paradigm. BMC Med. 2019;17(1):133. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28. Lyon AR, Connors E, Jensen‐Doss A, et al. Intentional research design in implementation science: implications for the use of nomothetic and idiographic assessment. Transl Behav Med. 2017;7(3):567‐580. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29. Landis‐Lewis Z, Brehaut JC, Hochheiser H, Douglas GP, Jacobson RS. Computer‐supported feedback message tailoring: theory‐informed adaptation of clinical audit and feedback for learning and behavior change. Implement Sci. 2015;10(1):12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30. Hekler EB, Rivera DE, Martin CA, et al. Tutorial for using control systems engineering to optimize adaptive Mobile health interventions. J Med Internet Res. 2018;20(6):e214. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31. Shute VJ. Focus on formative feedback. Rev Educ Res. 2008;78(1):153‐189. [Google Scholar]
  • 32. Åström KJ, Murray RM. Feedback Systems: An Introduction for Scientists and Engineers. 2nd ed. Princeton: Princeton University Press; 2021:528. [Google Scholar]
  • 33. Kluger AN, DeNisi A. The effects of feedback interventions on performance: a historical review, a meta‐analysis, and a preliminary feedback intervention theory. Psychol Bull. 1996;119(2):254‐284. [Google Scholar]
  • 34. Ajjawi R, Regehr G. When I say … feedback. Med Educ. 2019;53(7):652‐654. [DOI] [PubMed] [Google Scholar]
  • 35. Ambrose SA, Bridges MW, DiPietro M, Lovett MC, Norman MK, Mayer RE. How Learning Works: Seven Research‐Based Principles for Smart Teaching. 1st ed. San Francisco, CA: Jossey‐Bass; 2010:301. [Google Scholar]
  • 36. Stone D, Heen S. Thanks for the Feedback: the Science and Art of Receiving Feedback Well. 1st ed. New York, New York: Viking; 2014:368. [Google Scholar]
  • 37. Kheterpal S. Clinical research using an information system: the multicenter perioperative outcomes group. Anesthesiol Clin. 2011;29(3):377‐388. [DOI] [PubMed] [Google Scholar]
  • 38. Colquhoun DA, Shanks AM, Kapeles SR, et al. Considerations for integration of perioperative electronic health records across institutions for research and quality improvement: the approach taken by the multicenter perioperative outcomes group. Anesth Analg. 2020;130(5):1133‐1146. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39. van der Geer E, van Tuijl HFJM, Rutte CG. Performance management in healthcare: performance indicator development, task uncertainty, and types of performance indicators. Soc Sci Med. 1982;69(10):1523‐1530. [DOI] [PubMed] [Google Scholar]
  • 40. Cassel CK, Conway PH, Delbanco SF, Jha AK, Saunders RS, Lee TH. Getting more performance from performance measurement. N Engl J Med. 2014;371(23):2145‐2147. [DOI] [PubMed] [Google Scholar]
  • 41. McGlynn EA, Asch SM. Developing a clinical performance measure. Am J Prev Med. 1998;14(suppl 3):14‐21. [DOI] [PubMed] [Google Scholar]
  • 42. Eddy DM. Performance measurement: problems and solutions. Health Aff Proj Hope. 1998;17(4):7‐25. [DOI] [PubMed] [Google Scholar]
  • 43. Gude WT, Brown B, van der Veer SN, et al. Clinical performance comparators in audit and feedback: a review of theory and evidence. Implement Sci IS. 2019;14(1):39. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44. Weissman NW, Allison JJ, Kiefe CI, et al. Achievable benchmarks of care: the ABCs of benchmarking. J Eval Clin Pract. 1999;5(3):269‐281. [DOI] [PubMed] [Google Scholar]
  • 45. Zhang J. A representational analysis of relational information displays. Int J Hum Comput Stud. 1996;45(1):59‐74. [Google Scholar]
  • 46. Higgins ET. Value from regulatory fit. Curr Dir Psychol Sci. 2005;14(4):209‐213. [Google Scholar]
  • 47. Carver CS, Scheier MF. Control theory: a useful conceptual framework for personality—social, clinical, and health psychology. Psychol Bull. 1982;92(1):111‐135. [PubMed] [Google Scholar]
  • 48. Locke EA, Latham GP. Building a practically useful theory of goal setting and task motivation: a 35‐year odyssey. Am Psychol. 2002;57(9):705‐717. [DOI] [PubMed] [Google Scholar]
  • 49. van Dijk D, Kluger AN. Task type as a moderator of positive/negative feedback effects on motivation and performance: a regulatory focus perspective. J Organ Behav. 2011;32(8):1084‐1105. [Google Scholar]
  • 50. Shannon CE, Weaver W. The Mathematical Theory of Communication. Urbana, Illinois: University of Illinois Press; 1963:148. [Google Scholar]
  • 51. van Dijk D, Kluger AN. Feedback sign effect on motivation: is it moderated by regulatory focus? Appl Psychol. 2004;53(1):113‐135. [Google Scholar]
  • 52. Weathers D, Poehlman TA. Defining, and understanding commitment to, activity streaks. J Acad Mark Sci. 2023;52:531‐553. doi: 10.1007/s11747-023-00944-4 [DOI] [Google Scholar]
  • 53. Kluger AN, DeNisi A. Feedback interventions: toward the understanding of a double‐edged sword. Curr Dir Psychol Sci. 1998;7(3):67‐72. [Google Scholar]
  • 54. Meeker D, Linder JA, Fox CR, et al. Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: a randomized clinical trial. JAMA. 2016;315(6):562‐570. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55. Munzner T. Visualization Analysis and Design. 1st ed. Boca Raton: A K Peters/CRC Press; 2014:428. [Google Scholar]
  • 56. Dowding D, Merrill JA, Onorato N, Barrón Y, Rosati RJ, Russell D. The impact of home care nurses' numeracy and graph literacy on comprehension of visual display information: implications for dashboard design. J Am Med Inform Assoc. 2018;25(2):175‐182. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 57. Hegarty M. Advances in Cognitive Science and Information Visualization. Score Reporting Research and Applications. New York: Routledge; 2018. Available from: https://www.taylorfrancis.com/ [Google Scholar]
  • 58. Rao VR. Applied Conjoint Analysis. Heidelberg: Springer; 2014. [Google Scholar]
  • 59. Bridges JFP, Hauber AB, Marshall D, et al. Conjoint analysis applications in health—a checklist: A report of the ISPOR good research practices for conjoint analysis task force. Value Health. 2011;14(4):403‐413. [DOI] [PubMed] [Google Scholar]
  • 60. Louviere JJ, Flynn TN, Marley AAJ. Best‐Worst Scaling: Theory, Methods and Applications. Cambridge, UK: Cambridge University Press; 2015:363. [Google Scholar]
  • 61. Weernink MGM, Janus SIM, van Til JA, Raisch DW, van Manen JG, IJzerman MJ. A systematic review to identify the use of preference elicitation methods in healthcare decision making. Pharm Med. 2014;28(4):175‐185. [DOI] [PubMed] [Google Scholar]
  • 62. Landis‐Lewis Z, Flynn A, Janda A, Shah N. A scalable service to improve health care quality through precision audit and feedback: proposal for a randomized controlled trial. JMIR Res Protoc. 2022;11(5):e34990. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 63. Zenger J, Folkman J. The ideal praise‐to‐criticism ratio. Harvard Business Review Digital Articles. 2013;2‐5. Available from: https://hbr.org/2013/03/the-ideal-praise-to-criticism [Google Scholar]
  • 64. Michie S, West R, Campbell R, Brown J, Gainforth H. ABC of Behaviour Change Theories (ABC of Behavior Change): An Essential Resource for Researchers. Policy Makers and Practitioners: Silverback Publishing (Silverback IS); 2014. [Google Scholar]
  • 65. Michie S, West R, Finnerty AN, et al. Representation of behaviour change interventions and their evaluation: development of the upper level of the behaviour change intervention ontology. Wellcome Open Res. 2021;6(5):123. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66. Michie S, Johnston M. Theories and techniques of behaviour change: developing a cumulative science of behaviour change. Health Psychol Rev. 2012;6(1):1‐6. [Google Scholar]
  • 67. Cane J, O'Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci IS. 2012;7:37. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 68. Crawshaw J, Meyer C, Antonopoulou V, et al. Identifying behaviour change techniques in 287 randomized controlled trials of audit and feedback interventions targeting practice change among healthcare professionals. Implement Sci. 2023;18(1):63. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 69. Marques MM, Wright AJ, Corker E, et al. The behaviour change technique ontology: transforming the behaviour change technique taxonomy v1. Wellcome Open Res. 2023;17(8):308. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 70. Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6(1):42. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Learning Health Systems are provided here courtesy of Wiley

RESOURCES