Using Clinical Vignettes to Assess Quality of Care for Acute Respiratory Infections

Abstract

Objective

Overprescribing of antibiotics for ARIs is common. Our objective was to develop and validate a vignette-based method to estimate clinician acute respiratory infection (ARI) antibiotic prescribing.

Data sources/study setting

We surveyed physicians (N=78) and retail clinic clinicians (N=109) between January-September 2013.

Study design

We surveyed clinicians using a set of ARI vignettes and linked the responses to electronic health record data for all ARI visits managed by these clinicians during 2012. We then created a new measure of antibiotic prescribing, the comprehensive ARI management rate. This was defined as not prescribing antibiotics for antibiotic-inappropriate diagnoses and prescribing guideline-concordant antibiotics for antibiotic-appropriate diagnoses (and also included appropriate use of streptococcal testing for the pharyngitis vignettes).

Data collection methods

We compared the vignette-based and chart-based comprehensive ARI management at the clinician level. We then identified the combination of vignettes that best predicted comprehensive ARI management rates, using a partitioning algorithm.

Principal findings

Responses to 3 vignettes partitioned clinicians into 4 groups with chart-based comprehensive ARI management rates of 61% (N=121), 50% (N=47), 31% (N=12), and 22% (N=7).

Conclusions

Responses to 3 clinical vignettes can identify clinicians with relatively poor quality ARI antibiotic prescribing. Vignettes may be a mechanism to target clinicians for quality improvement efforts.

Introduction

There are approximately 130 million ambulatory visits per year in the US for acute respiratory infections (ARIs) such as pharyngitis, rhinosinusitis, and bronchitis.¹ ARIs are the most common symptomatic reason patients seek care in the US and account for half of all adult antibiotic prescriptions and three-quarters of all pediatric antibiotic prescriptions.² A large fraction of these prescriptions are inappropriate,³ resulting in increased antibiotic resistance, unnecessary adverse drug events, and increased health care costs. Despite considerable efforts such as the Centers for Disease Control and Prevention’s (CDC) Get Smart campaign,⁴ there have been only modest improvements in inappropriate antibiotic prescribing.⁵

A fundamental principle of performance improvement is that “You can’t fix what you don’t measure.” Prior work has demonstrated wide variation among physicians in their ARI antibiotic prescribing.⁶ Targeting clinicians who are most likely to inappropriately prescribe antibiotics for quality improvement efforts may be one method of decreasing inappropriate prescribing.

However, given the changing landscape of ARI care, interventions should not solely target physicians in primary care offices. Many patients are now receiving care in non-traditional settings, such as retail clinics. Retail clinics, located in drug stores and grocery stores and typically staffed by NPs, are an innovative delivery model that provides walk-in care for ARIs.^7,8 There are approximately 5.4 million yearly visits to the 1200 retail clinics in the US and almost two thirds of these visits are for ARIs.⁷

Clinicians’ antibiotic prescribing behavior – whether in a physician’s office or retail clinic – can be directly observed during patient encounters or gleaned via chart review. However, both of these approaches are limited by their feasibility given the amount of effort required to gather data. Assessing clinician-decision making via clinical vignettes has been promoted as a more efficient way to measure clinician quality because it avoids the time and expense of direct encounter observation or chart review.⁹ Another practical advantage is that case-mix adjustment is not necessary when using vignettes. Vignettes have been used extensively in the health care, and have been validated for measuring the management of a wide variety of conditions such as breast cancer,¹⁰ obstetric emergencies,¹¹ chronic obstructive pulmonary disease,¹² diabetes mellitus,¹² and depression,¹² as well as physician compliance with preventive care,¹³ but clinical vignettes have not been validated for ARI.

The goals of this exploratory study were to develop clinical vignettes to characterize clinicians’ antibiotic prescribing for ARIs. We developed a set of clinical vignettes which we deployed using a survey. We then linked vignette-based antibiotic prescribing to EHR-based antibiotic prescribing to determine which vignettes allowed us to identify clinicians with lower quality of antibiotic prescribing based on their actual antibiotic prescribing behaviors, including the decision to treat with antibiotics (and test when applicable), as well as the choice of antibiotics in concordance with evidence-based guidelines. To ensure broader applicability, we tested the tool in both physician offices and retail clinics.

Methods

Study population and data collection

We sampled clinicians in two settings: (1) primary care physicians’ offices in a large integrated health system that was affiliated with an academic medical center; and (2) a national retail clinic chain with clinics in 19 states. Clinicians in the physicians’ offices were all physicians with either an MD or a DO. Clinicians in the retail clinics were almost all nurse practitioners with a small number of physician assistants. The health system and retail clinic were chosen for feasibility reasons, given the affiliation of one of the authors and the willingness of a particular retail clinic chain to participate.

We obtained ele ctronic health record data for all adult ambulatory visits for ARIs to the clinicians at both sites during 2012. Based on prior work,¹⁴ we defined ARI diagnoses using the following ICD-9 codes: streptococcal pharyngitis (034.x), otitis media (381.x, 382.x), sinusitis (461.x), pneumonia (481.x, 482.x, 483.x, 485.x, 486.x), non-specific upper respiratory infection (URI; 460.x, 465.x), non-streptococcal pharyngitis (462.x), and bronchitis (466.x, 490.x, 491.21). We randomly selected 200 clinicians in each setting from those who had at least 25 visits for ARIs during the prior calendar year of 2012. The self-administered online survey was emailed to clinicians in January 2013 (physicians’ offices) and September 2013 (retail clinics). The clinicians were offered $50 as an incentive for participation. The initial email invitation was followed by two email reminders sent at 5-7 day intervals to those who had not responded.

The study protocol was approved by the Human Subjects Protection Committees at the University of Pittsburgh and Harvard University.

Clinical vignettes

In a review of the literature we did not identify any vignettes related either directly to antibiotic prescribing or to behaviors similar to antibiotic prescribing in primary care (e.g. imaging for lower back pain), so we created 9 vignettes. We used three sets of conditions with similar presenting symptoms: sinusitis/URI, pneumonia/bronchitis, and streptococcal pharyngitis/viral pharyngitis (see Supplemental Digital Content Table 1 for full list of vignettes). For each of three sets we created three vignettes: one where antibiotics and/or testing were clearly indicated; one with an “intermediate” indication for testing and/or antibiotics; and one in which antibiotics and/or testing were clearly not indicated. For sinusitis/URI symptoms, the three vignettes were persistent sinusitis (antibiotics indicated), non-persistent sinusitis (intermediate; antibiotics not indicated), and non-prolonged URI (antibiotics not indicated).¹⁵ For pneumonia/bronchitis symptoms, the three vignettes were pneumonia with an infiltrate on chest x-ray (antibiotics indicated), moderate acute bronchitis with a negative chest x-ray (intermediate; antibiotics not indicated), and mild acute bronchitis for which a chest x-ray is not indicated (antibiotics not indicated).¹⁶ For pharyngitis symptoms, the three vignettes were pharyngitis with Centor scores¹⁷ of 4 (antibiotics are indicated and can be given without testing; or can test and treat with antibiotics if the test is positive), 2 (intermediate; testing is indicated and antibiotics should be prescribed if the test is positive), and 0 (neither testing nor antibiotics indicated).¹⁸

For the six vignettes related to sinusitis/URI and pneumonia/bronchitis symptoms, we asked clinicians whether they would or would not prescribe antibiotics. For the three pharyngitis vignettes, we asked clinicians if they would prescribe antibiotics without testing, test and only prescribe antibiotics if positive, or neither test nor prescribe antibiotics. If clinicians indicated that they would prescribe antibiotics with or without testing, they were asked to type in their antibiotic choice as free text. All vignettes indicated clearly that the patient had no allergies and no comorbidities that would affect the decision to treat with antibiotics. For each respondent, we randomized the order in which the vignettes were presented.

In addition to the vignettes, the survey included questions on clinician specialty, gender, hours worked, and years since completing training.

Comprehensive measure of ARI antibiotic prescribing

We created a new quality measure, which we called comprehensive ARI antibiotic prescribing. ARI antibiotic prescribing is a commonly used measure, and is defined by the fraction of ARI visits that result in an antibiotic prescription. Such a measure is crude in that it does not assess the appropriateness of the antibiotic prescribed and the appropriateness of testing performed (e.g., Group A Streptococcus testing for pharyngitis). The new measure we created called comprehensive ARI management was defined as not prescribing an antibiotic for antibiotic-inappropriate diagnoses (defined as URI, bronchitis, non-streptococcal pharyngitis)¹⁴ or prescribing for antibiotic-appropriate diagnoses (sinusitis, pneumonia, streptococcal pharyngitis, otitis media)¹⁴ and choosing a guideline concordant antibiotic. Guideline concordance was defined as amoxicillin-clavulanate or amoxicillin for sinusitis,^15,19 a macrolide or doxycycline for outpatient pneumonia in a patient without significant comorbidity,²⁰ amoxicillin or penicillin for streptococcal pharyngitis,^18,21 and amoxicillin for otitis media.²² We also included testing for streptococcal pharyngitis as part of the measure (for the vignettes only), with the appropriateness of testing based on the Centor score as described above.

We measured comprehensive ARI antibiotic management for all clinicians using both electronic health record (EHR) data and vignette responses. We also measured overall ARI antibiotic prescribing which was defined as the fraction of ARI visits that were associated with an antibiotic prescription for use in sensitivity analyses.

Analysis

We assessed which vignettes were most predictive of EHR-based comprehensive ARI management rates. As a first step, we calculated vignette response scores, which was a crude sum of the vignette scores; each vignette for which a clinician provided comprehensive management was assigned a response of 1. We stratified the EHR-based comprehensive management rate by vignette score to assess the relationship between the two measures. We then calculated the Spearman correlation on the vignette response scores and EHR-based comprehensive management rates.

As a second step, we conducted a Classification And Regression Tree (CART) analysis²³ - a statistical technique that uses recursive partitioning analysis – to cluster the vignette responses into groups that best predict clinicians’ EHR-based comprehensive ARI management rates. The CART method involves the segregation of different values of the classification variables (the vignettes) through a decision tree composed of progressive binary splits. Every value of each predictor variable (vignette) is considered as a potential split threshold (or node), and the optimal split is selected based on impurity criterion (the reduction in the residual sum of squares due to a binary split of the data at that tree node, where a tree represents a specific set of vignette groupings). As a sensitivity analysis, we repeated this analysis using overall ARI antibiotic prescribing as the outcome.

Results

Our response rate was 30% overall – 34% (109/325) for retail clinicians and 27% (78/288) for physicians in office-based practices. We do not include respondents who started the survey but did not complete any of the vignettes or other survey questions in our response rate or analyses. Compared to responders, non-responders had significantly lower comprehensive ARI management rates (46% versus 52%, p=0.003). Compared to clinicians in physician offices, clinicians in retail clinics were more often women, had not been in practices as long, and worked fewer hours per week (Table 1). Retail clinicians had higher performance on the comprehensive ARI prescribing measure (69% versus 35%, p<0.001) (Table 1).

Table 1.

Characteristics of survey respondents

	All (N=187)	Physicians’ offices (N=78)	Retail clinics (N=109)	p-value
Gender (% female)	72%	41%	94%	<0.001
Average number of years in practice	13.2 ± 9.5	16.8 ± 10.3	10.6 ± 7.9	<0.001
Hours in practice per week  0-15  16-25  26-40  >40	3% 8% 61% 29%	1% 6% 37% 56%	4% 9% 78% 9%	<0.001
Degree  MD  DO  NP  PA		79% 21% - -	- - 97% 3%
Specialty  Internal medicine  Family medicine/practice  Medicine-pediatrics  Acute care		47% 52% 2% -	- 96% - 4%
Overall antibiotic prescribing rate	61%	53%	67%	<0.001
Comprehensive ARI management rate	54%	35%	69%	<0.001

Comprehensive ARI management in vignettes

Depending on the vignette, performance across clinicians on the comprehensive ARI management measure ranged between 35% and 95% (Table 2). The underlying deficits in quality - no antibiotic prescription or inappropriate antibiotic for antibiotic-appropriate diagnosis vs. antibiotic prescribing for antibiotic-inappropriate diagnosis – are summarized in footnotes to Table 2.

Table 2.

Comprehensive* ARI management based on response to vignettes, stratified by ARI type (N = 187 clinicians with 1,683 responses to vignettes)

There were also differences by condition in performance between retail clinicians and physicians (Supplemental Digital Content Tables 2a and 2b) across many of the vignettes. For example, physicians had higher performance on comprehensive management for pneumonia that requires treatment (largely due to the choice of specific antibiotic on the nurse practitioners’ part at retail clinics, as opposed to the decision to prescribe at all). Nurse practitioners had higher performance on comprehensive management for one of the acute bronchitis vignettes (physicians were more likely to prescribe antibiotics).

Partitioning clinicians based on vignette responses

The EHR-based comprehensive ARI management rate increased as the total score on the vignettes increased (Supplemental Digital Content Table 3). The correlation between the vignette response score and the EHR-based comprehensive management rate was 0.280, (p<0.05). The CART analysis of comprehensive ARI management based on vignette responses identified 3 vignettes that best partitioned clinicians based on their performance on the EHR-based comprehensive ARI management measure (Figure 1). Based on their correct management of the intermediate (moderate) acute bronchitis vignette, 121/187 clinicians were categorized into the highest performance group (61% EHR-based comprehensive management). Based on incorrect management of intermediate (moderate) acute bronchitis vignette and more severe pharyngitis (Centor score of 4), we categorized 7/187 into the lowest performance group (22% EHR-based comprehensive management). Based on their response to the persistent sinusitis vignette, the remaining clinicians were divided into two groups: correct response (47/187 clinicians, 50% EHR-based comprehensive management) and incorrect response (12/187, 31% EHR-based comprehensive management).

Figure 1.

CART analysis of vignette-based ARI management to predict EHR-based performance on comprehensive management

Of note, based on the CART analysis, we could have further divided the lowest performing group further, but given that this partitioned group was so small (N=7) and the clinical significance uncertain, we chose to not partition further in order to simplify the algorithm and thus make results as useful as possible to decision makers.

When applied separately to clinicians in retail clinics and clinicians in physicians’ offices, the (Supplemental Digital Content Figures 1a and 1b) the vignettes worked less well to partition retail clinicians because very few (N=2) responded incorrectly to the persistent sinusitis vignette and none responded incorrectly to the pharyngitis Centor score of 4 vignette.

As a sensitivity analysis, we also performed a CART analysis using overall ARI antibiotic prescribing rates as the outcomes (Supplemental Digital Content Figure 2). Clinicians could once again be partitioned into groups with different overall prescribing rates based on their performance on 3 clinical vignettes. When applied separately to clinicians in retail clinic and physicians’ offices, the same vignettes partitioned clinicians into similar groups (though again the absolute overall ARI antibiotic prescribing rates differed) (Supplemental Digital Content Figures 3a and 3b).

Conclusions

Clinical vignettes have a number of advantages for measuring the quality of care provided by clinicians. Once developed, they are low cost to deploy compared to the more labor-intensive approach of measuring quality through chart review or in-person observation. In addition, clinical vignettes avoid the need for case-mix adjustment, which is particularly important when comparing across clinicians or across institutions that may see highly variable patient populations. Finally, clinical vignettes are more feasible to compare performance across different institutions, as collecting data from multiple distinct EHRs is often infeasible or impractical. We created and identified a set of three clinical vignettes that distinguish clinicians in terms of comprehensive ARI management.

One common criticism of vignettes is that it is unclear whether vignette responses reflect actual clinical practice or merely physician knowledge. However, multiple studies in other aspects of clinical care have shown that vignettes reflect actual behavior,^24,25 including the use of computed tomographic or magnetic resonance imaging for back pain,²⁶ inadequate use of warfarin in atrial fibrillation,²⁷ and common outpatient conditions.^9,12 Our study further extends this important literature specifically to ARI care and a comprehensive measure of ARI antibiotic prescribing. Further study of these vignettes in other clinician populations would help to bolster or refute the generalizability of our findings. Such studies would also help to validate that clinicians behave in the same way in real-life when it comes specifically to antibiotic prescribing for ARIs as they do in hypothetical scenarios.

Given the high rates of inappropriate antibiotic prescribing and associated costs and adverse events, one mechanism to improve quality is to focus on low-quality clinicians. The ARI vignettes could be used to identify such clinicians. For example, a head of clinic could ask clinicians to respond to the three vignettes. Those with likely very poor comprehensive ARI management rates (~ 10% of clinicians in our study) could be targeted for further investigation. Once identified, clinic leaders could abstract a random sample of charts from these clinicians to confirm non-guideline concordant care and identify the underlying problem(s). This would allow for a better tailored and targeted intervention. For bronchitis, for example, this could consist of education on treatment guidelines or describing how best to discuss with a patient that antibiotics are not necessary.

In addition to providing a tool to identify low-quality clinicians, our study also contributes a novel approach of measuring antibiotic prescribing. In prior studies,^6,28,29 we and others have typically used overall antibiotic prescribing rate or broad-spectrum antibiotic prescribing rate to capture prescribing quality. However, these rates may be met with resistance from clinicians given that some ARI diagnoses may actually warrant antibiotic treatment. We created a new “comprehensive” ARI management rate that penalizes clinicians for prescribing for antibiotic-inappropriate diagnoses for which antibiotics are never indicated, while giving them credit for choosing guideline-concordant antibiotics when choosing to treat antibiotic-appropriate diagnoses. This rate could be used in future studies to capture the nuances of antibiotic prescribing and selection.

Our study does have some limitations. Although we were able to effectively partition clinicians into groups based on vignette responses, the differences in antibiotic prescribing (or non-comprehensive management) between the groups was sometimes small. Thus there may not be an obvious cut-off between poor quality and sufficient quality.

We observed some differences between physicians and retail clinicians in their responses to vignettes and antibiotic prescribing, but one set of clinicians was not consistently superior. In terms of vignette responses physicians and retail clinicians had similar performance on 4 vignettes, better performance by physicians on 2 vignettes, and better performance by retail clinicians on 3 vignettes. Overall antibiotic prescribing rate was better among physicians, however comprehensive ARI management was better among retail clinicians. These findings are consistent with prior literature where quality of care is similar at these two sites.^6,30

We also had a response rate of 30%, which may limit the generalizability of our findings as non-responders had worse performance on comprehensive ARI management. However, this finding could be of interest in and of itself, as failure to respond to vignettes may be an additional way to screen for poor quality prescribers given the poor quality of actual prescribing (based on EHR data) observed in non-respondents. An additional limitation is that we compared two very different settings, but only one health system and one chain of retail clinics for feasibility reasons, which may limit the generalizability of our findings given known regional differences in prescribing patterns.³¹ Evaluating the use of these vignettes in other health care systems and retail clinics may be helpful in confirming their usefulness.

In conclusion, we created and piloted the use of a new set of vignettes in ARI antibiotic prescribing. Response to these vignettes is correlated with a clinician’s management of ARI conditions in practice and we believe these vignettes could be used to quickly identify clinicians whose ARI management could be improved. Further studies will be needed to confirm whether our vignettes can identify and improve the performance of clinicians with lower quality.