The protocol of improving safe antibiotic prescribing in telehealth: A randomized trial

Abstract

Background:

The CDC estimates that over 40% of Urgent Care visits are for acute respiratory infections (ARI), more than half involving inappropriate antibiotic prescriptions. Previous randomized trials in primary care clinics resulted in reductions in inappropriate antibiotic prescribing, but antibiotic stewardship interventions in telehealth have not been systematically assessed. To better understand how best to decrease inappropriate antibiotic prescribing for ARIs in telehealth, we are conducting a large randomized quality improvement trial testing both patient- and physician-facing feedback and behavioral nudges embedded in the electronic health record.

Methods:

Teladoc^® clinicians are assigned to one of 9 arms in a 3 × 3 randomized trial. Each clinician is assigned to one of 3 Commitment groups (Public, Private, Control) and one of 3 Performance Feedback groups (Benchmark Peer Comparison, Trending, Control). After randomly selecting ⅓ of states and associated clinicians required for patient-facing components of the Public Commitment intervention, remaining clinicians are randomized to the Control and Private Commitment arms. Clinicians are randomized to the Performance Feedback conditions. The primary outcome is change from baseline in antibiotic prescribing rate for qualifying ARI visits. Secondary outcomes include changes in inappropriate prescribing and revisit rates. Secondary analyses include investigation of heterogeneity of treatment effects. With 1530 clinicians and an intra-clinician correlation in antibiotic prescribing rate of 0.5, we have >80% power to detect 1–7% absolute differences in antibiotic prescribing among groups.

Discussion:

Findings from this trial may help inform telehealth stewardship strategies, determine whether significant differences exist between Commitment and Feedback interventions, and provide guidance for clinicians and patients to encourage safe and effective antibiotic use.

1. Background

Judicious antibiotic use reduces resistance and protects patients from unnecessary harm. Important advances in antibiotic stewardship have been achieved in outpatient settings, but little is known about stewardship in the rapidly growing telehealth sector. One hundred and sixteen million ambulatory antibiotics are prescribed for acute respiratory infections (ARIs) in the U.S.; about half of these are inappropriate, unnecessarily introducing risks. [1,2] Over two-thirds of all visits for ARIs may not require an in-person encounter. [3] Telehealth is an attractive option for ARIs, easing the burden on patients and reducing community exposure that might spread infections. [4] Studies of quality and safety have shown that telehealth performance is comparable to in-person primary care. [5-8] Currently, we know little about how to curtail overuse of antibiotics in telehealth. [9] Our prior pragmatic randomized trials have shown that CDC Core Element interventions constructed using insights from behavioral economics and social psychology can greatly reduce inappropriate prescribing in outpatient settings. [10-12] We previously found that clinician-initiated and patient-visible precommitments decreased antibiotic prescribing, but it is unknown whether the clinician commitment, patient viewing the commitment, the clinician being aware of the patient viewing the commitment, or some combination accounted for the improvement. [11] We also found that telling clinicians how they were performing relative to their “top-performing” peers significantly reduced inappropriate antibiotic prescribing, but it is unknown whether benchmarked feedback depends on negative framing for underperforming clinicians (“you are not a top performer”). [10,12]

2. Materials and methods

2.1. Overview

The trial for Improving Safe Antibiotic Prescribing in Telehealth will be implemented in a national telehealth provider, Teladoc^® Health. The service provides 24/7 access to clinicians, with patients typically opting to see the first available clinician rather than scheduling a visit for a future time, reducing the impact of clinician selection often seen in other settings. Beyond care for respiratory infections, Teladoc^® Health provides primary care, mental health care, and specialty care. The target population for this study is primary care clinicians. In this 3 × 3 randomized trial, we will adapt and test two aspects of CDC Core Elements, each with two variations. The primary outcome is the change in antibiotic prescribing rate for acute respiratory infection visits.

Within each of two intervention types—Commitment and Feedback—two variations will be assessed. The necessity of making commitments public (vs. private) will be tested by varying whether or not a clinician’s commitment to responsible antibiotic stewardship is shared with patients. The framing of performance feedback will be tested by varying whether this feedback focuses on labeling clinicians as “top performers” or “not top performers” vs. indicating whether or not they are included in a group trending toward lower antibiotic prescribing.

The intervention period is 12 months in length for all participants, with a 6-month follow-up period to measure the interventions’ persistence. All study procedures have been reviewed by the University of Southern California IRB (HS-20-00214) and determined to be exempt from IRB review.

2.2. Inclusion and exclusion criteria

Telehealth clinicians with prescribing privileges and one or more acute respiratory infection visits are included in the study. Eligible encounters include pediatric and adult telehealth visits for all Acute Respiratory Infections (regardless of specific etiology), including Sinusitis, Bronchitis, Influenza, Otitis Media, Nasopharyngitis, Upper Respiratory Infections, and COVID-19 [2] (see detail in Appendix A).

2.3. Randomization procedures

Fig. 1 displays the steps in our randomization procedures, which are applied simultaneously for Performance Feedback and Commitment intervention types in a 3 × 3 factorial design.

Fig. 1.

Intervention and state randomization procedures. Schematic is representative of proportions, not realized numbers.

For Commitment Interventions, a semi-cluster randomized design is required because the Public Commitment intervention includes patient-facing content that is delivered based on the US State (or District of Columbia) in which a patient resides. To address cluster-randomization by state, each clinician is assigned a modal state—-the state from which the greatest number of their patients originate—typically their state of licensure. States are first randomized to the Public Commitment group (⅓ of states) or non-Public Commitment groups (⅔ of states, Private Commitment and Control). State randomization was constrained to draw from the possible permutations of states that balance clinician count, visit volume, and mean antibiotic prescribing rate. Clinicians whose modal state is in the ⅔ of non-Public Commitment states are randomly assigned at the clinician level to Control or Private Commitment with equal probability, resulting in approximately ⅓ of clinicians assigned to each arm (see Appendix B, Table 1). For the Performance Feedback, all clinicians are assigned with equal probability to Benchmark Peer Comparison Feedback, Trending Feedback, and Control regardless of Commitment intervention assignment (see Appendix B, Table 2). Clinician-level randomizations are stratified to ensure balance across baseline characteristics including visit volume (consults per year), antibiotic prescribing rate for acute respiratory infections and COVID-19, and average member satisfaction (percent of responses “Outstanding” or “Good”).

2.4. Interventions

The telehealth service provider will leverage their existing electronic medical record and patient scheduling platform to implement this quality improvement initiative to reduce antibiotic prescribing. The personalized intervention messages are summarized in Table 2, and schematics of how interventions are visually presented to patients and clinicians in the Teladoc platform are presented in Appendix C. The design and prior related work is described in detail below. In brief, messages can appear in two contexts (1) the clinician’s home screen - the location where messages, lists of upcoming appointments, and other general notifications appear or (2) a message displayed in the encounter note screen (for patient-specific messages). On the home screen, clinicians assigned to Public or Private Commitment arms are initially forced to choose whether or not to commit before they can advance to encounters. To limit redundancy, other home screen messages can be acknowledged or dismissed for 30 days and will not reappear unless underlying information is refreshed. Encounter note banner messages appear in every encounter, as they may change depending on patients’ responses to notifications about antibiotic stewardship commitment.

Table 2.

Location and cadence of clinician and patient-facing messages (with display color for feedback messages).

	Message	Location and frequency*	Intervention content
1	Clinician-Facing Message: Standard of Practice (all groups, including control)	Monthly e-mail	Clinicians continue to receive a monthly scorecard over email providing feedback about antibiotic prescribing and other performance measures (See Appendix D)
2a	Clinician-Facing Message: Public Commitment Solicitation Message	Displayed in EHR Home Screen, Persistent until forced choice of opt-in or opt-out is complete.	Commitment Response 1) Record and share my commitment with my patients OR 2) I am not committed to the new guidelines, followed by a text box to type their name. (See Appendix E for full text).
2b	Clinician-Facing Message: Public Commitment-Committed	Displayed in EHR Home Screen, Persistent until acknowledged or updated (monthly)	“Thank you for your commitment to responsible antibiotic prescribing, your commitment is shared with your patients.”
2c	Patient-Facing Message: Public Commitment States - Notification of Clinicians’ commitment	Displayed at in the patient portal at the end of visit scheduling sequence	Patient Notification Response 1) Patient Acknowledged Commitment - “I understand my physicians’ commitment” OR 2) Patient Requested More Information - ” I do not understand and need more information” (See Appendix E for full text).
2d	Clinician-Facing Message: Clinician Alert with Patient Response to Commitment Notification	Displayed at top of screen in banner during patient encounter	Messages displayed according to Table 1.
3a	Clinician-Facing Message: Private Commitment Solicitation Message	Displayed in EHR Home Screen, Persistent until forced choice of opt-in or opt-out is complete.	Commitment Response 1) Record my commitment or 2) Do not include me in the commitment, followed by a text box to type their name. (See Appendix E)
3b	Clinician-Facing Message: Private Commitment-Committed	Displayed in EHR Horne Screen, Persistent until acknowledged or updated (monthly)	“Thank you for your commitment to responsible antibiotic prescribing.” (See Appendix E)
3c	Relevant Complaint	Displayed at top of screen in a banner in the encounter note	“This patients’ chief complaint may be relevant to your antibiotic stewardship commitment.”(See Appendix E)
4a	Clinician-Facing Message: Feedback: Benchmark Top Performers (best 3 deciles)	Displayed in EHR Horne Screen, Persistent until acknowledged or updated (monthly)	“You are a Top Performer in your group. Your antibiotic prescribing rate is X%. Top performers in your group typically prescribe antibiotics in X % of visits.” (See Appendix E)
4b	Clinician-Facing Message: Home Screen Feedback: Benchmark Not-Top Performers (worst 7 deciles)	Displayed in EHR Horne Screen, Persistent until acknowledged or updated (monthly)	“You are not a Top Performer. Top performers in your group typically prescribe antibiotics in Y% of visits. Follow the latest guidelines here.” (See Appendix E)
5a	Clinician-Facing Message: Feedback: Trending Top Performers (best 3 deciles)	Displayed in EHR Home Screen, Persistent until acknowledged or updated (monthly)	“Your antibiotic prescribing rate is X%. Stay in the growing number of clinicians in your group that have stopped inappropriate antibiotic prescribing. Continue to follow the guidelines here.” (See Appendix E)
5b	Clinician-Facing Message: Feedback: Trending Not-Top Performers (worst 7 deciles)	Displayed in EHR Home Screen, Persistent until acknowledged or updated (monthly)	“Your antibiotic prescribing rate is Y% (where Y% is the prescribing rate of the third decile). Don’t be left behind! Join the growing number of clinicians in your group who prescribe antibiotics only when clearly indicated. Follow the latest guidelines here.” (See Appendix E)
6	Control	No Messaging	No Messaging

^*Messages in the home screen are displayed with a button to acknowledge receipt. After being acknowledged, messages are not displayed for 30 days unless new information is included (as in the feedback messages).

Commitment (Public, Private, Control). Earlier studies left open the question of whether the patient-facing or clinician-facing components of public commitments to antibiotic stewardship are responsible for effectiveness. [11] Therefore, we will test two variations of the commitment intervention: one devoted to personal commitment and the other designed to include both patient and clinician-facing components, assigned based on clinicians’ modal state and patient state.

The Commitment interventions have two components – the first component is a one-time solicitation of the commitment from the clinician and the second is the recurring display of the commitment to clinicians (and to patients in the case of Public Commitment). The solicitation is a one-time forced choice. Patients originating in “Public Commitment” states are notified of the Teladoc commitment campaign when they request a consult. The display step has two physician-facing contexts. First, a reminder on the home screen that can be dismissed for 30 days and second, a reminder in the context of each encounter. In the case of Public Commitments, the patient’s response to the commitment notification is presented to the clinician in the context of the encounter. In the case of Private Commitments, there is simply a reminder in each encounter that the patient complaint may be relevant.

Patient response to clinician commitment (Public Commitment states only)

Clinicians randomized to the Public Commitment will make a commitment to evidence-based use of antibiotics that is shared with their patients. Clinicians randomized to the Private Commitment arm will be given the opportunity to make a personal commitment to evidence-based use of antibiotics that is not shared with their patients. For both arms, reminders of the commitment are displayed on the clinician’s home screen (See Appendix C Figs. A and B; Table 2 Rows 2a-3c).

To replicate the Public Commitment intervention in Meeker et al. [11] in the telehealth setting, patient-facing components of Public Commitments are required to be displayed before visits. Any personalized messages to patients before encounters are constrained based upon the patient state, and for licensing reasons, clinicians tend to treat patients primarily from their state(s) of licensure. Thus, the first level of randomization for the Commitment arm of the study is the state level (see 2.3 Randomization Procedures).

Due to recent relaxation in licensing restrictions, patients may be arbitrarily assigned on a first-come, first-serve basis to clinicians assigned to either of these arms. This results in encounter-level quasirandomization that will also allow us to explore, on an encounter-level basis, whether it is necessary for both clinician and patient to be aware of the commitment, or if clinician perception of patient awareness is sufficient to change practice. (See workflow diagrams for details of encounter-level assignments in Appendix C).

Patients in the ⅓ of states assigned to Public Commitment will see the following notification of clinicians’ commitment at the end of their visit request. (Clinician commitment is described in greater detail below.)

Patients will select: 1) I understand my physicians’ commitment OR 2) I do not understand and would like more information. Patients may also continue without making a selection or close the screen. Clinicians in this arm who are committed to safer antibiotic prescribing are notified about the patient response in the EHR during the visit.

Clinicians randomized to the Public or Private Commitments are asked to complete one of two commitments:

1. Public Commitment: Upon the first login after intervention assignment clinicians are presented one time with the following text requiring a response; one of the following options must be selected: 1) Record and share my commitment with my patients OR 2) I am not committed to the new guidelines, followed by a text box to type their name. For clinicians who opt in, the commitment nudge messages reminding them of their commitment are now displayed on their personal home screen and within a patient visit. Clinicians will receive a notice during the encounter as to how the patient responded to the notification of commitment.

Clinician attestation: Public Commitment

1. Private Commitment: Clinicians are presented with the following text and given the following options: 1) Record my commitment OR 2) Do not include me in the commitment followed by a text box to type their name. For clinicians who choose option 1, the “Thank you for your commitment” message is displayed on their personal clinician home page the first time they log in until it is dismissed, and every 30 days thereafter. In every encounter note page, clinicians see a message “This patient’s chief complaint may be relevant to your antibiotic stewardship commitment.” If a patient from a state assigned to receive notifications of clinicians’ Public Commitment has an encounter with a clinician that is in the Private Commitment arm, no information about the patient’s response to the commitment is displayed. These messages are displayed to the clinicians regardless of the patient’s chief complaint, based on the fact that more than half of encounters are for ARIs.

Clinician attestation: Private Commitment

Clinicians randomized to the Commitment Control arm will receive no intervention.

Section 2.4 describes details and control flow logic for each intervention.

Performance Feedback (Benchmark Peer Comparison, Trending, Control). Two framings of performance feedback are compared to each other and to controls. While prior work [10] used strongly negative benchmark framing for individuals not in the top performance decile (“You are not a top performer”), this language has been met with some resistance among clinical quality improvement professionals, frequently seen as peers themselves. Thus we also test a gentler, “trending” message that conveys the same information to individuals with the same performance cutoffs but highlights the increasing number of clinicians prescribing appropriately. Performance Feedback is based on overall performance benchmarks; clinicians with antibiotic prescribing rates in the lowest 3 deciles in each census region are designated to receive positive messages; the remaining clinicians receive messages recommending improvement. This fixed cutoff figure is a 31% acute respiratory infection visit antibiotic prescribing rate, the third decile based on prescribing data from January 2019–July 2021. The prescribing metrics and messaging are designed to align with enterprise-wide performance feedback practices. All messages provide a link to clinical practice guidelines based on recent literature about treatment of ARIs. [13-20] For performance feedback interventions, a minimum of 8 qualifying visits in the past 30 days is required for a clinician to receive a message, and the fixed thresholds are set according to baseline distributions of antibiotic prescribing rates over several years which included the recent COVID-19 pandemic. “Top Performer” status is inclusive in the case of ties where antibiotic prescribing is less than or equal to the 30th percentile (31%). This cutoff is fixed and will not change throughout the study period.

Clinicians are randomized with equal probability to Benchmark Peer Comparison Feedback, Trending Feedback, and Control (See Appendix C Diagram C; Table 2 Rows 4a-5b).

1. Benchmark Peer Comparison Feedback Message: If the clinician’s mean monthly antibiotic prescribing rate for ARIs is below the 3rd decile, where better performance is indicated by a position in a lower decile, clinicians will receive the following message in the EHR home screen: “You are a Top Performer. Your antibiotic prescribing rate is X%. Top performers in your group typically prescribe antibiotics in X% of visits. Continue to follow the guidelines here.” If the clinician prescribing rate is above the 3rd decile, where better performance is indicated by a position in a lower decile, clinicians will receive the following message: “You are not a Top Performer. Top performers in your group typically prescribe antibiotics in Y% of visits. Follow the latest guidelines here.”

2. Trending Feedback Message: If the clinician’s mean monthly antibiotic prescribing rate for acute respiratory infections is equal to or below 31% (fixed cutoff based on baseline data 3rd decile), where better performance is indicated by equal to or below 31%, clinicians will see the following message with a link to the clinical practice guidelines: “Your antibiotic prescribing rate is X%. Stay in the growing number of clinicians in your group that have stopped inappropriate antibiotic prescribing. Continue to follow the guidelines here.” If the clinician’s mean monthly antibiotic prescribing rate for ARIs is >31%, where better performance is indicated by a lower percentage, clinicians will see the following message with a link to the clinical practice guidelines: “Your antibiotic prescribing rate is Y%. Don’t be left behind! Join the growing number of clinicians in your group who prescribe antibiotics only when clearly indicated. Follow the latest guidelines here”.

Regardless of intervention assignment, all clinicians will also continue to receive a standard monthly scorecard containing a personalized performance report (a month by month detail of individual performance over the past 12 months) (Appendix D) over email that provides feedback about antibiotic prescribing and other performance measures. The scorecard indicates “Top Performer Goal 15-31% over 6-12 mo” for antibiotic prescribing.

2.4.1. Intervention workflows and messaging

Intervention messages are displayed in two places in the EHR, the clinician EHR home screen and the patient visit note in the context of a particular patient.

Fig. 2 displays the Clinician EHR Home Screen commitment attestation and performance feedback messaging in green. Banner (in yellow) displays on the Home Screen with the Commitment message above the Performance message, if applicable.

Fig. 2.

Clinician EHR home screen.

Table 1 outlines which home screen and patient visit note messages are visible in the Commitment arm based on clinician commitment status and patient response status. Table 2 displays the Message name, Location and Frequency, and Content for all study arms. All table cells are colored to reflect the color schemes used for messaging in the EHR platform.

Table 1.

Logic and content of public and private commitment messages.

Clinician Intervention & Commitment Status	Clinician Specific Message on Home Screen*	Patient Response Status
		Encounter note from a commitment state	Patient does not respond to notification	“I understand”	“I would like more information’
		Patient-Specific Clinician Message at Top of Visit Note
Public Commitment - Commitment Pending	Solicitation Message, Public Commitment	No Message	No Message	No Message	No Message
Public Commitment - Declined Commitment	No Message	No Message	“The patient has been shown the providers’ commitment.”	“Your patient has been shown the providers’ commitment to responsible antibiotic prescribing.”	“Your patient requested more information about your commitment to antibiotic stewardship if relevant.”
Public Commitment - Committed	“Thank you for your commitment to responsible antibiotic prescribing. Your commitment is being shared with your patients.”	“This patient’s chief complaint may be relevant to your antibiotic stewardship commitment.”	“Your patient has been shown your commitment to appropriate antibiotic prescribing.”	“Your patient acknowledged your commitment to responsible antibiotic prescribing.”	“Your patient requested more information about your commitment to antibiotic stewardship if relevant.”
Private Commitment - Commitment Pending	Commitment Solicitation Message (Private Commitment)	No Message	No Message	No Message	No Message
Private Commitment - Declined Commitment	No Message	No Message	No Message	No Message	No Message
Private Commitment - Committed	“Thank you for your commitment to responsible antibiotic prescribing.” (i.e. Private Commitment Attestation Confirmation/Reminder)	“This patient’s chief complaint may be relevant to your antibiotic stewardship commitment.”	No Message	No Message	No Message
Control	No Message	No Message	No Message	No Message	No Message

^*Messages on the home screen are displayed with a button to acknowledge receipt. Once a message has been acknowledged, it will not display again for 30 days.

2.5. Underlying behavioral science rationale

The intervention design draws on several principles of behavioral and social science. These are summarized in Table 3.

Table 3.

Rationale and descriptions of interventions.

Mechanism	Citation of record	Relevant intervention components
All Interventions
Availability and salience of judicious antibiotic prescribing	Taylor SE & Fiske ST 1978 [21]	Messages periodically displayed in patient- and clinician-facing user interfaces
Commitment Interventions
True patient demand for antibiotics	Feather 2021 [22]	Patient Messages describing commitment of clinicians to safe practices. Patients in states assigned to Public Commitment will see the clinician’s commitment before the visit. (See Appendix E Fig. 9b)
Patients do not engage with guidelines and clinicians’ commitment to antibiotic prescribing	S. Shy am Sundar 2012 [23]	Patients interact with content and can communicate in meaningful ways with clinicians to “acknowledge commitment” or “request more information about the commitment” (subsequently displayed to clinicians)
Create a sense of personal/public accountability; draw on reputational concerns.	Heider 1946 [24] Cialdini & Goldstein 2004 [25] Bern 1972 [26]	Private Commitment Clinicians will be given the following options: 1) Record my commitment OR 2) Do not include me in the commitment, followed by a text box to type their name.
Consistency between words and actions	Lerner & Tetlock 1999 [27]	Public Commitment Clinicians will be given the following options: 1) Record and shore my commitment with my patients OR 2) I am not committed to the new guidelines, followed by a text box to type their name.
Perceived patient demand for antibiotics	Coenen et al. 2006 [28] Kohut et al. 2020 [29]	Clinicians see patients’ responses to concerns.
Performance Feedback Interventions
Correcting or changing perceptions of normative behavior	Schultz et al. 2007 [30]	People change their behavior based on how they think compared to descriptive and/or injunctive social norms.
Attraction to trending social norms	Mortensen et al. 2019 [31] Sparkman & Walton 2017 [32]	Individuals want to join a growing group of people engaging in a particular behavior (even if it is the minority).
Desire to obtain and retain exclusive/elite status and designations	Gallus 2017 [33]	“You are a top performer, you are not a top performer”
“Backsliding” among Top Performers due to perceptions of performance above norms	Brandon et al. 2019 [34] Schultz et al. 2007 [30]	“You are a top performer” Injunctive Norm
Perception that feedback is not professionally or socially functional	S. Shyam Sundar 2012 [23] Joinson et al. 2009 [35] Joinson et al. 2009 Sundar et al. 2014 [35,36] Joinson et al. 2009 Sundar et al. 2014	Interactive acknowledgement of messages.
“Backsliding” due to habituation to repeated content	Hall & Rodriguez 2017 [35-37]	Minimize repetitive content.
Inertia/Resistance to change in prescribing practices	S. Shyam Sundar 2012 [23] Joinson et al. 2009 [35]	Benchmark Peer Comparison Feedback “You are a Top Performer. Your antibiotic prescribing rate is X%. Top performers in your group typically prescribe antibiotics in X% of visits.” “You are not a Top Performer. Top performers in your group typically prescribe antibiotics in Y% of visits.”
	Mortensen et al. 2019 [31] Sparkman and Walton 2017 [32]	Trending Feedback “Your antibiotic prescribing rate is X%. Stay ih the growing number of clinicians in your group that have stopped inappropriate antibiotic prescribing.” “Your antibiotic prescribing rate is Y%. Don’t be left behind! Join the growing number of clinicians in your group who prescribe antibiotics only when clearly indicated.”

3. Measures

3.1. Primary outcome measure

The primary outcome is the change in antibiotic prescribing rate during the intervention period compared to baseline for Acute Respiratory Infections. Analysis compares changes in intervention to controls. Acute respiratory infection visits are identified using the International Classification of Diseases, version 10 (ICD-10) codes including: non-specific upper respiratory infections, otitis media, sinusitis, pharyngitis, bronchitis, influenza, and COVID-19. [38]

3.2. Secondary outcome measure

As in many primary care encounters, microbiology is not available in telehealth to ascertain etiology before treatments are ordered. Yet guideline concordance between documented diagnosis and treatment remains an important goal of both stewardship and safety. Professionally endorsed diagnostic guidelines that have been validated in both telehealth and in-person care may apply and are included in the clinical practice guidelines we provide. [39,40] For this reason, the secondary outcome measure is change in inappropriate antibiotic prescribing rate for documented ARI type. Over the 12 month study period, we will assess change in inappropriate antibiotic prescribing rate for acute respiratory infections where antibiotics are never appropriate based on International Classification of Diseases, version 10 (ICD-10) codes as defined in Chua et al. [2] as well as COVID-19 (U07.1). Details of value sets for drugs and ICD-10 codes for this project can be found in the technical Appendix A.

3.3. Diagnostic shifting

To investigate the possibility that interventions led to diagnosis shifting (i.e., changes in clinicians’ diagnostic coding habits), we will test whether potentially antibiotic-appropriate acute respiratory tract infection diagnoses (e.g., pneumonia, chronic sinusitis) increased as a proportion of all acute respiratory tract infection diagnoses in a way that differs across intervention arms. [1]

3.4. Data collection and management

Data will be drawn from the Teladoc^® Electronic Health Record, including orders for antibiotics and encounter information.

3.5. Statistical analysis plan

Hierarchical linear mixed-effects models will be used to calculate adjusted odds of antibiotic prescribing across the variable set with a clinician random intercept. The unit of analysis will be the encounter, with splines for each intervention assignment that are equal to 0 before the intervention is activated and the number of days since the intervention was initiated otherwise, as well as a time covariate. Primary analysis models will include an indicator for clinician assignment to each of the interventions, using the control condition as a reference variable and indicators for whether the patient was alerted of the commitment. The equation is as follows:

$$\pi ={\beta }_0+{\beta }_{TIME}{x}_{ij}+{\beta }_{TX}{x}_{ij}+{\beta }_{TIME\text{-}TX}{x}_{ij}+{\epsilon }_i+{\eta }_j$$

where $\pi =ln\left(\frac{p}{1-p_{ji}}\right)$ and $p_{ij}$ is the indicator an antibiotic was prescribed by provider $\mathrm{j}$ at the i^th visit. A linear spline represents the number of days elapsed (or until) the intervention was activated (${\beta }_{TIME}$) the “time in treatment”, equal to 0 before the intervention and days elapsed after the intervention (${\beta }_{TIMETX}$), and an indicator for the treatment assignment (${\beta }_{TX}$) as well as interaction terms. With this model, we control for the fact that each clinician may receive zero, one, or two interventions.

We will assess the significance of each model coefficient independently, as well as omnibus tests to evaluate (a) the effectiveness of any Feedback or Commitment intervention compared to controls and (b) the effectiveness of each variation of Feedback and Commitment interventions compared to the other variation. Secondary analyses will include additional interaction terms to assess whether there are measurable interactions between Feedback and Commitment indicators. We will also assess changes in inappropriate antibiotic prescribing for the subsets of encounters with diagnoses where antibiotics are not indicated, including COVID-19. [2] We will also conduct analyses that investigate the extent to which each message exposure and cumulative exposure to messages influence antibiotic prescribing. To protect patient privacy, analyses are conducted in the Telehealth Service Provider environment. De-identified results as well as summary tables will be shared with USC for interpretation and further analysis. No identified data is transferred or disclosed.

3.6. Sample size and power calculations

The study is a 3 × 3 factorial design, with 3 groups in the Commitment Intervention (Public, Private and Control) and 3 groups in the Feedback Intervention (Benchmark Peer Comparison, Trending, and Control). For the purpose of power analysis, we assume independence between the Commitment interventions and the Feedback interventions. Our primary outcome is the change in antibiotic prescribing rates between each intervention group and controls, assuming independence between Commitment and Feedback interventions (additive effects). We conservatively estimate 510 clinicians per group (i.e. 255 in each cell of the 3 × 3 design). That is, the power analysis assumes we will have 510 clinicians in the Public Commitment group, 510 in the Private Commitment group, and 510 in the Commitment Control group, with the same distribution in Benchmark Peer Comparison, Trending Feedback, and Control. The baseline intra-clinician correlation in prescribing rate of 0.5 was used for power calculations (See Appendix G). With these assumptions we have 80% power to detect differences between groups ranging from 1% to 7% absolute percent differences in antibiotic prescribing rates. Secondary outcomes, including differences between active intervention arms, will require adjustment for multiple comparisons. [41]

3.7. Intervention fidelity monitoring plan

Using the audit logs and functionality tests developed during engineering, we will measure how frequently the interventions are triggered. Appendix F shows table shells for the measures we will monitor to verify intervention delivery, including tracking interface interactions (click tracking) and message displays for each of the intervention components. In quarterly quality control checks, clinician testers assigned to each of the possible interventions will provide screenshots demonstrating the messages are being delivered and displayed as expected. Testing clinicians will not be included in the analysis.

3.8. Safety assessment plan

Data for safety assessments will be drawn from the Teladoc^® EHR, customer surveys, and linked insurance claims provided by payers for a subset of Teladoc patients. The Data and Safety Monitoring Board (DSMB) convened on October 30, 2020 to review our study design, interventions, and safety assessment protocol. The DSMB approved the following patient safety metrics as stopping criteria: (1) Significant elevations in self-reported visits for untreated bacterial infections, (2) Differences between intervention and control groups for repeat Teladoc^® encounters within 7 days of index with same issue, and 3) Differences between intervention and control groups for patients referred to an ED setting or 911 protocol within a week of the index visit (See Appendix H). We will also actively monitor data for significant differences in clinician prescribing practices and patient self-reported case resolution and satisfaction between intervention and control groups. The DSMB will convene every 6 months throughout the course of the study. We will perform an interim analysis of safety data in a blinded fashion to evaluate adverse event data and compare the rates of monitoring measures after 6 months of data has been collected. There is no additional interim analysis planned for this study as it is minimal risk.

4. Discussion

Findings from this randomized controlled trial may help inform antibiotic stewardship strategies for telehealth care delivery and help answer key behavioral science questions related to curbing antibiotic overuse. Telehealth is a novel setting for this type of study–encounters with dedicated telehealth service providers may be distinctive from primary care with respect to patient demand and relationships, and telehealth workflows create opportunities to precisely deliver and test decision support nudges and feedback.

This study is distinctive from other studies of clinical decision support tools because telehealth allows factorial randomization to both physicians and patients using the telemedicine EHR and patient portal, respectively.

We will better understand how Commitment and Feedback antibiotic stewardship interventions, demonstrated in primary care, translate to the telehealth setting. Additionally, we will better understand if there are differences between implementation details of these interventions. For example, if Public Commitment, wherein clinicians are aware that patients see their stewardship attestation, is more effective than Private Commitment, our findings will be consistent with findings showing that public commitments improve upon goal attainment in comparison to private commitments. [42] Similarly, if the two variations of feedback messages are equally effective, but distinct from controls, it may alleviate the need to employ stronger, negative language in feedback messages. Labeling a clinician as “Not a Top Performer” may not be necessary if gentler language about being below trending is effective. If interventions are not effective, additional strategies may be required to overcome distinctive aspects of the telehealth setting, most notably the absence of personal, face-to-face communications and social interactions between patients, clinicians, and their colleagues responsible for stewardship.

Abbreviations:

CDC

Centers for Disease Control and Prevention

ARI

Acute Respiratory Infection

Appendix. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.cct.2022.106834.

Data availability

No data was used for the research described in the article.