Online Versus In-Person Screening, Brief Intervention, and Referral to Treatment Training in Pediatrics Residents

Abstract

Background

Pediatricians underestimate the prevalence of substance misuse among children and adolescents and often fail to screen for and intervene in practice. The American Academy of Pediatrics recommends training in Screening, Brief Intervention, and Referral to Treatment (SBIRT), but training outcomes and skill acquisition are rarely assessed.

Objective

We compared the effects of online versus in-person SBIRT training on pediatrics residents' knowledge, attitudes, behaviors, and skills.

Methods

Forty pediatrics residents were randomized to receive either online or in-person training. Skills were assessed by pre- and posttraining standardized patient interviews that were coded for SBIRT-adherent and -nonadherent behaviors and global skills by 2 trained coders. Thirty-two residents also completed pre- and postsurveys of their substance use knowledge, attitudes, and behaviors (KABs). Two-way repeated measures multivariate analyses of variance (MANOVAs) and analyses of variance (ANOVAs) estimates were used to assess group differences in skill acquisition and KABs.

Results

Findings indicated that both groups demonstrated skill improvement from pre- to postassessment. Results indicated that both groups increased their knowledge, self-reported behaviors, confidence, and readiness with no significant between-group differences. Follow-up univariate analyses indicated that, while both groups increased their SBIRT-adherent skills, the online training group displayed more “undesirable” behaviors posttraining.

Conclusions

The current study indicates that brief training, online or in-person, can increase pediatrics residents' SBIRT skills, knowledge, self-reported behaviors, confidence, and readiness. The findings further indicate that in-person training may have incremental benefit in teaching residents what not to do.

What was known and gap

Approximately one third of pediatric residencies offer training to prepare physicians to identify and address substance abuse, yet the impact of training on knowledge and skills often is not assessed.

What is new

Screening, Brief Intervention, and Referral to Treatment (SBIRT) training increased residents' knowledge, self-reported behaviors, and confidence.

Limitations

Single institution study, small sample, and lack of data for performance in practice reduce generalizability.

Bottom line

Online and in-person training increased residents' skills and confidence. In-person training was more effective for teaching residents what not to do.

Introduction

The Substance Abuse and Mental Health Services Administration (SAMHSA) recommends universal screening, brief intervention, and/or referral to treatment (SBIRT) for risky substance use as part of routine health care.¹ Pediatricians play an important role in the prevention of risky substance use and related problems among children and adolescents,^2–6 yet pediatricians often underestimate substance use and fail to appropriately screen for risk.^7,8 Providers report lack of confidence in their skills to manage substance abuse as a barrier,⁹ highlighting the need for quality training in SBIRT.

SBIRT includes several steps: (1) screening, or quickly assessing for substance use risk and the need for intervention or referral; (2) brief intervention, or increasing patients' awareness of the impact of substance use and enhancing motivation for behavioral change; and (3) referral to treatment, providing access to specialty care.¹⁰ The American Academy of Pediatrics (AAP) recommends pediatricians become knowledgeable about SBIRT, and 32% of pediatrics residency directors reported requiring any substance use content in their programs.¹¹

To address training needs, SAMHSA funded 17 medical residency SBIRT training cooperatives¹⁰ with training implemented in multiple residency programs through in-person training and/or online modules. Several studies have demonstrated the effectiveness of resident SBIRT training, including a recent study which demonstrated increases in knowledge and skills for internal medicine residents.¹² Significant time constraints in residency education make the addition of new content challenging for program directors; many have addressed this challenge by implementing computer-based training.¹³ There are limited studies comparing online to in-person residency training in medical education literature. Findings support equal or slightly better performance in learners trained online than that in peers with face-to-face instruction.^14,15 Evidence of the efficacy of online training in SBIRT could allow broader curriculum implementation. To date, no studies have compared the 2 approaches in a SBIRT curriculum. We compared online versus in-person training for SBIRT to determine whether the type of training impacts pediatrics residents' knowledge, attitudes, behaviors, and skills in implementing SBIRT, as assessed by pre- and posttraining standardized patient (SP) interactions.

Methods

Pediatrics residents at the University of Maryland Medical Center received SBIRT training as part of a SAMHSA-funded program. Training competencies were developed using motivational interviewing principles and SAMHSA guidelines.^11,16 Pediatric-specific information was consistent with the AAP's “Substance Abuse SBIRT for Pediatricians” policy statement.²

Measures

SBIRT skills were evaluated during taped interactions with SPs' pre- and posttraining. Tapes were coded using the MarylanD MD's Making a Difference (MD3) SBIRT coding scale,¹⁷ adapted for use with adolescent patients and used to evaluate 16 SBIRT-adherent (“desirable”) behaviors, 7 SBIRT-nonadherent (“undesirable”) behaviors, and 2 global skills (table 1).

TABLE 1.

Skills Assessed With the Adapted MD3 SBIRT Coding Scale

Residents completed self-reported knowledge, attitudes, and behaviors (KAB) questionnaires before and after training. The knowledge portion included 6 multiple-choice questions addressing SBIRT facts. The attitudes section consisted of 24 questions addressing substance use beliefs,^18,19 and the behaviors portion included 20 questions addressing the frequency with which residents performed substance use–related aspects of patient care for each substance type, using a 5-point Likert scale. The KAB also included 9 questions assessing confidence to perform SBIRT activities and 2 questions assessing readiness to screen and provide brief interventions, both using an 11-point Likert scale.

Procedures

All residents completed SP interviews pre- and posttraining within 3 weeks after SBIRT training. Immediately prior to the pretraining interview, residents completed a KAB questionnaire. Residents were then instructed to conduct the SBIRT protocol with a teenage SP who reported drinking alcohol and smoking cigarettes. Four teenagers were trained as SPs with identical patient information. All SP interactions lasted 10 minutes and were videotaped.

After the first SP interview, residents were assigned to either in-person or online training. Condition assignments were partially based on practical (ie, scheduling) factors and were not truly random. However, training assignment was not associated with any specific resident characteristics, level of experience, or other variables.

Posttraining SP interviews were conducted within 3 weeks following training and involved the same case. The posttraining KAB questionnaire was e-mailed to residents in both training groups 6 to 8 months posttraining to capture longer term changes in knowledge, attitudes, and self-reported behaviors.

Training Conditions

Online

After the pretraining SP interaction, residents in the online condition were sent instructions by e-mail for completing 5 online modules consisting of voice-narrated slideshow presentations, with a total run time of 120 minutes. Modules were developed by project leaders with expertise in SBIRT, were tailored to pediatrics,^20,21 and included case examples, videos, and links to additional resources (www.sbirt.umaryland.edu). Module completion was confirmed via mandatory 3-question assessments. All residents completed all 5 modules.

In-Person

Residents in the in-person group attended a 2-hour training session. The content was the same as online training but was delivered as a lecture by MD- and PhD-level trainers. Two example videos were shown, and residents completed role play exercises in pairs with instructor feedback.

Coding SP Tapes

Two graduate students with MD3 SBIRT coding scale experience independently coded the 40 pre- and 40 postresident SP interactions. A random sample of 16 tapes was independently coded by both coders to assess interrater reliability using a 2-way random model intraclass correlation coefficient (ICC), estimated separately for adherent behaviors, nonadherent behaviors, and global ratings. Coders were blinded to whether a tape was pre- or posttraining and which tapes were double-coded. Only audio was used in coding, following previously described procedures.²²

The study was reviewed and approved by the Institutional Review Board at the University of Maryland Baltimore County.

Statistical Analyses

Statistical analyses were conducted using SPSS version 22.0 software (IBM Corp). Two-way repeated measures multivariate analyses of variance (MANOVAs) were estimated to assess for differences in SP skill acquisition between training groups and over time. Dependent variables were adherent and nonadherent behaviors in 1 pair of analyses and global ratings of collaboration and empathy in another. If analyses indicated significant pre/post changes, univariate analyses of variance (ANOVAs) were estimated to determine which dependent variable(s) differences emerged. In addition, a series of 2-way repeated measures ANOVAs were estimated to assess for differences in KAB variables between training groups and over time (within-subject and pre versus post).

Results

Forty-two pediatrics residents received SBIRT training; 2 did not complete SP interviews and were excluded from analyses. There were 16 (40%) postgraduate year (PGY)-1s, 14 (35%) PGY-2s, and 10 (25%) PGY-3s distributed across the 18 residents in the online group and 22 in the in-person group. There were no statistically significant differences between the 2 training conditions on demographic variables.

SBIRT Skills

Interrater reliability for the 16 double-coded tapes was in the “excellent” range,²³ with ICC values of 0.95 for adherent behaviors, 0.96 for nonadherent behaviors, and 0.84 for global ratings. Distribution of the SP skills data was not significantly different from a normal distribution according to a Shapiro-Wilk test (P > .05).

Means and standard deviations of SBIRT skills are presented in table 2. There were no statistically significant differences between SBIRT skills in the randomized groups prior to training.

TABLE 2.

SBIRT Skill Rating on the MD3 SBIRT Coding Scale Pre- and Posttraining

There were no differences between groups in adherent versus nonadherent behaviors before or after training, but both groups showed significant improvement from pre- to posttraining (P < .001). Both groups increased their adherent skills from pre- to posttraining (F_1,38  =  32.98, P < .001) and did not differ by training type (f i g u r e). Nonadherent behaviors changed from pre- to posttraining. This change differed by training type (F_1,38  =  4.87, P < .05), with the online group displaying more nonadherent behaviors posttraining (mean  =  1.50, SD  =  1.86) than the in-person group (mean  =  0.68, SD  =  0.84) (f i g u r e ). The largest increase in nonadherent behaviors among the online group was for “lecturing and/or using medical jargon,” with 17% (3 of 18) of residents displaying this behavior in pretraining, compared to 39% (7 of 18) of residents in posttraining. Although the online group displayed fewer nonadherent behaviors pretraining, this difference was not statistically significant (table 2).

FIGURE.

Change in SBIRT-Adherent and -Nonadherent Behaviors Displayed During SP Interactions Pre- and Posttraining by Training Type

Abbreviations: SBIRT, Screening, Brief Intervention, and Referral to Treatment; SP, standardized patient.

There were no significant differences in global ratings (P > .05), levels of collaboration, or empathy displayed during SP interviews, either between training groups or from pre- to posttraining.

Knowledge, Attitudes, and Behaviors

A subsample of 32 residents completed both pre- and post-KAB surveys (15 in the online training group, 17 in the in-person training group, and 8 were omitted due to noncompletion of post-KAB survey). There were no significant differences between the 32 residents with complete data and the 8 residents without a postassessment on any preassessment measure, training assignment, pre- or post-SP ratings, or demographic variable, except that noncompleters were more likely to be men (38% versus 10%, P < .05).

There were no significant differences between any KAB variables pretraining for either group (table 3). For knowledge, self-reported behaviors, confidence, and readiness, findings indicated that both groups changed from pre- to posttraining but did not differ by group. For attitudes, there were no significant differences either between training groups or from pre- to posttraining.

TABLE 3.

Changes From Pre- to Posttraining in Residents' SBIRT Knowledge, SBIRT Confidence, Current Self-Reported Screening, BI Behaviors, and Self-Reported Readiness to Screen and Do BIs

Discussion

In this study, both in-person and online SBIRT training effectively increased pediatrics residents' SBIRT skills, knowledge, self-reported behaviors, confidence, and readiness. Further investigation, however, revealed a nuanced interaction of training type and time for undesirable behaviors; the online training group showed a slight increase whereas the in-person training group showed a reduction. This suggests that in-person training may be superior to online training in teaching residents what not to do. The reasons for this discrepancy are not clear, but it might be attributable to the role play, practice, and feedback that the in-person group received, which contributed to greater confidence or a stronger grasp of SBIRT goals.

Additionally, although specific SBIRT skills increased, neither group demonstrated a change in global ratings of collaboration or empathy, suggesting training did not impact overall communication style. This is not surprising given the brevity of the training, and is consistent with previous studies.^24,25

Another finding was that neither group's attitudes about substance use treatment changed from pre- to posttraining. This is likely due to the focus on skill acquisition rather than attitude change. It also is possible that there was a brief change in attitudes that was not sustained over 6 months.

This study has several limitations. We did not include a no-training control group that would have allowed us to rule out changes contributable to time only. Another limitation was the use of identical SP cases in pre- and postassessments, which could have contributed to practice effects. We did not perform a power calculation to determine the minimum number of subjects needed per group. The small sample size and single institution cohort raise questions about the generalizability of the findings. Finally, our study did not evaluate actual resident performance, and there is evidence to show that increased confidence does lead to improved clinical performance.²⁶

Conclusion

Pediatrics residents' SBIRT skills, knowledge, self-reported behaviors, and perceived confidence and readiness can be improved through both in-person and online training. Online training is feasible and does not require designated training space or teachers. There may be a benefit to in-person training in minimizing undesirable behaviors. Future research should explore the utility of online and in-person training to maximize SBIRT implementation.