A Randomized Educational Intervention Trial to Determine the Effect of Online Education on the Quality of Resident-Delivered Care

Abstract

Background

There is limited research on whether online formative self-assessment and learning can change the behavior of medical professionals.

Objective

We sought to determine if an adaptive longitudinal online curriculum in bone health would improve resident physicians' knowledge, and change their behavior regarding prevention of fragility fractures in women.

Methods

We used a randomized control trial design in which 50 internal medicine resident physicians at a large academic practice were randomized to either receive a standard curriculum in bone health care alone, or to receive it augmented with an adaptive, longitudinal, online formative self-assessment curriculum delivered via multiple-choice questions. Outcomes were assessed 10 months after the start of the intervention. Knowledge outcomes were measured by a multiple-choice question examination. Clinical outcomes were measured by chart review, including bone density screening rate, calculation of the fracture risk assessment tool (FRAX) score, and rate of appropriate bisphosphonate prescription.

Results

Compared to the control group, residents participating in the intervention had higher scores on the knowledge test at the end of the study. Bone density screening rates and appropriate use of bisphosphonates were significantly higher in the intervention group compared with the control group. FRAX score reporting did not differ between the groups.

Conclusions

Residents participating in a novel adaptive online curriculum outperformed peers in knowledge of fragility fracture prevention and care practices to prevent fracture. Online adaptive education can change behavior to improve patient care.

What was known and gap

Few studies have assessed change in behavior as a measure of effective educational interventions.

What is new

Residents exposed to an online curriculum in bone health outperformed peers on knowledge tests and evidence-based clinical practices.

Limitations

Single site, single specialty study reduces the ability to generalize from the findings.

Bottom line

An online curriculum can change resident clinical behaviors to improve care.

Introduction

Assessment can be a powerful formative tool, with several studies demonstrating that multiple-choice question curricula can improve knowledge acquisition and retention among medical trainees.^1–4 There is limited research to determine whether formative assessment can also change the behavior of medical trainees and improve patient care. We sought to determine the effectiveness of formative assessment to improve quality outcomes pertaining to care by resident physicians.

In 2010, approximately 43 million Americans were at risk for fragility fracture, while 2 million experienced a fracture due to low bone density.⁵ The United States Preventive Services Task Force (USPSTF)⁶ recommends screening all women aged 65 years and older with dual x-ray absorptiometry scans (DEXA); women aged 60 and older with risk equal to that of a 65-year-old white woman should also be screened. Fracture risk determination, either by T-score alone or in combination with the calculation of the fracture risk assessment tool (FRAX) score, then identifies patients who require treatment of high-risk osteopenia and osteoporosis with effective therapies (such as bisphosphonates and denosumab).^5–7 Despite this, a minority of patients receive effective care,⁸ and residents have deficiencies in knowledge and practice regarding fragility fracture prevention.⁹

We designed an intervention to address knowledge of bone health and the identification and management of patients at increased risk for fracture. We hypothesized that residents participating in this online adaptive education curriculum would demonstrate improved knowledge (as measured via a multiple-choice test) and improved care practices (as measured with chart review) when compared with their peers who received the standard curriculum alone.

Methods

Setting and Participants

The 52 junior (second-year) and senior (third-year) residents in the continuity clinic at Brigham and Women's Hospital, a large urban academic institution, were invited to participate. The decision to participate in the study did not impact the residents' standing in the program; all data were deidentified and stored securely. Participating residents were entered into a drawing to win 1 of 4 iPad devices. Institutional grant funds supported the project.

The continuity clinic practice includes approximately 80 resident physicians and 40 general internal medicine faculty physicians. The clinic cares for 19 000 patients with approximately 42 000 patient visits each year. Patients come from a variety of socioeconomic and cultural groups; the majority have multiple chronic medical problems, and many live in medically underserved parts of metropolitan Boston.

Baseline Audit and Power Analysis

An audit of the primary care practices of 5 randomly chosen interns at the end of their first year found a median of 5 female patients aged 60 years and older (range 4 to 6) per intern. Seventy percent of patients eligible for osteoporosis screening had received appropriate testing; screening rates varied widely by provider (range 40% to 100%). This audit established the need for an intervention and further informed study feasibility and number of residents to include for adequate power. We estimated that we would need to evaluate 133 patients to have an 80% power to demonstrate a change of 15% in the screening rate.

Intervention and Control Group Activities

All residents received a 1-hour case-based session on osteoporosis care and fracture prevention as part of the mandatory residency ambulatory curriculum. Residents in the control group received an e-mail containing a 25-item, case-based, multiple-choice self-assessment. Answers were delivered via e-mail after residents responded to the self-assessment.

Residents in the intervention group received the same 25 multiple-choice items delivered repeatedly via e-mail link over a 3- to 6-month period. Each link included 1 to 3 questions. After residents selected their best answer, immediate feedback was provided, including the correct answer, key teaching points, and answer explanations. Questions were e-mailed 1 additional time 28 days later if answered correctly; when incorrectly answered, questions were recycled at 14-day intervals until answered correctly on 2 separate occasions. The length of time for completion of the curriculum depended on the number of incorrect responses made by each resident.

Test Reliability and Validity

A women's health primary care physician blueprinted questions to learning objectives and authored questions using optimal question design.¹⁰ Two endocrinologists independently reviewed the 25 items for content validity. Three subsequent revisions were made in response to changes in calcium-prescribing practice during curriculum development. Following final content review, all 25 questions were administered to 33 volunteers to establish item difficulty and discrimination. Volunteers with varying experiences (from second-year medical students to attending endocrinologists) were intentionally recruited to establish scoring range and threshold performance levels for clinicians of varying experience. Psychometric data were calculated using the Integrity software system (Castle Rock Research Corp). Seven questions were eliminated at this stage: 3 questions had low corrected point biserial correlation indices (CPBR scores), and 4 were redundant in scope.

The knowledge instrument had good internal reliability (Kuder-Richardson Formula 20 of 0.814, mean difficulty of 0.48 for residents).¹¹ Item discrimination ranged from 0.12 to 0.75 (mean CPBR 0.39). Overall performance on the 25 question set improved with seniority: mean scores for medical students (n = 18), residents and generalist attendings (n = 9), and attending endocrinologists (n = 5) were 7.75 (31%), 17.5 (70%), and 19.5 (78%), respectively. The scores from the volunteer group were normally distributed.

Patient Outcomes

Patient-level outcome data were obtained from the Research Patient Data Registry at Partners Healthcare. Eligible patients included women older than 60 who had seen a participating resident in clinic for at least 1 continuity care visit, excluding acute appointments or hospital follow-up visits that typically do not include health maintenance discussions. The proportion of female patients appropriately screened for osteoporosis was the primary clinical outcome. As per USPSTF guidelines, appropriate screening entailed at least 1 DEXA scan for a patient after age 65, or after age 60 when additional risk factors were present. Residents “appropriately screened” patients if there was evidence of a DEXA completed at our center or if the resident note included results of a DEXA performed elsewhere.

Secondary outcomes included documentation of a FRAX score calculation for each osteopenic patient and appropriate bisphosphonate prescription for patients with osteoporosis or high-risk osteopenia. Residents received credit for calculating a FRAX score if their note documented a FRAX score or if the radiologist reported the FRAX score in the DEXA report. Appropriate bisphosphonate prescribing was determined by chart review of patients' medication lists and residents' documented plans. Credit for bisphosphonate prescribing was also given if residents documented cessation of bisphosphonate use after 5 years of treatment in appropriate patients.¹²

The study was approved by the Institutional Review Board at Partners Healthcare.

Statistical Design and Analysis

Statistical analysis was performed using R Project for Statistical Computing (The R Foundation) on an intention-to-treat basis. Knowledge outcomes were compared using the Student t test. Clinical outcomes were compared using the Fisher exact test.

Results

Participants

Fifty residents were recruited and randomly allocated to either a control or intervention group using random number generation (figure). One resident subsequently opted out of the intervention arm, leaving 24 intervention group residents and 25 control group residents. There were no significant differences between groups in resident seniority (10 [42%] and 11 [44%] junior residents and 14 [58%] and 14 [56%] senior residents, respectively) or sex (16 [67%] and 17 [68%] men and 8 [33%] and 8 [32%] women, respectively).

FIGURE.

Summary of Trial Design and Implementation

Abbreviations: MCQ, multiple-choice question; RPDR, Research Patient Data Registry; MRN, medical record number.

Participation

Of the 24 residents in the intervention curriculum, 21 residents completed registration and began to receive the e-mailed curriculum. Of those who participated, 6 residents (28%) completed the entire curriculum and 12 (56%) responded to at least 75% of the curriculum. An average of 64 total responses were required to complete the curriculum.

Knowledge Assessment

Twenty-one residents in the intervention (88%) and 20 control residents (80%) completed the postintervention knowledge assessment (table). Residents in the intervention arm correctly answered 13.2 questions on average, while residents in the control arm correctly answered 11.6 questions (73% versus 66% correct; P = .04; effect size = 0.65). A completers' analysis showed that the mean score for residents who completed at least 75% of the curriculum was 19.5 (78% correct).

TABLE.

Quality of Care Outcomes

Clinical Performance

In an intention-to-treat analysis, residents in the intervention group screened 216 of 227 patients for osteoporosis, while control group colleagues screened 206 of 231 patients (95.2% versus 89.2%, P = .02). The rate of inappropriate screening was low and similar in both groups, at 16 and 12, respectively (7.4% versus 5.8%, P = .56). Residents in the intervention group were significantly more likely to treat patients at high risk for fragility fracture with bisphosphonates, treating 57 of 75 appropriate patients compared with control group residents who treated 47 of 80 patients (76% versus 59%, P =  .03). FRAX scores were infrequently reported in notes by both groups, with intervention residents calculating scores 13 times when 67 were appropriate, and control residents calculating scores 15 times when 87 were appropriate (19% versus 17%, P = .89). Incorporating radiologists' FRAX score calculations increased this percentage for both groups equally (31 of 67 [46%] in the intervention group versus 36 of 87 [41%] in the control, P = .65; table).

Discussion

This randomized controlled study demonstrates that an evidence-based curriculum delivered online in a simple format with continuous self-assessment meaningfully impacted residents' knowledge of bone health care when measured 10 months after the start of the intervention. Additionally, intervention group residents provided higher quality of care in the year of their intervention: they screened more patients for low bone density, screened more appropriately, and effectively treated more patients at high risk for fracture.

The intervention was associated with a significant increase in the appropriate use of bisphosphonates. Given the +17% difference in bisphosphonate prescribing by the intervention group, the number needed to educate to prompt 1 appropriate bisphosphonate prescription is approximately 5.8. The number needed to treat to prevent 1 fracture has been cited as 15.¹³ Thus, in our cohort, in which residents cared for 3 osteoporotic patients on average, the number needed to educate to prevent 1 fracture is 29 residents. While the number needed to treat to prevent hip fracture is higher than to prevent any other fracture (number needed to treat =  90),¹⁴ the costs associated with hip fracture have been estimated at nearly $30,000 in the year following the fracture.¹⁵ Given approximately 11 740 junior and senior residents across US internal medicine residencies, if this relatively inexpensive intervention was disseminated to this cohort, the estimated cost savings attributable to hip fracture prevention alone could exceed $2 million.¹⁶

Several learning theories could explain these positive results. Repeated reminders of the content maximized its cognitive availability. Secondly, the use of multiple-choice questions capitalized on the testing effect.¹⁷ Finally, cognitive dissonance theory posits that people feel aversive tension when their attitudes and behaviors are in conflict; this tension then leads to increased self-awareness and behavior change to reconcile the 2 again. In this case, there is dissonance between residents' behaviors (providing care that does not meet guidelines) and attitudes (a desire to provide high-quality care). Such dissonance could encourage residents to change practice behaviors.¹⁸

Our study was limited to 1 cohort at a single center. Further research is needed to clarify the extent to which our findings are generalizable to other institutions or programs. In addition, while the majority completed at least 75% of the curriculum, the intervention might have been more effective if completion rates had been higher. Further studies could delineate which factors—number of questions completed, weeks participating in the curriculum, number of related patient encounters during curriculum participation—are most important in mediating behavior change.

Conclusion

This randomized, controlled educational intervention demonstrates that residents participated meaningfully in an adaptive online curriculum and that participation led to improved knowledge of bone health care at 10 months. Residents in the intervention were significantly more likely than their peers to adopt the taught behaviors, including screening for low bone density and appropriately treating patients at risk for fragility fracture with bisphosphonates, with all of these practices improving patient care. The degree of resident participation in the online education suggests that most trainees find the approach acceptable.