Skip to main content
. 2016 Nov 30;2(2):e16. doi: 10.2196/mededu.6288

Table 3.

Summary of findings of included studies by primary information and communication technology (ICT) intervention.

ICT
intervention		 Study Interventions Outcome(s) Effect size Conclusion
Website





Balamuth, 2010 [22] Web-based 1-page summary sheet of guidelines (n=128)
Weblink to guidelines (n=109)		 Knowledge: correctly diagnosed patients
ORa (95% CI)		 0.82 (0.49-1.4) No statically significant difference between 2 groups in correctly diagnosing patients according to guidelines. Participants using the Web-based 1-page summary reported that the supplemental materials were more simple to use when compared with the weblink group.



Perceived ease of use: simplicity of supplemental materials
OR (95% CI)		 6.1 (2.8-13.6)

Bell, 2000 [23] Self-study Web-based guidelines (n=79)
Print-based guidelines (n=83)		 Knowledge: median (95% CI) score (out of 20) after immediate posttest Web-based: 15.0 (14.0-15.0)
Print based: 14.5 (14.0-15.0) P=.20		 No statistically significant difference in knowledge at immediate posttest or after 4-6 months. Web-based guideline users were more satisfied with learning.



Knowledge: median (95% CI) score (out of 20) after 4-6 months Web-based: 12.0 (11.0-13.0)
Print based: 11.0 (10.0-12.0); P=.12)



Perceived ease of use: median (95% CI) learner satisfaction scores (range 5-20, higher = better) Web-based: 17.0 (16.0-18.0)
Print-based: 15.0 (15.0-16.0); P<.001

Schroter, 2011 [17] Website with educational modules (n=48)
Waiting list (n=33)		 Knowledge: mean % change (SD) from baseline knowledge at 4 months Web-based plus Web material: 47.4% (12.6) to 66.8% (11.5)
Web-based material only: 47.3% (12.9) to 67.8% (10.8); P=.19		 No statistically significant differences in knowledge change or usability between the 2 groups. Participants in Web-based tool plus Web material group found it to be useful. Usefulness was not measured in the other group.



Perceived usefulness: % of participants who reported the tool to be very useful/useful Web-based plus Web material: 77%
Web-based material only: NRb

Sassen, 2014 [24] Website with educational modules (n=48)
Waiting list (n=33)		 Intention to use material to educate patients: mean (SD) score out of 7 (higher = easier) at baseline and 12 months Website: 6.25 (1.00), 6.06 (1.11)
Waiting list: 5.87 (1.15), 6.02 (0.91), P=.12		 No statistically significant differences in intention to use and barriers between interventions groups at 12 months.



Barriers to using the material to educate patients: mean (SD) score out of 7 (higher = easier) at baseline and 12 months Website: 3.11 (1.17), 3.18 (1.12)
Waiting list: 2.78 (1.01), 2.63 (0.96), P=.46

Wolpin, 2011 [25] Website enhanced learning (additional case studies) (n=33)
Website with usual care instructions (same content, without case studies) (n=36)		 Knowledge: mean (SD) score % on knowledge content of CPGsc pretest and immediate posttest Overall (pooled both groups): 79.28% (12.17), 82.32% (13.84), P=.10
Website (enhanced) 78.18% (11.1), 79.39% (15.0)
Website (usual): 80.28% (13.2), 85.0% (12.3)		 No statistically significant difference in knowledge or satisfaction at posttest between intervention groups. No statistically significant differences were seen between interventions groups for both outcomes.



Perceived ease of use: overall satisfaction with learning experience, mean (SD) score (1-5, higher = very satisfied), pretest and immediate posttest Overall (pooled both groups): 4.08 (0.860)
Website (enhanced) 78.18 (11.1), 79.39 (15.0)
Website (usual): 80.28 (13.2), 85.0 (12.3), P=.13
Computer software




Bullard, 2004 [26] Wirelessly networked mobile computer program (n=10)d
Desktop computer program (n=10)d 		Perceived usefulness: “impact on efficiency” mean (95% CI) score out of 7 Wireless: 3.2 (2.6-3.8)
Desktop: 4.3 (4.0-4.6), P=.02		 Statistically significant greater satisfaction for several items (“impact on efficiency,” “increase use of CPGs,” and “saving time”) when using the wireless computer compared with the desktop computer. Other satisfaction items such as “configuration,” “availability,” “reduced communication with staff and patients,” and “accessibility” did not show statistically significant differences (results not shown). Participants appeared to be indifferent regarding the usability of the wireless computer for their efficiency.



Perceived usefulness: “increased use of CPGs” mean (95% CI) score out of 7 (7 = excellent) Wireless: 4.1 (3.6-4.6)
Desktop: 3.5 (2.9-4.0), P=.03



Perceived usefulness: “wireless computer program made participant more efficient,” mean (95% CI) score out of 7 (7 = strongly agree) Wireless: 3.30 (2.33-4.27)
Desktop: NR

Butzlaff, 2004 [27] CPGs via CD-ROM/Internet (n=53)
No intervention (n=66)		 Knowledge: median (IQRe) score out of 25 at baseline CD/Internet: 13 (12-16)
No intervention: 13 (10-15.25), P=.40		 There was no statistically significant difference between intervention groups at baseline and ~70 postintervention in knowledge scores.



Knowledge: median (IQR) score out of 25 at ~70 days posttest CD/Internet: 15 (12-17)
No intervention: 13 (11-15.25), P=.10

Jousimaa, 2002 [28] CD-ROM computer-based guidelines (n=72)
Textbook-based guidelines (n=67)		 Skills: compliance with CPGs, “laboratory examinations,” OR (95% CI) 1.07 (0.79-1.44) There was no statistically significant difference between intervention groups for compliance with CPGs for laboratory, radiological, or physical examinations.



Skills: compliance with CPGs, “radiological examinations,” OR (95% CI) 1.09 (0.81-1.46)



Skills: compliance with CPGs, “physical examinations,” OR (95% CI) 0.74 (0.51-1.06)


















































Web-based workshops




Epstein, 2011 [20] Web-based didactic education session/workshop (n=27)
No intervention (received intervention after 6 months) (n=22)		 Skills: compliance with CPGs, “use of parent ratings of ADHD[f] during assessment,” mean % change from baseline at 6 months Web: 23.8%
No intervention: 5.7%, P=.03		 Statistically significant changes from baseline to 6 months were seen among participants complying with CPG-recommended ADHD care practices, with the exception of 1 recommendation, “Use of parent ratings of ADHD to monitor treatment responses” (results not shown).



Skills: compliance with CPGs, “use of teacher ratings of ADHD during assessment,” mean % change from baseline at 6 months Web: 22.6%
No intervention: 6.0%, P=.04



Skills: compliance with CPGs, “use of [Diagnostic and Statistical Manual of Mental Disorders (Fourth Edition)] ADHD criteria during assessment,” mean % change from baseline at 6 months Web: 47.3%
No intervention: 17.9%, P=.03



Skills: compliance with CPGs, “use of outside provider for ADHD diagnosis,” mean % change from baseline at 6 months Web: –60.7%
No intervention: –10.7%, P<.001



Skills: compliance with CPGs, “use of teacher ratings of ADHD to monitor treatment responses,” mean % change from baseline at 6 months Web: 38.7%
No intervention: 6.3%, P=.003

Fordis, 2005 [29] Live Web-based CMEg workshop (n=51)
Web-based CME workshop (n=52)
No intervention (n=20)		 Knowledge: the 2 active CME interventions combined: mean % change (95% CI) from baseline to immediate posttest 31.0% (95% CI 27.0%-35.0%), P<.001 A statistically significant improvement in knowledge was seen over time for both Web-based interventions groups. A statistically significant decrease in appropriately screening patients was seen in the live Web-based CME group at 12 weeks posttest compared with baseline. No statistically significant differences were seen for screening patients between interventions groups. There was a statistically significant increase in the proportion of patients appropriately treated by the Web-based CME group compared with the live CME and control groups. Participants in the Web-based interventions were satisfied with the learning experience.



Knowledge: the 2 active CME interventions combined: mean % change (95% CI) from baseline to 12 weeks posttest 36.4% (95% CI 32.2%-40.6%), P<.001



Knowledge: the 2 active CME interventions combined: mean % change (95 CI) from immediate posttest to 12 weeks posttest 5.4% (95% CI 2.6%-8.2%)



Skills: patients appropriately screened for dyslipidemia, mean % change (95% CI) from baseline to 12 weeks postintervention Live Web-based: −3.3 (−5.9 to −0.7)
Web-based: −0.1 (−2.9 to 2.6)
No intervention: −0.8 (−3.5 to 1.8), P=.24



Skills: patients appropriately treated for dyslipidemia, mean % change from baseline to 12 weeks postintervention Live Web-based: −1.1 (−4.9 to 2.7)
Web-based: 5.0 (1.0-9.1)
No intervention: 1.2 (−2.8 to 5.1), P=.04



Perceived usefulness: % of participants satisfied with the learning experience Live Web-based: 100% (49/49)
Web-based: 94% (44/47)
No intervention: NR
Computerized decision support system



Gill, 2011 [30] EHRh-based clinical decision support (n=53)
No intervention (n=66)		 Skills: % of patients receiving guideline-concordant care, OR (95% CI) EHR: 25.4%
No intervention: 22.4%, OR 1.19 (1.01-1.42)		 There was a statistically significant difference favoring the EHR intervention compared with no intervention for the proportion of patients receiving guideline-concordant care.

Peremans, 2010 [31] EHR-based clinical decision support (n=15)
Empowered patient group (n=15)
No intervention (n=13)		 Skills: consultation and prescribing skills based on a 48-item checklist, mean difference (95% CI) from baseline to 5 months postintervention EHR: –1.79 (–4.97 to 1.65)
Empowered: 4.92 (1.96-7.89)
No intervention: –0.91 (–3.37 to 1.92)		 The empowered patient group was the only group that had improved consultation and prescribing skills scores after 5 months postintervention and the only intervention that demonstrated a statistically significant difference compared with no intervention.
Electronic educations game

Kerfoot, 2009 [21] Electronic game/survey 2 questions every 2 days (n=735)
Electronic game/survey 4 questions every 4 days (n=735)		 Knowledge: median % (IQR) scores for knowledge test baseline Electronic game 2 questions every 2 days: 48% (18)
Electronic game 4 questions every 4 days: 45% (15)		 Both electronic game cohorts demonstrated statistically significant improvements in knowledge compared with baseline.



Knowledge: median % (IQR) scores for knowledge test postintervention (12 or 24 weeks), P value Electronic game 2 questions every 2 days: 100% (3)
Electronic game 4 questions every 4 days: 98% (8), P<.001


















































Email





Lobach, 1996 [19] Biweekly emails of computer-based audit/feedback program (n=22)
No intervention (n=23)		 Skills: median % (IQR) participant compliance with guidelines, P value Email: 35.3% (NRi)
No intervention: 6.1% (NRi), P=.01		 The email intervention demonstrated statistical significance in greater compliance with guidelines compared with no intervention.

Stewart, 2005 [32] Email Web-based learning for 2 evidence-based modules (type 2 diabetes, prevention) (n=27)
Waiting list (n=31)		 Knowledge: mean (SD) score (out of 100) at baseline Email (diabetes): 66.8 (14.1)
Email (prevention): 53.8 (12.8)
Waiting list (diabetes): 68.6 (10.4)
Waiting list (prevention): 51.9 (9.5)		 The intervention group (prevention module) demonstrated statistically significant improvements compared with the control group for knowledge at 2 and 6 months, as well as compliance at 6 months. There was no statistically significant difference with the diabetes modules.



Knowledge: mean (SD) score (out of 100) at 2 months postintervention, P value Email (diabetes): 72.7 (14.1)
Email (prevention): 63.8 (17.6)
Waiting list (diabetes): 67.7 (16.8), P=.57
Waiting list (prevention): 50.5 (13.8), P=.002



Knowledge: mean (SD) score (out of 100) at 6 months postintervention, P value Email (diabetes): 73.2 (7.7)
Email (prevention): 65.7 (15.2)
Waiting list (diabetes): 68.6 (11.4), P=.14
Waiting list (prevention): 53.3 (10.5), P=.004



Skills: mean (SD) score for compliance with guidelines (out of 100) at baseline Email (diabetes): 53.8 (12.5)
Email (prevention): 52.2 (11.1)
Waiting list (diabetes): 51.2 (11.6)
Waiting list (prevention): 51.1 (14.4)



Skills: mean (SD) score for compliance with guidelines (out of 100) at 2 months postintervention, P value Email (diabetes): 51.7 (12.9)
Email (prevention): 52.2 (11.7)
Waiting list (diabetes): 51.6 (9.5), P=.90
Waiting list (prevention): 47.7 (13.8), P=.11



Skills: mean (SD) score for compliance with guidelines (out of 100) at 6 months postintervention, P value Email (diabetes): 47.1 (9.2)
Email (prevention): 55.0 (10.0)
Waiting list (diabetes): 50.8 (9.1), P=.14
Waiting list (prevention): 50.0 (14.4), P=.03















Multifaceted




Bernhardsson, 2014 [33] Multifaceted: implementation seminar/group discussion, website, and email reminders (n=168)
No intervention (n=88)		 Knowledge: change in % of participants who were aware that guidelines exist from baseline to 1-year follow-up, P value Intervention: 27.9%
No intervention: 7.3%, P=.02		 There was a statistically significant difference favoring the intervention group for change in awareness, knowledge of where to find guidelines, and accessibility of guidelines at 1-year follow-up. There were no significant differences in frequent use of CPGs.



Knowledge: change in % of participants who knew where to find guidelines from baseline to 1-year follow-up, P value Intervention: 25.2%
No intervention: 4.8%, P=.007



Perceived ease of use: change in % of participants who felt guidelines were easy to access from baseline to 1-year follow-up, P value Intervention: 17.4%
No intervention: −4.3%, P<.001



Skills: change in % compliance with use of CPGs (frequently or almost always) Intervention: 9.2%
No intervention: −0.2%, P=.30

Chan, 2013 [34] Multifaceted: in-person education session and Web-based support (n=31)
No intervention (n=22)		 Beliefs about capabilities: change in % (95% CI) of participants who were self-confident in following CPGs at 2 weeks postintervention Intervention: 25.9% (4.2 to 45.5)
No intervention: 6.3% (−2.0 to 32.1)		 There were statistically significant improvements in self-confidence to use, satisfaction in following, and willingness to follow CPGs among the intervention group at 2 weeks postintervention. There were no significant improvements among the control group.



Perceived usefulness: change in % (95% CI) of participants who were satisfied in following CPGs at 2 weeks postintervention Intervention: 40.7% (16.1-59.6)
No intervention: −12.5 (−37.3 to 12.7)



Intention: willingness to use new CPGs, mean score change (95% CI) (out of 4, 4=all CPGs) at 2 weeks postintervention Intervention: 0.74 (0.36-1.1)
No intervention: 0.19 (−0.10 to 0.48)

Desimone, 2012 [35] Multifaceted: in-person education, Web-based support, printed materials (n=11)
Usual education (n=11)		 Knowledge: mean % (SD) of correct responses (11 items) at baseline Multifaceted: 69% (1.7)
Usual education: 76% (1.2)		 There was a statistically significant improvement in knowledge in both groups at 1 month postintervention. There were no observable differences between groups (between-group statistical analyses not performed).



Knowledge: mean % (SD) of correct responses (11 items) at 1 month postintervention, P value Multifaceted: 83% (2.1), P=.003
Usual education: 84% (1.4), P=.02

McDonald, 2005 [36] Multifaceted: email reminder with provider prompts, patient education material, and clinical nurse specialist outreach (n=97)
Email reminder of recommendations (n=121)
Usual care (n=118)		 Skills: adjusted mean difference in probability that participant assessed bowel movement based on CPG compared with usual care, P value Email reminder: –5.7, P=.02
Multifaceted: –2.7, P=.26		 In the email reminder intervention group, there was a decrease in performance, as the probability of nurses completing bowel movement assessments was statistically significantly lower compared with usual care. There was no statistically significant difference compared with the multifaceted group. Other nurse assessment and instruction practices did not reach statistical significance when the email reminder and multifaceted interventions were compared with usual care (results not shown).

Fretheim, 2006 [18] Multifaceted: educational outreach visit, audit and feedback at outreach visit, computerized reminders, risk assessment tools, patient information material, telephone follow-up (n=257)
Passive guideline dissemination (no additional active promotion or encouragement for use of guidelines) (n=244)		 Skills: mean change in % participants prescribing in concordance to CPGs from baseline to 12 months, between-group difference RRj (95% CI) Multifaceted: 11.5%
Passive dissemination: 2.2%, 1.94 (1.49-2.49)		 There was a statistically significant difference in participants prescribing in concordance to CPGs from baseline to 12 months favoring the multifaceted group compared with passive guidelines dissemination. No statistically significant differences were demonstrated for differences in participants performing risk assessments at 12 months.



Skills: between-group difference in mean % participants performing risk assessments according to CPGs at 12 months, RR (95% CI) 1.04 (0.60-1.71)

Shenoy, 2013 [37] Multifaceted: Web-based education, audit, feedback (n=24)
Mailed guidelines (n=21)		 Knowledge: mean change (95% CI) in total score (18 clinical vignettes) from baseline to 12 weeks postintervention 0.04 (1.22-1.31) There was no statistically significant change in knowledge between intervention groups from baseline to 12 weeks postintervention. There was no statistically significant difference between intervention groups for the proportion of patients receiving CPG-adherent care at 12 weeks postintervention (results not shown).

aOR: odds ratio.

bNR: not reported.

cCPG: clinical practice guideline.

dCrossover design with same participants in both groups.

eIQR: interquartile range (25th to 75th percentile).

fADHD: attention-deficit/hyperactivity disorder.

gCME: continuing medical education.

hEHR: electronic health record.

iIQR values illustrated in a diagram; however, values are not explicit.

jRR: relative risk.