Interventions for promoting information and communication technologies adoption in healthcare professionals

Abstract

Background

Information and communication technologies (ICT) are defined as digital and analogue technologies that facilitate the capturing, processing, storage and exchange of information via electronic communication. ICTs have the potential to improve information management, access to health services, quality of care, continuity of services, and cost containment. Knowledge is lacking on conditions for successful ICT integration into practice.

Objectives

To carry out a systematic review of the effectiveness of interventions to promote the adoption of ICT by healthcare professionals.

Search methods

Specific strategies, defined with the help of an information specialist, were used to search the Cochrane Effective Practice and Organisation of Care Group (EPOC) register and additional relevant databases. We considered studies published from January 1990 until October 2007.

Selection criteria

Randomised controlled trials (RCTs), controlled clinical trials (CCTs), controlled before/after studies (CBAs), and interrupted time series (ITS) that reported objectively measured outcomes concerning the effect of interventions to promote adoption of ICT in healthcare professionals’ practices.

Data collection and analysis

Two reviewers independently assessed each potentially relevant study for inclusion. We resolved discrepancies by discussion or a third reviewer. Two teams of two reviewers independently extracted data and assessed the quality of included studies. A meta-analysis of study outcomes was not possible, given the small number of included studies and the heterogeneity of intervention and outcomes measures. We conducted qualitative analyses, and have presented the results in a narrative format.

Main results

Ten studies met the inclusion criteria. Nine of them were RCTs. All studies involved physicians as participants (including postgraduate trainees), and one study also included other participants. Only two studies measured patient outcomes. Searching skills and/or frequency of use of electronic databases, mainly MEDLINE, were targeted in eight studies. Use of Internet for audit and feedback, and email for provider-patient communication, were targeted in two studies. Four studies showed small to moderate positive effects of the intervention on ICT adoption. Four studies were unable to demonstrate significant positive effects, and the two others showed mixed effects. No studies looked at the long-term effect or sustainability of the intervention.

Authors’ conclusions

There is very limited evidence on effective interventions promoting the adoption of ICTs by healthcare professionals. Small effects have been reported for interventions targeting the use of electronic databases and digital libraries. The effectiveness of interventions to promote ICT adoption in healthcare settings remains uncertain, and more well designed trials are needed.

BACKGROUND

Information and communication technologies (ICTs) have the potential to address many of the challenges that healthcare systems are currently confronting. Globally, “ICT encompasses all those digital and analogue technologies that facilitate the capturing, processing, storage and exchange of information via electronic communication” (Health Canada 2006). The term eHealth is also increasingly used to refer to ICT in the healthcare domain (Eysenbach 2001; Oh 2005). Thus, this review considers healthcare ICT and eHealth as equivalent.

The following classification identifies the five broad categories of ICT (Open Clinical 2006):

1. Electronic Medical Records (including patient records, clinical administration systems, digital imaging & archiving systems, e-prescribing, e-booking);

2. telemedicine and telecare services;

3. health information networks;

4. decision support tools for healthcare professionals;

5. internet-based technologies and services.

Each of these ICT categories encompasses various applications that have specific functions in healthcare settings. Those applications have the potential to improve information management, access to health services, quality and safety of care, continuity of services, and cost containment (Gov. Canada 1999). The evidence on the efficacy of some applications, such as teleconsultations (Currell 2000), email consultations (Car 2004a), computerised health records (Erstad 2003), and clinical information retrieval technologies Pluye 2005), is limited or shows limited effect. However, other ICT applications, such as some decision support systems (Garg 2005; Kawamoto 2005), computerised reminders (Bennett 2003), computerised advice on drug dosage (Durieux 2008), and interactive health communication applications (Murray 2005) have shown benefits for the healthcare system and may improve patient health outcomes. Furthermore, patients want clinicians to use ICT (Car 2004b). With increasing computerisation in every sector of activity, ICTs are expected to become tools that are part of healthcare professional practice. Nonetheless, it appears that ICTs such as electronic medical records and the Internet remain underused by healthcare professionals (Berner 2005; Brooks 2006). Human and organisational factors have been identified as the main causes of ICT implementation failure (Aarts 2004; Lorenzi 1997; Pagliari 2005).

The optimal integration of ICT into healthcare professionals’ practices should be based upon the highest level of scientific evidence available with respect to implementation strategies (Grimshaw 2004; Moehr 2000). Previous studies have found that training is a major determinant of ICT adoption by healthcare profession and influences the integration of these technologies into clinical practices (Allen 2000; Kronick 2003). However, many factors may influence the effectiveness of educational strategies, such as characteristics of the learner, the intervention itself, characteristics of the behaviour that the intervention is trying to change, and the context in which the intervention is conducted (Farmer 2003).

Characteristics of the intervention include the source of the information, the content, and the channel by which it is delivered (Kanouse 1995; Marriott 2000). The credibility of the information source is based on several elements, such as expertise and knowledge, trustworthiness, and message attributes (Tseng 1999; Wathen 2002). Research on the specific characteristics of successful interventions to promote ICT adoption into clinical practice is limited. Nevertheless, some studies on effective strategies to change clinical practice indicate that training interventions need to be sufficiently persuasive, informative, and relevant to the learner (Davis 1999). Personalised feedbacks and/or tailored messages may be more useful for changing behaviour than non-tailored messages (Bull 2001 Kreuter 1996; Kreuter 2001).

Characteristics of professional groups, organisations, and context also influence the success of ICT implementation. At the group level, the introduction of ICT can advance corresponding modifications to roles and responsibilities of healthcare professionals, which represents a potential source of resistance to adoption (Markus 1983). Likewise, conflicts can be present with respect to the delimitation of professional boundaries (Massaro 1993; McLaughlin 1998). Furthermore, structural and contextual characteristics of healthcare organisations can influence the integration of ICT into clinical practices (Kimberly 1981). Factors such as hospital location and size, participation of professionals in decision making, and management support, have been found to impact on ICT adoption (Carman 1996; Gagnon 2005; Lapointe 1999).

The effectiveness of interventions aiming at the integration of ICT applications into healthcare professionals’ practices are likely to be influenced by various factors pertaining to individual, group, organisational, and contextual characteristics, and by the very nature of the intervention (Grol 2004; Grol 2007). Furthermore, interrelations between these factors are important to consider, as they can influence how scientific evidence is integrated into practice (Street 1997). However, ICTs remain underused by healthcare professionals, and knowledge is lacking on the best strategies to integrate them into their practice. Furthermore, existing literature appears to be conflicting with respect to effective interventions for promoting ICT adoption by healthcare professionals. In summary, it is imperative to synthesise knowledge regarding ICT adoption by healthcare professionals in order to inform decision-makers about effective strategies to promote the integration of these technologies in healthcare systems.

OBJECTIVES

The objective of this systematic review was to assess the effectiveness of interventions to promote the adoption of ICT by health-care professionals.

To address this question, we considered two comparisons:

1. any particular type of intervention specifically designed to promote ICT adoption in healthcare professionals compared to no intervention, standard practice or any other particular type of intervention;

2. multifaceted intervention specifically designed to promote ICT adoption in healthcare professionals compared to no intervention, standard practice or any single intervention specifically designed to promote ICT adoption in healthcare professionals.

METHODS

Criteria for considering studies for this review

Types of studies

We considered the following study designs: randomised controlled trials (RCTs); controlled clinical trials (CCTs); controlled before/after studies (CBAs); and interrupted time series analyses (ITS) with a clearly defined point in time when the intervention occurred and at least three data points before and three after the intervention (Ramsay 2003). We considered studies published in all languages from January 1990 to October 2007.

Types of participants

Study participants were healthcare professionals providing clinical care to patients, including professionals in training (residents, fellows, and other registered healthcare professionals), exposed to an intervention aimed at promoting the adoption or use of any type of ICT in their practice. We excluded studies that included only students who did not provide clinical care to patients.

Types of interventions

We included any type of intervention described in the EPOC data collection checklist (EPOC 2007) to promote the adoption and use of any type of ICT (electronic medical record, telemedicine/telehealth, health information networks, decision support tools, Internet-based technologies and services).

The intervention had to go beyond the simple provision of, or access to, an ICT application; i.e. a planned strategy to promote the adoption or use of the ICT application should have been implemented.

Types of outcome measures

Two types of outcome measures were of interest in this review. First, we considered objective measures of the adoption or use of the ICT application by healthcare professionals (e.g. the number of teleconsultations conducted by a physician via videoconference, the frequency of use of information retrieval systems). We excluded studies with only self-reported measures. We selected healthcare professionals’ performance, in terms of ICT adoption and use, as the primary outcome of interest for this review as it is hypothesised to directly influence process outcomes, and, ultimately, health and cost outcomes.

Second, we considered any objective measure of general clinical performance or process outcome (e.g. number of tests ordered or decision to prescribe a particular drug), or patient health outcomes (e.g. blood pressure, length of hospital stay). We also included measures of healthcare professionals’ knowledge, attitudes or satisfaction as secondary outcomes, as they may provide useful complementary information. However, we did not include studies reporting only knowledge, attitudes or satisfaction, with no objective measure of our main outcome of interest -- namely, adoption or use of ICT-- in accordance with the methodology of the Cochrane Effective Practice and Organisation of Care Review Group (EPOC).

Search methods for identification of studies

We used specific strategies defined with the help of an information specialist to search the EPOC register (See SPECIALISED REGISTER under GROUP DETAILS) and the database of studies awaiting assessment. We conducted additional searches on Cochrane Database of Systematic Reviews, MEDLINE, EMBASE, Ovid, Database of Abstracts of Reviews of Effects (DARE), Cumulative Index of Nursing and Allied Health Literature (CINAHL), Biosis Previews, Psychological Abstracts (PsycINFO), Current Content, Health Services/Technology Assessment Text (HSTAT), Dissertation Abstracts, Educational Resources Information Center (ERIC), Proquest, ISI Web of Knowledge, Latin American and Caribbean Health Sciences (LILACS), and Ingenta.

We developed search strategies for electronic databases using the methodological component of the EPOC search strategy combined with selected ICT terms and free text terms relating to ICT. The following are the ICT data terms that have been used in the MEDLINE search strategy. This search strategy was translated for searching the other databases, using the appropriate controlled vocabulary as applicable.

1. Local Area Networks/

2. exp Telemedicine/

3. (telemedicine or telehealth).tw.

4. exp computer communication networks/

5. internet/

6. (ict or information communication technolog$).tw.

7. (remote communication$ or remote consultation$).tw.

8. Information Services/

9. (ehealth or e-health).tw.

10. (digital divide or information poverty).tw.

11. (internet or email or www or world wide web or virtual or web site or website).tw.

12. (e-learning or elearning or telecommunicat$).tw.

13. Databases, Bibliographic/

14. exp Medical Informatics Applications/

15. Medical Records Systems, Computerized/

16. Reminder Systems/

17. (computerised reminder$ or computerised reminder$).tw.

18. Patient Identification Systems/

19. or/1-17

We followed up and obtained for assessment any relevant references from studies found through the above strategies. We also searched publications citing the retrieved articles through the ISI Science Citation Index.

Data collection and analysis

Screening

We screened all titles and abstracts and retrieved full text copies of all potentially relevant studies. Two reviewers, randomly chosen among six members of the research team (MPG, FL, ML, PF, PP, JG) then independently assessed each study for eligibility. We resolved discrepancies by consensus or involvement of a third reviewer.

Data abstraction and quality assessment of included studies

Two teams of reviewers (MPG/MD and JC/CP) independently and systematically assessed characteristics of the studies and extracted data using the EPOC Data Collection Checklist (EPOC 2007). We contacted study authors if data were missing or needed clarification; all contacted authors replied. We resolved discrepancies in ratings by consensus and involvement of a third reviewer where necessary. We assessed the methodological quality of eligible studies using the criteria of the Cochrane Effective Practice and Organisation of Care Review Group (EPOC) (see ’Editorial information’ under ’Group details’ for ’Methods used in review’) for RCT, CCT, CBA or ITS. Given the limited number of identified studies in this review, we did not set a minimum cut-off for study inclusion based on methodological quality.

Two independent reviewers rated each RCT study included in the review (high, moderate or low protection against bias) based on three main criteria: concealment of allocation, blinded or objective assessment of primary outcome(s), and follow up of participants (at least 80%). We also considered three other criteria: baseline measurement, reliable primary outcome measure, and protection against contamination. We assigned a rating of ’High’ protection against bias if the first three criteria were met and there were no important concerns related to the last three criteria; ’Moderate’ if one or two of the three first criteria were scored as “not clear” or “not done”; and ’Low’ if more than two of the first criteria were scored as “not clear” or “not done” (Jamtvedt 2006). We resolved discrepancies by consensus.

For ITS, we assessed the methodological quality of the included study using specific criteria (Ramsay 2003): 1) intervention occurred independently of other changes; 2) intervention was unlikely to affect data collection; 3) the primary outcome was assessed blindly or was measured objectively; 4) the primary outcome was reliable or was measured objectively; 5) the composition of the data set at each time point covered at least 80% of the participants; 6) the shape of the intervention effect was prespecified; 7) a rationale for the number and spacing of data points was described; 8) the study was analysed appropriately using time series techniques.

Analysis

We used methods proposed by Grimshaw (Grimshaw 2004a) to guide data analysis and presentation. Statistical analysis considered both dichotomous and continuous process outcome measures. We calculated relative risk differences for dichotomous variables and standardised mean differences (SMD) for continuous variables. In cases where there was insufficient data to calculate effect sizes for these outcome measures, we presented results of studies as reported by the authors.

For each study, we reported the main results in natural units (for example, mean number of log-ons per physician). When baseline data were not available, we expressed results as the relative percentage change (the difference between post-intervention values in the experimental and control groups divided by the post-intervention values in the control group). This score allows estimation of a standardised effect for each outcome, which can be interpreted and pooled across studies regardless of the original measurement scale (Laird 1990). In addition, in studies that reported pre- and post-intervention mean or proportion for both experimental and control groups, we calculated the absolute change from baseline (change in experimental group values minus change in control group values), with 95% confidence interval (CI) where possible. For ITS, we reported the main outcomes in natural units and one or two parameters: the change in the level of outcome immediately after the introduction of the intervention and/or the change in the regression slopes (Ramsay 2003). At least one of these estimates is necessary for interpreting the results of each comparison. For example, there could have been no change in the level of outcome immediately after the intervention, but there could have been a significant change in the slope.

We classified interventions according to the EPOC Data Collection Checklist, which includes four broad types of interventions: professional, financial, organisational, and regulatory. We have categorised factors that may impact on their effectiveness (effect modifiers) as described below. These factors have been identified as effect modifier in previous studies aiming at changing health-care professional practices (Burgers 2003; Foy 2002; Grilli 1994; Grol 1998; Kanouse 1995; Kreuter 1996; Marriott 2000; Wathen 2002).

Factors that may influence the effectiveness of an intervention to promote ICT adoption/use in healthcare settings are:

1. type of technology: electronic medical record, telemedicine/telehealth, health information networks, decision support tools, Internet-based technologies and services;

2. potential adopter: target group of healthcare professionals;

3. practice setting: academic hospital, general or community-based practices;

4. intervention design: tailored versus non-tailored intervention (e.g., personalised versus generic);

5. mode of delivery: a) intervention delivered individually versus intervention delivered in group; b) intervention delivered by a person versus intervention delivered by any other mode (e.g. online instructions, printed material);

6. timing/frequency: intervention given only once versus continuous intervention.

We identified these factors as potential effect modifiers. Given the limited number of studies included in the review, we only explored the heterogeneity by considering comparisons relative to the first category of factors (type of technology) in relationship to the effect size (effect modifier).

RESULTS

Description of studies

See: Characteristics of included studies; Characteristics of excluded studies.

The search generated a total of 47,979 references. From these references, we excluded 47,916 based on title alone or on the title and abstract, and selected 63 studies for detailed evaluation. Following the evaluation by two independent reviewers, we included 10 studies (see ’Characteristics of included studies’ table). Nine studies were RCTs (Bradley 2002; Cabell 2001; Cheng 2003; Erickson 1998; Haynes 1991; Haynes 1993; Haynes 2006; Katz 2003; Magrabi 2007) and one was an ITS (Simon 2005). Eight studies were conducted in North America--three in Canada (Haynes 1991; Haynes 1993;Haynes 2006) and five in the USA (Bradley 2002; Cabell 2001; Erickson 1998; Katz 2003; Simon 2005). Of the remaining two, one was conducted in Hong Kong, China (Cheng 2003), and one in Australia (Magrabi 2007). All studies included physicians as participants (including postgraduate trainees), and one study also included other healthcare professionals (Cheng 2003).

The behaviour of healthcare professionals was measured objectively in all studies, with the exception of one study that measured change in structure of service delivery. In this last study (Katz 2003), the use of a triage-based email communication system between patient and physicians was measured as well as its effect on phone communication volume and rates of patient visits. Two studies measured patient outcomes (Katz 2003; Simon 2005), but only one study (Simon 2005) used an objective measure of patient outcome.

Targeted ICT and outcome measures

Eight studies targeted use of electronic databases and digital libraries, often MEDLINE, in clinical settings (Bradley 2002; Cabell 2001; Cheng 2003; Erickson 1998; Haynes 1991; Haynes 1993; Haynes 2006; Magrabi 2007). Five of these studies used frequency of use of information retrieval systems and searching skills as the main outcome measures (Cabell 2001; Erickson 1998; Haynes 1991; Haynes 1993; Haynes 2006). One of them measured frequency of use, usefulness, and satisfaction with the system (Haynes 2006). A sixth study (Magrabi 2007) considered the frequency of system use as the main outcome measure, but was also interested in factors that influenced usage. Two studies focused on improving searching skills, but also measured frequency of use (Bradley 2002; Cheng 2003).

An Internet-based audit and feedback (Simon 2005) and a triage-based email system for communication between patient and physician (Katz 2003) were the other ICTs studied. One of the studies (Simon 2005) assessed the effectiveness of providing Internet-based audit and feedback to physicians in order to improve care of patients with diabetes and hypertension. The other study (Katz 2003) assessed the utilisation of an email triage system and its effect on the use of resources in two university-affiliated primary care centres.

Interventions for promoting ICT use or adoption

Eight out of 10 studies evaluated interventions categorised as “professional” according to the EPOC Checklist (Bradley 2002; Cabell 2001; Cheng 2003; Erickson 1998; Haynes 1993; Haynes 2006; Magrabi 2007; Simon 2005). Two studies evaluated other types of intervention: a financial intervention (targeted to healthcare professionals) (Haynes 1991) and an organisational intervention (Katz 2003).

Professional interventions

There were four types of professional interventions reported: educational meetings; distribution of educational materials; educational outreach visits; and audit and feedback.

Educational meetings were the sole intervention in two studies (Cheng 2003; Erickson 1998) and a component of a multifaceted intervention in one study (Cabell 2001). In the first two studies (Cheng 2003; Erickson 1998) the intervention was a basic training in information retrieval including practical sessions. However, the mode of delivery and the duration varied between those studies. One study (Erickson 1998) used an intervention that consisted of individual tutoring and hands-on instruction, with medical residents performing searches on MEDLINE in one group, and all searches performed by the instructor in the other group. In the second study (Cheng 2003), the intervention was a three-hour training workshop with supervised hands-on practice. The third study combined a one-hour didactic session in small group with other types of intervention (Cabell 2001).

Two studies (Haynes 2006; Magrabi 2007) evaluated the effect of access to educational material only. In one study (Haynes 2006), the intervention consisted of access to an Internet-based addition to an existing digital library; this addition provided alerts to new articles and maintained a cumulative database of alerts. In the other study (Magrabi 2007), the intervention group received advanced-level online training in the use of an evidence retrieval system in addition to a basic-level training provided to all participants before randomisation. Three other studies also used educational material as a component of a multifaceted intervention. One of them (Cabell 2001) used well-built clinical question cards; another (Bradley 2002) distributed a floppy disk containing EBM search hedges. In the third study (Simon 2005), participants were encouraged to access a website to obtain audit and feedback on their practice, with accompanying educational material.

Educational outreach visits were used in three studies (Bradley 2002; Cabell 2001; Haynes 1993). They were one of many components in a multifaceted intervention in two studies (Cabell 2001; Haynes 1993) and the main component of the intervention in the third (Bradley 2002). In this last study (Bradley 2002), the intervention was real-time instruction given by a librarian on how to retrieve information from online resources. In another study (Haynes 1993), the intervention combined access to a clinical preceptor and individualised feedback given by a librarian. In this same study, all participants (intervention and control groups) were also provided a two-hour basic training before randomisation. Individual practical sessions in clinical question building and searching the medical literature were used with two other types of intervention (didactic session and educational material) in the last study (Cabell 2001).

The fourth type of professional intervention, audit and feedback, was used in one study as the main intervention (Simon 2005). This intervention consisted in encouragement to use a website that provided audit and feedback on physician’s application of guidelines recommendations.

Financial interventions

Only one study (Haynes 1991) described a financial intervention. This study aimed to assess the effect of introducing user fees for MEDLINE searching in clinical settings. This intervention differed from the others since it tested the negative effect of introducing user fees on ICT use by healthcare professionals. As such, the main outcome expected was a decrease in the use of ICT.

Organisational intervention

In the last study (Katz 2003), the main intervention was providing access to a triage-based email system for communication between patients and their providers in primary care. Patients of the physicians in the intervention group were encouraged to use this new system in several ways. Effect of this triage-based email system on clinical resource use and patient and physician satisfaction was assessed.

Multifaceted intervention versus single intervention

Half of the studies (Bradley 2002; Cabell 2001; Haynes 1993; Katz 2003; Simon 2005) evaluated a multifaceted intervention that included professional intervention in four of these studies (Bradley 2002; Cabell 2001; Haynes 1993; Simon 2005). The remaining multifaceted intervention was organisational and included structural intervention (access to a triage-based email system) and patient-oriented interventions (incentive to use the system) (Katz 2003). One study (Erickson 1998) compared two interventions but at the outset, the two intervention groups were combined and compared with a control group (no intervention). The other studies (Cheng 2003; Haynes 1991; Haynes 2006; Magrabi 2007) presented comparisons between one single intervention specifically designed to promote ICT adoption in healthcare professionals and no intervention (Cheng 2003; Haynes 1991) or standard practice (Haynes 2006; Magrabi 2007).

Risk of bias in included studies

Randomised Controlled Trials (RCT)

We included nine RCTs in the review, including four cluster randomised controlled trials (C-RCT) (Bradley 2002; Cabell 2001; Haynes 2006; Magrabi 2007) and five professional randomised controlled trials (Cheng 2003; Erickson 1998; Haynes 1991; Haynes 1993; Katz 2003).

The methodological quality of selected studies is presented in Table 1. Of the nine RCTs, only one had a low risk of bias (high quality) (Haynes 2006). The remaining studies were of moderate quality, except for one of them (Katz 2003) which had a high risk of bias (low quality). Four studies had adequately concealed allocation (Bradley 2002; Cheng 2003; Erickson 1998; Haynes 2006) whereas, in the remaining studies (Cabell 2001; Haynes 1991; Haynes 1993; Katz 2003; Magrabi 2007), it was unclear. Six studies reported blinded assessment of main outcomes (or objective outcome variables) (Cabell 2001; Erickson 1998; Haynes 1991; Haynes 1993; Haynes 2006; Magrabi 2007). The remaining studies were assessed as “not clear” or “partially met”. Follow up of participants was good for main outcomes in six studies (Bradley 2002; Cabell 2001; Haynes 1991; Haynes 2006; Katz 2003, Magrabi 2007 inadequate or “not met” in two studies (Cheng 2003; Erickson 1998) and assessed as “not clear” in another one (Haynes 1993).

Table 1.

Interventions for promoting ICT adoption: effects on utilization

Study ID	Type of intervention	Primary outcome(s)	Effect size	Conclusion
Bradley 2002	Educational outreach visits; educational materials	(1) Quality of the search strategy (post intervention) : mean average (1 = hightest)	(1) Score difference: − 0.08, 95% CI = −1.32 to 1.16, P = 0.90	No significant effect of the intervention on quality of the search strategy immediately after intervention
		(2) Quality of the search strategy (6-month follow-up): mean average (1= hightest)	(2) Score difference : −2.75, 95% CI = −4.73 to −0.77, P = 0.007	Intervention improved quality of the search strategy 6 months after the intervention
Cabell 2001	Didactic meetings; educational materials; Educational outreach visits	(1) Search frequency (number of log-ons)	No data for effect size calculations. Median: 2.1 vs 4. 4, P < 0.001	A simple educational intervention increased resident searching activity
Cheng 2003	Educational meeting (workshop)	(1) Proportion of professionals able to provide adequate clinical question	(1) Intervention effect: 31%, 95% CI = 22% to 39%, P < 0.00001	The intervention increased the proportion of clinicians able to provide adequate clinical question
Erickson 1998	Educational meeting (individual tutorial)	(1) Proportion of professionals who performed 3rd search	(1) Intervention effect: − 5%, 95% CI = −41% to 30%, P = 0.76	No effect of the intervention on the proportion of residents performing the search
		(2) Proportion of professional who performed 4th search	(2) Intervention effect: − 6%, 95% CI = −45% to 33%, P = 0.76	No effect of the intervention on the proportion of residents performing the search
		(3) Search frequency (mean number of log-ons)	(3) Intervention effect: 0. 22, 95% CI = −0.36 to 0. 80, P = 0.45	No significant difference was found as the result of the teaching intervention on search frequency
Haynes 1991	Financial interventions	(1) Proportion of professionals who searched MEDLINE	(1) Intervention effect: −35%, 95% CI = −57% to −13%, P = 0.002	An economic intervention (introducing user fees for MEDLINE searching) was found to significantly reduce the proportion of professionals who conducted a search
Haynes 1993	Educational outreach visits; audit and feedback	(1) Proportion of professionals successful at the 4th search	(1) Intervention effect: −5%, 95% CI = −24% to 14%, P = 0.62	No significant effect of the intervention on the proportion of participants who conducted a successful search
		(2) Proportion of professionals successful at the 8th search	(2) Intervention effect: 8%, 95%CI = −11% to 27%, P = 0.39	No significant effect of the intervention on the proportion of participants who conducted a successful search
Haynes 2006	Educational material (Internet)	(1) Proportion of professionals using the digital library	(1) Intervention effect: 13%, 95% CI = 2% to 23%, P = 0.02	Significant benefits ot intervention on proportion of professionals using the digital library
		(2) Search frequency (mean log-ons/month/user)	(2) Intervention effect : 0. 34, 95% CI = 0.03 to 0.65, P = 0.03	The intervention increased the search frequency among those who used the digital library service
Katz 2003	Organisational intervention (access to a triage-based email system)	(1) Average number of weekly patient emails per 100 scheduled visits	No data for effect size calculations Adusted IRR for differences in trends = 3.6, 95% CI = 2.1 to 6.2, P < 0.001 (trend of increased email volume)	Intervention increased the volume of email but without decrease in phone volume and visit no-shows
Magrabi 2007	Educational material (online training)	(1) Frequency of use (mean number of searches)	(1) Intervention effect: −0. 07, 95% CI = −0.37 to 0. 23, P = 0.64	No significant effect of the intervention on frequency of use
Simon 2005	Audit and feedback	(1) Frequency of residents accessing website	4/12 residents accessed the website	The study reported no effect of the intervention given to all participants

The C-RCTs were conducted to minimise contamination between the study groups, but in two (Bradley 2002; Cabell 2001) of the four C-RCTs, authors mentioned that they could not be certain that contamination between groups was totally prevented. In two RCTs (Erickson 1998; Haynes 1993), it was also not possible to know if contamination between groups was totally prevented. In one study (Katz 2003), the investigators identified a possible mis-classification bias since some patients categorised as “intervention patients” could be patients of control physicians and vice versa.

Interrupted Time Series (ITS)

One study was an interrupted time series (Simon 2005) in which data were collected every month during the study (24 months) to determine whether the intervention was associated with a change in practice, controlling for pre-intervention level. According to quality criteria for ITS (Ramsay 2003), this study has good methodological quality (five quality criteria were met, and three were classified as “not clear”). However, the lack of participation of residents in the Internet-based intervention impeded detecting impact of the main intervention (audit and feedback) on practice.

Effects of interventions

Comparison 1. Any particular type of intervention specifically designed to promote ICT adoption in healthcare professionals compared to no intervention, standard practice, or any other particular type of intervention

1.1. Educational meetings compared to no intervention, standard practice, or any other intervention

Two RCTs evaluated educational meetings alone (Cheng 2003; Erickson 1998).

Dichotomous process outcome measures

There were three dichotomous comparisons of healthcare professional behaviour reported in two RCTs (Cheng 2003; Erickson 1998). Cheng 2003 reported that the educational intervention increased the proportion of clinicians able to provide an adequate clinical question (relative difference = 0.31, 95% CI: 0.22, 0.39; P < 0.00001). Erickson 1998 found no effect of the intervention on the proportion of residents performing two assigned searches (third search: relative difference = −0.05; ns; fourth search: relative difference = −0.06; ns).

Continuous process outcome measures

One continuous process outcome comparison was reported in one trial (Erickson 1998). Erickson 1998 reported a non-significant increase of 0.22 in the mean number of log-ons in the intervention group but no post-intervention data were provided for the control group.

1.2 Distribution of educational materials compared to no intervention, standard practice, or any other intervention

Two C-RCTs evaluated access to educational materials alone (Haynes 2006; Magrabi 2007).

Dichotomous process outcome measure

One study (Haynes 2006) reported that the proportion of physicians using the digital library service was significantly higher in the intervention group (relative difference = 0.13, 95% CI 0.02 to 0.23; P = 0.02).

Continuous process outcome measure

Two continuous process outcome comparisons were reported in two trials. In one study (Haynes 2006), the intervention increased the mean number of log-ons per month per user among those who used the digital library service SMD = 0.34, 95% CI 0.03 to 0.65; P = 0.03). In the other study (Magrabi 2007), the frequency of searches in a digital database did not increase in the group that had received an advanced training compared to basic training (SMD = −0.07; ns). Indeed, the mean number of searches was higher among participants who received only a basic training, but this effect was not significant.

1.3 Financial interventions

Dichotomous process outcome measure

The only RCT that assessed a financial intervention (Haynes 1991) found a significant negative impact of introducing user fees on ICT use by healthcare professionals. Introducing user fees to access MEDLINE was found to significantly reduce the proportion of participants who conducted a search (relative difference = −0.35, 95% CI: −0.57 to −0.13; P = 0.002).

Comparison 2. Multifaceted intervention specifically designed to promote ICT adoption in healthcare professionals compared to no intervention, standard practice, or any single intervention specifically designed to promote ICT adoption in healthcare professionals

Five studies (Bradley 2002; Cabell 2001; Haynes 1993; Katz 2003; Simon 2005) tested the effect of a multifaceted intervention versus no intervention or standard practice. There were no studies testing multifaceted versus single intervention.

Dichotomous process outcome measures

There were three dichotomous comparisons reported in one RCT (Haynes 1993) and one ITS (Simon 2005). The RCT (Haynes 1993) used a multifaceted intervention combining educational outreach visits and audit and feedback, and the ITS (Simon 2005) used audit and feedback as the main component of the multi-faceted intervention. Haynes did not report any significant improvement in the proportion of participants who conducted a successful search (defined as a search retrieving at least one relevant reference) for either of the two searches assessed (fourth search: relative difference = −0.05; ns; eighth search: relative difference = 0.08; ns). However, in this study (Haynes 1993), clinicians in both groups improved their search performance compared to baseline data (first search); this effect could be related, according to the authors, to the basic introduction given to all participants. The ITS (Simon 2005) reported no effect of the intervention on residents’ utilisation of the audit and feedback website. Over a one-year period, only four of the twelve residents accessed their profiles on the website; three of them visited their site only once.

Continuous process outcome measures

There were three continuous process outcomes comparisons in two C-RCTs (Bradley 2002; Cabell 2001) and another in a RCT (Katz 2003). One of the C-RCTs (Cabell 2001) reported a significant positive effect of the multifaceted intervention versus no intervention on residents’ use of a digital library. This study found that the median number of log-ons significantly increased by 2.1 fold in the intervention group. However, it was not possible to compute a SMD for this outcome, since only medians were reported in the article and the authors could not provide complementary data. The other C-RCT (Bradley 2002) used educational outreach visits as the main component of the intervention and reported mixed effects of this intervention on resident searching skills. This study used a scale to rate the quality of the search strategy, with 1 representing the highest score and 5 the lowest; a negative difference should be interpreted as an increase in search quality. The post-intervention data did not show a significant effect of the intervention on the quality of the search strategy (SMD = −0.08; ns). We obtained complementary data from the authors regarding the quality of the search strategy six months post-intervention. A significant increase in the quality of the search strategy was found (SMD = −2.75, 95% CI −4.73 to −0.77; P = 0.007). The study by Katz (Katz 2003) showed a positive effect of the intervention on the volume of emails (incident rate ratio 3.6, 95% CI 2.1 to 6.2). However, email volume increase did not reduce phone communication or visit no-shows, as hypothetised by researchers. Furthermore an additional mail intervention provided to a portion of the patients treated by physicians in the intervention group may have acted as a co-intervention and affected the outcomes.

Patient outcome measures

The impact of ICT adoption on patient outcomes was only measured in two studies (Katz 2003; Simon 2005). In one of these studies (Simon 2005), the intervention had no discernible effect on adherence to practice guidelines for diabetes or hypertension. In the second study (Katz 2003), patients outcomes were self-reported and consisted in perceptions of email benefits, barriers, and general communication with physician. The study reported few between-group differences in patients’ attitudes toward electronic communication and communication in general.

Subgroup analysis: Interventions for promoting adoption or use of information retrieval systems

Among the ten studies included, eight studies (Bradley 2002; Cabell 2001; Cheng 2003; Erickson 1998; Haynes 1991; Haynes 1993; Haynes 2006; Magrabi 2007) focused on information retrieval systems (digital libraries and electronic databases). One of these studies (Haynes 1991) used a financial intervention which was distinct from the others that all used professional interventions. Among the seven studies using professional interventions, three (Cabell 2001; Cheng 2003; Haynes 2006) reported significant positive effect of the intervention on ICT adoption. Two of the three studies (Cabell 2001; Cheng 2003; Erickson 1998) that used educational meetings and one of the two studies (Haynes 2006; Magrabi 2007) that provided access to educational materials showed positive effects of the intervention. One study (Bradley 2002) that used educational outreach visits and educational material showed a positive impact but only six months after the intervention. Another study (Haynes 1993) that used educational outreach visits and audit and feedback did not report positive effect of the multifaceted intervention.

DISCUSSION

There are very few experimental studies on interventions promoting the integration of ICT in the practice of healthcare professionals. Although this review aimed at a wide range of ICT applications, we found only 10 eligible studies. Seven of these studies were published after 2000, and five of them since 2003. One explanation for the limited research in this area could be that ICT use is a relatively new phenomenon in health care. In addition, many potentially relevant studies were rejected because of their non-experimental design as required by the Cochrane Effective Practice and Organisation of Care Group.

Studies included in this review focused mainly on electronic databases and digital libraries (eight out of 10). A plausible explanation for that is the fact that information retrieval systems are among the first applications that have penetrated the healthcare sphere with the development of digital versions of MEDLINE in the 1980’s. There are several studies about trials of implementation of other types of technology (e.g. electronic health records, telemedicine, computerised order entry). However, these studies were excluded because they did not describe explicitly strategies for adoption of the technology. Most studies that focused on other types of technology were also rejected because of the lack of experimental designs assessing their implementation. In a review about the effectiveness of training health professionals in literature search skills using electronic health databases, Garg (Garg 2003) proposes an explanation to the scarcity of research in this area, arguing that research on educational aspects in the healthcare field neither attracts high levels of funding nor has the prestige associated with the research effort in clinical sciences.

Results from the reviewed studies are mixed and only a few found that the intervention implemented was successful in promoting ICT adoption in healthcare professionals. Most of the interventions included in this review were at the level of healthcare professionals (Bradley 2002; Cabell 2001; Cheng 2003; Erickson 1998; Haynes 1993; Haynes 2006; Magrabi 2007; Simon 2005). Among them, only three studies (Cabell 2001; Cheng 2003; Haynes 2006) reported a significant positive effect of the intervention on ICT adoption among healthcare professionals. One study (Bradley 2002) showed a positive impact, but only six months after the intervention. In the light of these mixed results, we could not draw conclusions about the effectiveness of one type of intervention over another.

The impact of ICT adoption on patient outcomes was measured in only two studies (Katz 2003; Simon 2005). In these two studies, the intervention had no discernible effect on patient outcomes. Despite some positive effects of reported interventions on health-care professional behaviour, there is no guarantee that increasing the use of ICT by healthcare professionals will lead to better patient outcomes.

All studies included were targeted at physicians or physicians in training (residents and clinical clerks). Only one study (Cheng 2003) also included other participants (nurses and other hospital clinicians). Studies targeted at other healthcare professionals (nurses mainly) were located in the literature, but they have been excluded from this review because of their design. Most of the studies reviewed (Bradley 2002; Cabell 2001; Erickson 1998; Haynes 1991; Haynes 1993; Katz 2003; Simon 2005) have been conducted in hospital training settings. This setting allows for a better control of the experimentation than in the context of a busy practice. Conversely, two studies (Cheng 2003; Haynes 2006) focused only on practising physicians in rural (Haynes 2006; Magrabi 2007) or urban (Cheng 2003) areas. Only one study (Magrabi 2007) have been conducted in various locations (rural and urban). However, results are too heterogeneous between studies for drawing any conclusion about effectiveness of intervention. In addition, some of the included studies involved small samples which limited the strength of the conclusions that can be drawn about the effects of the interventions.

Finally, no cut-off was set to exclude studies based on their quality. Many included studies may present methodological limitations that could affect the validity of reported outcomes. For instance, some studies have used multiple comparisons for the same sample without adjusting their p-value which could affect the significance of the results. Also, some studies used a cluster randomised trials design in order to overcome the contamination between study groups (Bradley 2002; Cabell 2001; Haynes 2006; Magrabi 2007). The consequence of adopting a C-RCT is that it has lower statistical power than a RCT of equivalent size (Donner 1981). More rigorous research is needed in the evaluation of the various effects of ICT applications and in exploring how to optimise the implementation of these technologies in the healthcare system.

AUTHORS’ CONCLUSIONS

Implications for practice

The limited evidence presented in this review did not allow identifying one best strategy for promoting the adoption of ICT applications in healthcare professionals. Most interventions that have been applied in the field of ICT implementation are educational in nature but present a great heterogeneity in their mode of delivery and their timing and frequency. Among these interventions, the effectiveness of individual or group training showed mixed results. The effectiveness of interventions providing feedback and instructions is also divergent across studies. Furthermore, other types of interventions, consisting mainly in patient and economic incentives, were not common and their effectiveness appears limited to specific outcomes. Another major limitation to this review is the fact that healthcare professionals targeted by the interventions consisted almost exclusively of physicians. Evidence is thus lacking regarding interventions aimed at other groups of health-care professionals. Given the scarcity of available research in the field of ICT implementation, there is an urgent need to develop a knowledge base to support the design, implementation and evaluation of interventions aimed at promoting the optimal integration of ICT in all groups of healthcare professionals’ practice.

Implications for research

Despite the inconclusive evidence found on the effectiveness of interventions for promoting ICT adoption in healthcare professionals, this review provides a basis to guide further research into the development of such interventions and their evaluation. It is thus important for researchers to advance their understanding of factors affecting ICT adoption in healthcare settings. Identifying key issues with respect to individual, professional, organisational, or systemic factors influencing ICT adoption would help to design more specific and tailored interventions. A complementary review assessing obstacles and facilitators to ICT adoption by healthcare professionals is being conducted by the team of reviewers and might provide relevant insight. This review will allow for considering other types of study designs that were rejected in the present review such as longitudinal, cohort, cross-sectional, and qualitative studies. It will also potentially cover other professional groups and other types of technology that are typically included in these study designs.

CHARACTERISTICS OF STUDIES

Characteristics of included studies [ordered by study ID]

Bradley 2002
Methods	C-RCT. Follow up of professionals: DONE Blinded assessment: UNCLEAR Overall protection against bias: MODERATE
Participants	10 residents in a neonatal intensive care unit. Country: USA.
Interventions	Real-time librarian instruction about EBM searching (during one month) + educational material. Control.
Outcomes	Search strategy, perceptions, satisfaction and opinion about use of MEDLINE
Notes	Type of Technology: electronic databases (MEDLINE).
Risk of bias
Item	Authors’ judgement	Description
Allocation concealment?	Yes	A - Adequate
Cabell 2001
Methods	C-RCT. Follow-up of professionals: DONE. Blinded assessment: DONE. Overall protection against bias: MODERATE.
Participants	48 residents in a university hospital-based internal medicine training program. Country: USA.
Interventions	One-hour didactic session + use of well-built clinical question cards + practical sessions in clinical question building. Control.
Outcomes	Use of library information system: number of log-ons, time spent searching, total searching volume, abstracts viewed, and full-text articles viewed
Notes	Type of Technology: electronic databases (MEDLINE). Data missed to perform effect size calculation.
Risk of bias
Allocation concealment?	Unclear	B - Unclear
Cheng 2003
Methods	RCT. Follow-up of professionals: NOT DONE. Blinded assessment: UNCLEAR. Overall protection against bias: MODERATE.
Participants	800 hospital clinicians. Country: Hong Kong (China).
Interventions	Three-hour training workshop (with supervised hands-on practice). Control.
Outcomes	Clinical question formulation, awareness, knowledge, confidence and use of databases, attitude towards the use of electronic information services, searching skills
Notes	Type of Technology: Internet, electronic databases
Risk of bias
Item	Authors’ judgement	Description
Allocation concealment?	Yes	A - Adequate
Erickson 1998
Methods	RCT. Follow-up of professionals: NOT DONE. Blinded assessment: DONE. Overall protection against bias: MODERATE.
Participants	31 obstetrics and gynecology residents training at an academic medical centre. Country: USA.
Interventions	One-hour individual tutorial on MEDLINE with hands-on instruction: resident performing the searches. One-hour individual tutorial with all searching conducted by the instructor. Control.
Outcomes	MEDLINE search frequency, duration, recall, precision and searcher satisfaction
Notes	Type of Technology: electronic databases (MEDLINE).
Risk of bias
Item	Authors’ judgement	Description
Allocation concealment?	Yes	A - Adequate
Haynes 1991
Methods	RCT. Follow-up of professionals: DONE (for main outcomes). Blinded assessment: DONE (for main outcomes). Overall protection against bias: MODERATE.
Participants	59 physicians and physicians-in-training of a teaching hospital. Country: Canada.
Interventions	Introducing user fees for MEDLINE searching in clinical settings (pay group). Continue searching without charge (no pay group).
Outcomes	Frequency and quality (number of citations per search, search recall, precision) of MEDLINE searching; effect on decision
Notes	Type of Technology: electronic databases (MEDLINE).
Risk of bias
Item	Authors’ judgement	Description
Allocation concealment?	Unclear	B - Unclear
Haynes 1993
Methods	RCT. Follow-up of professionals: UNCLEAR. Blinded assessment: DONE (for main outcomes). Overall protection against bias: MODERATE.
Participants	308 physicians and physicians-in-training from 6 departments of a teaching hospital. Country: Canada.
Interventions	Before randomisation: 2 hours of basic training. Access to a clinical perceptor experienced in MEDLINE searching + audit & feedback about search quality from a study librarian. Control.
Outcomes	Nb of searches, search performance (nb relevant citations, precision, recall), cost and time/session, perception of searches worth and other perceptions
Notes	Type of Technology: electronic databases (MEDLINE).
Risk of bias
Item	Authors’ judgement	Description
Allocation concealment?	Unclear	B - Unclear
Haynes 2006
Methods	C-RCT. Follow-up of professionals: DONE (for relevant outcomes). Blinded assessment: DONE (for main outcomes). Overall protection against bias: HIGH.
Participants	203 physicians in a relatively sparsely populated area. Country: Canada.
Interventions	Access to a full-serve version (of an existing digital library): included discipline-specific alerts to new articles and database of accumulated alerts. A self-serve version: included a passive guide to evidence-based literature
Outcomes	Utilisation of the service, utility, use of relevant evidence-based information, clinical usefulness
Notes	Type of Technology: electronic databases, email for alerts.
Risk of bias
Item	Authors’ judgement	Description
Allocation concealment?	Yes	A - Adequate
Katz 2003
Methods	RCT. Follow-up of professionals: DONE. Blinded assessment: NOT DONE (for main outcomes). Unit of analysis error. Overall protection against bias: LOW.
Participants	98 physicians and residents in 2 university-affiliated primary care centers. Country: USA.
Interventions	Access to a Triage-based email system promoted to the patients of intervention physicians (by cards given during clinic visits + flyers mailed + automatic response to each email). Control (no access to this triage-based email system).
Outcomes	Patient email use, phone calls, and visit distribution by physician; attitude about communication (physician and patient)
Notes	Type of Technology: email for provider-patient communication
Risk of bias
Item	Authors’ judgement	Description
Allocation concealment?	Unclear	B - Unclear
Magrabi 2007
Methods	C-RCT. Follow-up of professionals: DONE (for relevant outcome). Blinded assessment: DONE. Overall protection against bias: MODERATE.
Participants	227 physicians (general practice) from across Australia. Country: Australia.
Interventions	Advanced online training which provided additional guidance to answer clinical questions. Control (standard tutorial).
Outcomes	Frequency of system use (associated with training); other outcomes not relevant here
Notes	Type of technology: Online evidence retrieval system.
Risk of bias
Item	Authors’ judgement	Description
Allocation concealment?	Unclear	B - Unclear
Simon 2005
Methods	ITS. Methodology quality: good. 5 criteria: DONE. 3 criteria: NOT CLEAR.
Participants	12 primary care internal medicine residents. Country: USA.
Interventions	Residents were encouraged to access the Internet to obtain audit and feedback on their practice, with accompanying pertinent educational material (through a personal letter + a face-to face meeting)
Outcomes	Proportion of residents who accessed their profiles and change following the intervention in the proportion of patients whose care followed national guidelines
Notes	Type of Technology: Internet for audit and feedback.
Risk of bias
Item	Authors’ judgement	Description
Allocation concealment?	Unclear	D - Not used

Allocation concealment: A - Adequate B - Unclear C - Inadequate D - Not used

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Brilla 2004	Type of design other than those included.
Butzlaff 2004	Intervention not targeted at promoting adoption and use of a ICT
Casebeer 2003	Primary outcomes do not mainly concern the adoption or use of the ICT application
D’Alessandro 2004	Type of design other than those included.
Eccles 2002	Intervention not targeted to promoting adoption and use of a ICT
Kronick 2003	Outcomes: self-reported measures only.
Lai 2006	Primary outcomes do not mainly concern the adoption or use of the ICT application
Lapinsky 2001	Intervention not targeted to promoting adoption and use of a ICT
Levick 2005	Type of design other than those included.
Liaw 2000	Type of design other than those included.
Marshall 2001	Outcomes: no objective measure of clinical performance or process outcome
Tai 1999	Outcomes do not concern the adoption or use of the ICT application

DATA AND ANALYSES

Comparison 1. Dichotomous data

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Proportion of professionals able to provide adequate clinical question	1	476	Risk Difference (M-H, Fixed, 95% CI)	0.31 [0.22, 0.39]
2 Proportion of professionals who performed 3rd search	1	31	Risk Difference (M-H, Fixed, 95% CI)	−0.05 [−0.41, 0.30]
3 Proportion of professionals who performed 4th search	1	31	Risk Difference (M-H, Fixed, 95% CI)	−0.06 [−0.45, 0.33]
4 Proportion of professionals using the digital library service	1	203	Risk Difference (M-H, Fixed, 95% CI)	0.13 [0.02, 0.23]
5 Proportion of professionals who searched MEDLINE	1	59	Risk Difference (M-H, Fixed, 95% CI)	−0.35 [−0.57, −0.13]
6 Proportion of professionals successful at the 4th search	1	82	Risk Difference (M-H, Fixed, 95% CI)	−0.05 [−0.24, 0.14]
7 Proportion of professionals successful at the 8th search	1	77	Risk Difference (M-H, Fixed, 95% CI)	0.08 [−0.11, 0.27]

Comparison 2. Continuous data

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Quality of the search strategy (post-intervention)	1	10	Std. Mean Difference (IV, Fixed, 95% CI)	−0.08 [−1.32, 1.16]
2 Quality of the search strategy (6-month follow up)	1	10	Std. Mean Difference (IV, Fixed, 95% CI)	−2.75 [−4.73, −0.77]
3 Mean signs-on	1	46	Std. Mean Difference (IV, Fixed, 95% CI)	0.22 [−0.36, 0.80]
4 Mean logins/month/user	1	165	Std. Mean Difference (IV, Fixed, 95% CI)	0.34 [0.03, 0.65]
5 Mean number of searches	1	193	Std. Mean Difference (IV, Fixed, 95% CI)	−0.07 [−0.37, 0.23]

Analysis 1.1.

Comparison 1 Dichotomous data, Outcome 1 Proportion of professionals able to provide adequate clinical question.

Analysis 1.2.

Comparison 1 Dichotomous data, Outcome 2 Proportion of professionals who performed 3rd search.

Analysis 1.3.

Comparison 1 Dichotomous data, Outcome 3 Proportion of professionals who performed 4th search.

Analysis 1.4.

Comparison 1 Dichotomous data, Outcome 4 Proportion of professionals using the digital library service.

Analysis 1.5.

Comparison 1 Dichotomous data, Outcome 5 Proportion of professionals who searched MEDLINE.

Analysis 1.6.

Comparison 1 Dichotomous data, Outcome 6 Proportion of professionals successful at the 4th search.

Analysis 1.7.

Comparison 1 Dichotomous data, Outcome 7 Proportion of professionals successful at the 8th search.

Analysis 2.1.

Comparison 2 Continuous data, Outcome 1 Quality of the search strategy (post-intervention).

Analysis 2.2.

Comparison 2 Continuous data, Outcome 2 Quality of the search strategy (6-month follow up).

Analysis 2.3.

Comparison 2 Continuous data, Outcome 3 Mean signs-on.

Analysis 2.4.

Comparison 2 Continuous data, Outcome 4 Mean logins/month/user.

Analysis 2.5.

Comparison 2 Continuous data, Outcome 5 Mean number of searches.