Abstract
Background and Objectives
Training programs designed to improve information management have been implemented but not adequately tested. Three critical components for information management were tested in a randomized control study: (1) knowledge of valid, synthesized summary information, (2) skills to use Web-based resources that provide access to these summaries, and (3) use of Web-based resources in clinical practice.
Methods
Twenty-four primary care practices were provided with computers and high-speed Internet access and then matched, with half randomly assigned to receive training and half to receive training at a later date. Training was designed to address knowledge, skills, and use of Web-based information. Outcomes were assessed by comparing baseline and follow-up questionnaires that focused on five conceptual domains related to Web-based resource use for patient care decisions and patient education.
Results
Compared to the delayed training group, the initial training group increased their knowledge and skill of Web-based resources and use for patient care decisions. Some measures of communication with patients about using Web-based resources and of incorporating use of Web-based resources into daily practice increased from baseline to follow-up for all participants.
Conclusions
Our findings suggest that training and providing computers and Internet connections have measurable effects on information management behaviors.
Evidence-based medicine (EBM), the “integration of best research evidence with clinical expertise and patient values,1” can be difficult to carry out in day-today clinical practice.2–4 Numerous studies have shown that providers have clinical questions that they never answer5–8 and that there are barriers to the implementation of EBM protocols,9,10 including not having the knowledge, skills, and time to access and process resources for patient care decisions.2–4,7,8,10–12
Family medicine has been at the forefront of promoting EBM in clinical practice by focusing on information management skills,2–4,13–15 defined as the ability to access and use electronically based summaries of evidence for patient care decisions.4 These skills are relevant not only for providers but also for patients, who are increasingly accessing health information through the Internet.16–19 Health care providers are encouraged to assist patients in their use of Web-based resources,20–24 which is consistent with evidence-based patient choice.25 Studies suggest, however, that while patients want recommendations of online health resources from their providers,26 few receive them.16,18
The growing availability and use of electronically based resources by patients and providers thus creates both pressure and opportunity for clinical practices to use these health information resources for patient care decisions and patient education.27 Providers may access resources via a number of technologies, including personal digital assistants (PDAs), but two recent randomized studies found that computers are used two to four times more often at work than PDAs.28,29 Studies also document an increase in the use of the Web by health care providers,11,27,30 but overall prevalence of use to answer clinical questions is still low compared to other resources (eg, textbooks and colleagues),7,8 which may not be up to date on current evidence.
While one obvious barrier for providers and practices to use Web-based resources in patient care decisions and patient education is lack of computers and high speed Internet access, another important barrier is insufficient knowledge of Web-based resources and skills to efficiently use these resources.2–4,10,12 Training programs developed to improve information management skills have focused primarily on skills, such as critical appraisal,31,32 or on a single Web-based resource, such as MEDLINE.33,34 Additionally, much of the training has been implemented in medical education settings.35–40 Training programs developed for community-based, primary care settings have not been adequately tested for effectiveness.37,41,42 One study has assessed training for rural physicians using a randomized design and found increases in frequency of use and comfort level in using Web-based resources between intervention and control groups.43
This paper describes and evaluates an intervention designed to promote information management in patient care decisions and patient education via Web-based resources among participants from primary care practices. The intervention addressed three critical components to adopting and maintaining information management behaviors: (1) knowledge of valid, synthesized information,6 (2) skills to use Web sites and search engines that provide access to these summaries,44,45 and (3) use of the Web sites and search engines in clinical practice. We hypothesized that the intervention would increase knowledge, skills, and use of Web-based resources among primary care practice participants.
Methods
The study, conducted since July 2004, involved 24 primary care practices located in New Hampshire and Vermont. The practices were recruited through a previous study,46 by a hospital computer network expert who knew of practices with few computers or poor Internet access, and through a presentation to a local preceptor board.47 The study used a randomized, matched-pairs design that allowed us to match practices on factors that could influence training outcomes, such as existing high-speed Internet connections and number of computers with Internet connections in the practice prior to the intervention. Practices were first matched by size, specialty, and location. Then one practice in each pair was randomly assigned to the initial training protocol (initial intervention group) and the other to the delayed training protocol (delayed intervention group). A crossover design ensured that both groups would eventually receive training, as shown in Figure 1. Dartmouth College's Committee for the Protection of Human Subjects approved the study protocol.
Figure 1.
Study Design
Practices
The 24 practices consisted of nine (37%) family medicine practices and five (21%) each of internal medicine, pediatrics, and multi-specialty practices. Practice size included five practices (21%) with a single provider, 13 practices (54%) with two to five providers, and six practices (25%) with six or more providers (range 6–16 providers). The number of patients seen per week ranged from 50 to 800.
At baseline, 33% of the recruited practices did not have high-speed connections, 54% had limited computer resources, and 38% had electronic medical record (EMR) systems.
To ensure that each of the 24 practices had at least the basic technological resources needed to access Web-based resources, each practice was provided with two computers, high-speed Internet connection (if not already in place), and financial support to maintain Internet connections. The computers and Internet connection were put in place for all practices after baseline data collection but before randomization into either the initial intervention or delayed intervention group.
Participants
The initial intervention group practices (n=12) selected a team of one to three individuals (providers, clinical, and office staff) for a total of 32, to participate in the initial training protocol. The delayed intervention group practices (n=12), knowing they would participate in the training later, selected similar teams, for a total of 32 participants.
Intervention
The training provided to the initial intervention group participants focused on two components: knowledge and skills to access and efficiently use a variety of Web-based resources and development of a practice-based plan to incorporate and increase the use of these resources for patient care decisions and patient education in participants' own practice settings. Teaching methods were used that have been shown to be successful in changing behaviors and patient care, such as case studies, hands-on practice, and sequenced sessions (learn-work-learn).48,49 As shown in Table 1, the training, conducted by biomedical librarians and faculty experienced in facilitating group learning, consisted of (1) two 6-hour workshops held 2 months apart, (2) two practice-based exercises using Web-based resources between the workshops, (3) a support phone call 1 month after the first workshop, and (4) a site visit 1 month after the second workshop to provide additional support.
Table 1.
Overview of Training Intervention and Resources Provided
| Activities | Accessing/Efficiently Using Resources | Practice Improvement Plan |
|---|---|---|
| Workshop I | • Search assignment based on a clinical case/exploration of several online resources through hands-on exerciseHO, SG
• Report findingsSG, LG • Review of resources and search strategies by librarians using Finding EBM Answers Quickly and Efficiently chart as guide (http://www.dartmouth.edu/~biomed/resources.htmld/guides/find_ebm.shtml)LG, HO • Practice search strategies for each resource using second clinical caseHO, SG • Give and review handouts to aid in locating and using resourcesLG • Review between-workshop exercise: identify and answer one clinical or patient education question using Web-based resources covered in workshopLG |
• Discuss experiences using Web-based information in practice for patient care decisions and patient educationLG
• Review baseline data on current Web-based resource use by providers (summary) and patients (per practice)TW • Create patient care flow diagramTW • Brainstorm ideas on how to change the process of care to use Web-based resources for patient care decisions and patient educationTW • Share findings/ideasLG • Review of potential tools available to promote use of Web-based resources (eg, NLM information prescription pads [http://www.informationrx.org/Splash.html])LG • Develop initial improvement plan to increase use of Web-based resources in day-to-day practiceTW |
| Between-workshop exercises | • Complete and send back clinical or patient education question exerciseI | • Implement initial improvement planTW |
| Workshop II | • Discuss/address search strategy issues identified in clinical question exerciseLG
• Review online resources again based on participant questionsLG, HO |
• Share experiences from implementation of improvement plansSG,LG
• Revise/expand improvement planTW • Develop measures to evaluate goalsLG,TW |
| Phone call/site visit | • Answer questions about resources
• Provide additional materials to aid in use of resources |
• Review progress on improvement plan
• Review of practice goals/successes/plans to improve; offer support/suggestions |
HO—Hands-on Internet searches (each participant had own computer to work on)
LG—Large-group presentation/discussion
SG—Small-group work (made up of individuals from different practices)
TW—Work by practice teams, with assistance from workshop faculty
I—Individual work
Measure
A questionnaire was developed to assess knowledge, skills, perceptions, and use of Web-based resources for patient care and education. The self-report questionnaire was developed after a review of the literature related to use of Web-based resources in clinical practice and consultations with biomedical librarians. The instrument was pilot-tested with 16 providers and staff not associated with the study and modified prior to use to reflect their feedback.
As shown in Table 2, questions selected for this study were chosen to represent five conceptual domains related to Web-based resource use for patient care decisions and patient education: (1) knowledge of and self-assessed skills for using Web-based resources, (2) frequency of resource use, (3) communication with patients about using resources for health information, (4) perceptions about the use of resources, and (5) incorporating use of Web-based resources into daily practice.
Table 2.
Composition of Domains Analyzed
| Domain | Questionnaire Item(s) |
|---|---|
| Knowledge and skills | |
| Scored from 1=not at all to 5=extremely | • How well you can find information you need on the Web |
| Scored from 1=not at all to 5=extremely | • How well you can find educational materials for your patients on the Web |
| Score=sum of four items scored from 1=not at all to 5=extremely (α=.93) | • How well you can do the following Web tasks:
- develop an effective online search strategy from my patient care questions - narrow search results using limits (eg, English only, age of patients, etc) - use terms such as “or” and “and” to widen or narrow my search - use subject headings (eg, MeSH terms) to develop a search |
| Score=sum of eight items scored from 1=not at all well to 5=extremely well (α=.92) | • How well you know how to use the following Web-based resources:
- Internet search engine (eg, Yahoo, Google) - MEDLINE (eg, via PubMed or OVID) - online clinical resource (eg, UpToDate, MDConsult) - online evidence-based review resource (eg, Cochrane) - online practice guidelines (eg, National Guidelines Clearinghouse, Institute for Clinical Systems Improvement) - medical professions Web site (eg, American Academy of Family Physicians) - medical organization Web site (eg, American Heart Association, Centers for Disease Control and Prevention) - patient-oriented health information Web site (eg, MedlinePlus, WebMD) |
| Frequency of resource use | |
| Score=sum of seven items scored from 1=never or seldom to 4=daily or almost daily (α=.80) | • How often in a typical month you use each resource listed to find information for making specific patient care decisions (your own decisions or to help someone else make a decision):
- Internet search engine (eg, Yahoo, Google) - MEDLINE (eg, via PubMed or OVID) - online clinical resource (eg, UpToDate, MDConsult) - online evidence-based review resource (eg, Cochrane) - online practice guidelines (eg, National Guidelines Clearinghouse, Institute for Clinical Systems Improvement) - medical professions Web site (eg, American Academy of Family Physicians) - medical organization Web site (eg, American Heart Association, Centers for Disease Control) |
| Scored from 1=never or seldom to 4=daily or almost daily | • How often in a typical month you use (eg, show to a patient, give to a patient, tell a patient about) Web-based materials (online and/or printed and kept on file for patients) |
| Communication with patients about using resources for health information | |
| Scored from 1=never to 5=daily or almost daily | How often you refer patients to specific Web sites |
| Perceptions about the use of resources | |
| Scored from 1=strongly disagree to 5=strongly agree | • I like it when patients bring in information they find on the Web |
| • I feel that the Web does or could help us provide better health education materials to patients | |
| • I feel that the Web does or could help us provide better clinical care to patients | |
| Incorporating use of resources into daily practice | |
| Scored from 1=never to 5=very often | • I use the Web during my work hours at this practice for patient education |
| • I use the Web during my work hours at this practice for patient care decisions | |
| Scored from 1=strongly disagree to 5=strongly agree | • I think there is leadership at this practice that encourages using Web resources for patient education |
| • I think there is leadership at this practice that encourages using Web resources for patient care decisions. |
The questionnaire was mailed to the practices for completion before they received their computers and high-speed Internet connection (baseline), 1 year later (follow-up I), approximately 3 months after the initial intervention group completed the training protocol, and 2 years later (follow-up II), approximately 3 months after the delayed intervention group completed the training protocol. While all practice members with patient contact in each of the 24 practices received questionnaires, for this study only participants who participated in the training protocol (initial or delayed) were analyzed to evaluate the effectiveness of that training.
Analysis
To evaluate the effectiveness of the training, we used a linear mixed model analysis to assess whether the initial intervention and delayed intervention groups were significantly different from each other and whether the measures differed significantly at baseline and follow-up I. Thus, there were two fixed factors to consider: group (initial intervention versus delayed intervention) and time (baseline versus follow-up I). While the participant completing the questionnaires was considered a random factor, participants work within practices where they share aspects of the environment unique to the practice. Thus, to avoid violating the statistical assumption of independence, a nesting function was used in the model to identify which participants were from the same practice, with the model making the appropriate adjustments in the error term.
The two-factor model created an interaction term that was the main item of interest in this study. This interaction term assessed whether the rate of change from baseline to follow-up I for the initial intervention group was significantly different from the rate of change for the delayed intervention group. Significant interaction terms, those reaching a P value less than or equal to .05, were further investigated with paired and independent t tests to determine the nature of the interaction. The interaction term had four possible pairwise contrasts; since multiple comparisons can increase the likelihood of Type I errors, the significance levels of these tests were adjusted using the Bonferroni adjustment.50 Thus, to reach a significance level equivalent to .05 unadjusted, the adjusted P value required for significance was .0125. If the interaction term was not significant, then the factor term comparing baseline to follow-up was examined to determine whether there was a significant effect of time. A significant effect of time indicated that the average score at follow-up I for all participants was significantly different from the average score at baseline for all participants.
The crossover design of the study allowed for replication of the training intervention. To assess the equivalency of both arms of the study, a linear mixed model analysis using a single fixed factor, group (initial intervention versus delayed intervention) was used to examine differences in the scores of participants from baseline for follow-up II. Again, a nesting function as described above was used to account for the fact that participants were clustered within practices.
Results
Of the 64 participants, 45 (70%) completed both baseline and follow-up I questionnaires: 78% (25/32) in the initial intervention group and 63% (20/32) in the delayed intervention group. Forty participants (63%) completed questionnaires at all three of the survey points: 72% (23/32) in the initial intervention group and 53% (17/32) in the delayed intervention group.
As shown in Table 3, there were no significant differences between the initial intervention and delayed intervention group participants at baseline. Most of the participants were female, and most were either a provider or clinic support staff member. There were no significant differences between those workshop participants who completed all questionnaires and those who did not. There were also no differences between initial intervention and delayed intervention group practices on these resources at baseline: (1) high-speed Internet connections (χ2=.540, P= .462), (2) adequate computer resources (χ2=.691, P=.431), or (3) presence of an EMR (χ2=.097, P=.756).
Table 3.
Descriptor Variables at Baseline
| Delayed Intervention Group | Initial Intervention Group | |
|---|---|---|
| Number of participants who completed both surveys | 20 | 25 |
| n (%) | n (%) | |
| Gender | ||
| Male | 3 (15%) | 7 (29%) |
| Female | 17 (85%) | 17 (71%) |
| Role in practice | ||
| Provider (MD, DO, PA, NP) | 8 (40%) | 10 (40%) |
| Clinic staff (RN, LPN, MA, SW) | 6 (30%) | 7 (28%) |
| Administrative support | 2 (10%) | 3 (12%) |
| Other | 4 (20%) | 5 (20%) |
| Mean (SD) | Mean (SD) | |
| Age (years) | 43.6 (11.1) | 46.0 (12.7) |
| Work at practice (years) | 6.3 (6.9) | 8.8 (8.6) |
| Hours work per week | 37.9 (9.7) | 41.2 (8.4) |
| A computer with Web access is available at work. Scored from 1=never to 5=most of the time | 4.5 (1.1) | 4.6 (0.9) |
| Computer at work has fast Internet connection. Scored from 1=never to 5=most of the time | 4.1 (1.4) | 4.3 (1.1) |
| Frequency of Web use at work for any reason. Scored from 1=never to 6=several times a day | 4.8 (1.4) | 4.8 (1.6) |
Table 4 displays the mean scores and standard deviations for the variables that had significant interaction terms and the results of the post-hoc pairwise comparisons. Table 5 displays the means and standard deviations for the variables that had a significant time effect for all participants.
Table 4.
Mean Scores, SDs, and Post-hoc Pairwise Comparisons for Significant Group by Time Interactions
| Delayed Intervention Group | Initial Intervention Group | |||||
|---|---|---|---|---|---|---|
| Measure | (a) Baseline | (b) Follow-up I | (c) Baseline | (d) Follow-up I | ||
| Knowledge and skills | Mean (SD) | Mean (SD) | Mean (SD) | Mean (SD) | F | P Value ≤.05 |
| Ability to find information on the Web | 3.4 (1.0) | 3.5 (1.0) | 3.1 (.9) d | 3.8 (.8) c | 5.45 | .025 |
| Ability to find educational materials on the Web for patients | 2.4 (1.3) | 2.4 (.8) | 2.1 (1.0) d | 3.1 (1.0) c | 7.25 | .012 |
| Composite of skills for finding information on the Web for patients | 10.6 (5.0) | 10.4 (4.8) | 7.6 (3.8) d | 11.0 (4.2) c | 6.11 | .019 |
| Skills in using a variety of Web-based resources | 13.9 (9.0) | 13.9 (7.4) d | 10.0 (6.6) d | 21.2 (6.9) b,c | 25.51 | <.001 |
| Frequency of resource use | ||||||
| How often I use the Web for patient care decisions | 10.1 (3.5) | 8.9 (3.2) | 7.8 (2.9) | 10.0 (3.1) | 6.59 | .017 |
| Perceptions about the use of resources | ||||||
| I like it when patients bring in information they find on the Web | 3.4 (1.0) | 3.5 (1.0) | 3.2 (.7) d | 3.8 (0.5) c | 4.19 | .05 |
| Incorporating use of resources into daily practice | ||||||
| I think there is leadership that encourages using Web resources for patient education | 2.9 (1.2) | 2.9 (1.2) | 2.6 (1.1) d | 3.5 (0.9) c | 4.66 | .038 |
SD—standard deviation
Note: Lower case letter denotes that, for a given row, the cell value is significantly different from the values in column denoted by the letter. Tests of differences for time (column a and b, c and d) used paired-means t test. Tests of intervention differences (columns a and c, b and d) were independent means t tests. A Bonferroni adjustment was used to control overall Type I error.
Table 5.
Mean Scores and SDs for Significant Effects of Time for All Participants
| Delayed Intervention Group | Initial Intervention Group | |||
|---|---|---|---|---|
| Measures | Baseline | Follow-up I | ||
| Frequency of resource use | Mean (SD) | Mean (SD) | F | P Value |
| How often in a typical month I use Web-based materials for patient education | 1.6 (0.9) | 2.0 (0.8) | 6.6 | .016 |
| Communication with patients about using resources for health information | ||||
| How often I refer patients to specific Web sites | 1.8 (0.9) | 2.5 (0.8) | 17.0 | <.001 |
| Incorporating use of resources into daily practice | ||||
| The degree to which I think there is leadership that encourages using Web resources for patient care decisions | 2.5 (1.2) | 3.5 (0.9) | 16.1 | <.001 |
| How often I use the Web during work hours for patient education | 1.8 (1.0) | 2.3 (1.0) | 12.0 | .002 |
| How often I use the Web during work hours for patient care decisions | 2.4 (1.2) | 2.9 (1.1) | 7.0 | .013 |
SD—standard deviation
Knowledge and Skills
There were significant group by time interactions for each of the four variables: (1) finding information on the Web, (2) finding educational material on the Web for patients, (3) a sum of specific search skills, and (4) skills using a variety of Web-based resources. In each case, pairwise comparisons revealed a significant increase from baseline to follow-up I for the initial intervention group, as shown in Table 4. For the variable, skills using a variety of Web-based resources, there was also a difference between the groups at follow-up I, with the initial intervention group indicating significantly more skills to use resources than the delayed intervention group.
Frequency
As shown in Table 4, while there was a significant interaction term for how often the participants reported using a variety of Web-based resources for patient care decisions (P=.017), the results of the comparison of baseline to follow-up I for the initial intervention group was not significant when adjusted for multiple comparisons (t10= −2.3, P=.041) since this adjustment required a P value of .0125 for significance. As shown in Table 5, all participants reported a significant increase from baseline to follow-up I in how often they used Web-based materials for patient education.
Communication
All participants' reports of how often they referred patients to specific Web sites showed a significant increase from baseline to follow-up I, as shown in Table 5.
Perception
Analysis of participants' liking when patients bring in Web-based information revealed a significant interaction term, as shown in Table 4. Pairwise comparisons showed a significant increase from baseline to follow-up I for the initial intervention group. There were no significant results for participants' reports of feeling the Web could help them provide better clinical care or better health education materials to patients.
Incorporation
As shown in Table 4, there was a significant interaction term for participants' beliefs regarding leadership at the practice that encourages using Web resources for patient education. Pairwise comparisons revealed a significant increase from baseline to follow-up I for the initial intervention group. As shown in Table 5, all participants reported significant increases from baseline to follow-up I for their beliefs regarding leadership at the practice that encourages using Web resources for patient care decisions and for how often they use the Web at work for patient education and for patient care decisions.
Replication of the Training Intervention
Given that the participants were randomized to either the initial intervention or delayed intervention groups, one would expect there to be no differences attributable to early or late training. Differences in the mean scores of participants from baseline to follow-up II are shown in Table 6. Results of the analyses showed a single significant effect of group, for the variable assessing how often in a typical month participants used Web-based materials for patient education, with participants in the delayed intervention group reporting a greater gain in use than participants in the initial implementation group (P=.047). These results indicate that the training intervention in both arms of the study produced substantively similar results.
Table 6.
Mean Differences and SDs between Scores at Baseline and Follow-up II
| Measures | Delayed Intervention Group | Initial Intervention Group |
|---|---|---|
| Knowledge and skills | Mean (SD) | Mean (SD) |
| Ability to find information on the Web | 0.8 (1.0) | 0.7 (1.0) |
| Ability to find educational materials on the Web for patients | 1.2 (1.4) | 1.4 (1.1) |
| Composite of skills for finding information on the Web for patients | 11.0 (6.9) | 11.7 (8.4) |
| Skills in using a variety of Web-based resources | 3.6 (2.8) | 3.2 (4.8) |
| Frequency of resource use | ||
| How often in a typical month I use Web-based materials for patient care decisions | 2.6 (3.5) | 4.3 (3.3) |
| How often in a typical month I use Web-based materials for patient education | 1.1 (1.3) | 0.2 (0.9) |
| Communication with patients about using resources for health information | ||
| How often I refer patients to specific Web sites | 1.2 (1.2) | 0.7 (1.0) |
| Perceptions about the use of resources | ||
| I like it when patients bring in information they find on the Web | 0.4 (0.7) | 0.6 (0.8) |
| I feel that the Web does or could help us provide better health education materials to patients | 0.3 (0.8) | 0.2 (0.6) |
| I feel that the Web does or could help us provide better clinical care to patients | 0.2 (0.8) | −0.1 (1.0) |
| Incorporating use of resources into daily practice | ||
| I use the Web during my work hours at this practice for patient education | 1.5 (0.9) | 0.9 (1.2) |
| I use the Web during my work hours at this practice for patient care decisions | 0.9 (1.3) | 1.2 (1.2) |
| I think there is leadership at this practice that encourages using Web resources for patient education | 1.0 (1.2) | 0.8 (1.3) |
| I think there is leadership at this practice that encourages using Web resources for patient care decisions | 1.1 (1.5) | 1.1 (1.2) |
SD—standard deviation
Discussion
The major findings of this crossover study for improving information mastery in primary care settings were reported increases in knowledge and skills about Web-based resources and frequency of use of these resources for patient care decisions, all of which were associated with the training intervention. Additionally, the study found that all participants reported increases in frequency of Web-based resource use for patient education, communication with patients about these resources, and incorporating resource use into daily practice. These increases were associated with all practices receiving additional computers and high-speed Internet connections through this study after baseline data collection.
Although several changes from baseline to follow-up were statistically significant, the magnitude of the follow-up scores warrants further discussion. For instance, the use of information management behaviors by participants could still be considered quite low (eg, on average, resources were “seldom” used during work hours or for patient education). Other measures did not change, such as participants' perceptions about the usefulness of the Web for providing better health education materials or clinical care for patients, although on average they agreed that it does (3.9 on a 5-point scale). In sum, significant room remains for improvement in information management skills, knowledge, and behaviors even after this fairly intensive training and support protocol.
Given these results, how do we move forward to improve information mastery in primary care settings? Theoretical approaches, such as the theory of reasoned action,51 would posit that behavior proceeds from knowledge, skills, and attitudes or perceptions. In this instance, increased knowledge and skills and already-positive, existing perceptions (3.9 on a 5-point scale) about the usefulness of Web-based resources have produced only a slight increase in behavior. This leads us to believe that other factors operating in the organizational environment are not currently reinforcing and/or attracting providers and staff members to information mastery behavior, at least in the case of Web-based resources. Factors may include perceived lack of time to use resources,7 the experience of not finding answers using Web-based resources or taking too long to find the answer,52 and the habit of established behaviors (eg, talking to colleague to find answers to questions). Training that occurs in primary care practice settings as clinical and patient education questions arise, although potentially resource intensive, may be more effective for addressing or improving some aspects of the organizational environment for information management.53–55 In addition, improvements in the structure and content of existing Web-based resources may go far to increase their use in practice.7,29,56,57
While a limitation of this study is the relatively small sample size that constrains the generalizability, selection bias is not a limitation because the carefully matched crossover design controlled for the desire to participate. Use of self-report questions as proxies for information mastery measures might be considered a limitation. While actual observation of participant skills might provide more objective measures, this would be both impractical and artificial. Although self-report has been thought of as less accurate than assessment by others, Shrauger and Osberg58 have noted that, given the appropriate circumstances, judgments of one's own behavior can be as effective as those made in other ways. While it is possible that response bias could have occurred among workshop participants, their responses did not show increases in all measures.
Three critical components for improving information management behaviors were tested in a randomized control study. While the findings suggest that both the training protocol and provision of computers and high-speed Internet access have measurable impacts, important questions remain to be explored.
Acknowledgments
This work was supported by the National Library of Medicine grant 5 G08 LM 008110, the Office of Community-based Education and Research at Dartmouth Medical School, and Dartmouth Biomedical Libraries.
The authors thank Karen Odato, Pamela Bagley, Tina Foster, and Ruth Burrill for their invaluable contributions to the development and implementation of the training interventions, the practice staff and providers whose participation made this study possible, and Lynn Foster-Johnson for her statistical expertise.
References
- 1.Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. second edition Churchill Livingstone; Edinburgh: 2000. [Google Scholar]
- 2.Guyatt GH, Meade MO, Jaeschke RZ, Cook DJ, Haynes RB. Practitioners of evidence-based care. BMJ. 2000;320:954–5. doi: 10.1136/bmj.320.7240.954. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.McColl A, Smith H, White P, Field J. General practitioners' perceptions of the route to evidence-based medicine: a questionnaire survey. BMJ. 1998;316:361–6. doi: 10.1136/bmj.316.7128.361. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Slawson DC, Shaughnessy AF. Teaching evidence-based medicine: should we be teaching information management instead? Acad Med. 2005;80(7):685–9. doi: 10.1097/00001888-200507000-00014. [DOI] [PubMed] [Google Scholar]
- 5.Ely J, Osheroff J, Gorman P, et al. A taxonomy of generic clinical questions: classification study. BMJ. 2000;321:429. doi: 10.1136/bmj.321.7258.429. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ramos K, Linscheid R, Schafer S. Real-time information-seeking behavior of residency physicians. Fam Med. 2003;35(4):257–60. [PubMed] [Google Scholar]
- 7.Ely JW, Osheroff JA, Chambliss ML, Ebell MH, Rosenbaum ME. Answering physicians' clinical questions: obstacles and potential solutions. J Am Med Inform Assoc. 2005;12(2):217–24. doi: 10.1197/jamia.M1608. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Gonzalez-Gonzalez AI, Dawes M, Sanchez-Mateos J, et al. Information needs and information-seeking behavior of primary care physicians. Ann Fam Med. 2007;5(4):345–52. doi: 10.1370/afm.681. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Green M, Ruff T. Why do residents fail to answer their clinical questions: a qualitative study of barriers to practicing evidence-based medicine. Acad Med. 2005;80(2):176–82. doi: 10.1097/00001888-200502000-00016. [DOI] [PubMed] [Google Scholar]
- 10.Hollander S, Lanier D. The physician-patient relationship in an electronic environment: a regional snapshot. Bull Med Libr Assoc. 2001;89(4):397–9. [PMC free article] [PubMed] [Google Scholar]
- 11.Bennett NL, Casebeer LL, Zheng S, Kristofco R. Information-seeking behaviors and reflective practice. J Contin Educ Health Prof. 2006;26(2):120–7. doi: 10.1002/chp.60. [DOI] [PubMed] [Google Scholar]
- 12.Chew F, Grant W, Tote R. Doctors on-line: using diffusion of innovations theory to understand Internet use. Fam Med. 2004;36(8):645–50. [PubMed] [Google Scholar]
- 13.Ebell MH. Point-of-care information that changes practice: it's closer than we think. Fam Med. 2003;35(4):261–3. [PubMed] [Google Scholar]
- 14.Alper BS. Practical evidence-based Internet resources. Fam Pract Manag. 2003;10(7):49–52. [PubMed] [Google Scholar]
- 15.White B. Making evidence-based medicine doable in everyday practice. Fam Pract Manag. 2004;11(2):51–8. [PubMed] [Google Scholar]
- 16.Diaz JA, Griffith RA, Ng JJ, Reinert SE, Friedmann PD, Moulton AW. Patients' use of the Internet for medical information. J Gen Intern Med. 2002;17:180–5. doi: 10.1046/j.1525-1497.2002.10603.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Fox S. [Accessed January 17, 2006];Health information online. 2005 May 17; Available at: http://www.pewinternet.org/pdfs/PIP_Healthtopics_May05.pdf.
- 18.Fox S. [Accessed December 15, 2006];Online health search 2006. 2006 October 29; Available at: http://www.pewinternet.org/pdfs/PIP_Online_Health_2006.pdf.
- 19.Mandl KD, Feit S, Pena BM, Kohane IS. Growth and determinants of access in patient e-mail and Internet use. Arch Pediatr Adolesc Med. 2000;154(5):508–11. doi: 10.1001/archpedi.154.5.508. [DOI] [PubMed] [Google Scholar]
- 20.Gomella LG. The wild, wild Web: resources for counseling patients with prostate cancer in the information age. Semin Urol Oncol. 2000;18(3):167–71. [PubMed] [Google Scholar]
- 21.Houston TK, Allison JJ. Users of Internet health information: differences by health status. J Med Internet Res. 2002;4:e7. doi: 10.2196/jmir.4.2.e7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Ling CA. Guiding patients through the maze of drug information on the Internet. Am J Health Syst Pharm. 1999;56(3):212–4. [PubMed] [Google Scholar]
- 23.Lorence DP, Greenberg L. The Zeitgeist of online health search: implications for a consumer-centric health system. J Gen Intern Med. 2006;21:134–9. doi: 10.1111/j.1525-1497.2005.0295.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Murray E, Lo B, Pollack L, et al. The impact of health information on the Internet on health care and the physician-patient relationship: national US survey among 1,050 US physicians. J Med Internet Res. 2003;5:e17. doi: 10.2196/jmir.5.3.e17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Eysenbach G, Diepgen TL. The role of e-health and consumer health informatics for evidence-based patient choice in the 21st century. Clin Dermatol. 2001;19(1):11–7. doi: 10.1016/s0738-081x(00)00202-9. [DOI] [PubMed] [Google Scholar]
- 26.Ford P. Is the Internet changing the relationship between consumers and practitioners? Journal for Healthcare Quality. 2000;22(5):41–3. doi: 10.1111/j.1945-1474.2000.tb00152.x. [DOI] [PubMed] [Google Scholar]
- 27.Reed MC, Grossman JM. [Accessed July 14, 2006];Growing availability of clinical information technology in physician practices. 2006 June; Data bulletin no. 31. Available at http://www.hschange.org/CONTENT/848/848.pdf. [PubMed]
- 28.Carroll AE, Christakis DA. Pediatricians' use of and attitudes about personal digital assistants. Pediatrics. 2004;113(2):238–42. doi: 10.1542/peds.113.2.238. [DOI] [PubMed] [Google Scholar]
- 29.Miller RH, Hillman JM, Given RS. Physician use of IT: results from the Deloitte Research Survey. J Healthc Inf Manag. 2004;18(1):72–80. [PubMed] [Google Scholar]
- 30.Bennett NL, Casebeer LL, Kristofco RE, Strasser SM. Physicians' Internet information-seeking behaviors. J Contin Educ Health Prof. 2004;24(1):31–8. doi: 10.1002/chp.1340240106. [DOI] [PubMed] [Google Scholar]
- 31.Crites GE, Chrisagis X, Patel V, Little D, Drehmer T. A locally created EBM course for faculty development. Med Teach. 2004;26(1):74–8. doi: 10.1080/0142159032000150467. [DOI] [PubMed] [Google Scholar]
- 32.Straus SE, Ball C, Balcombe N, Sheldon J, McAllister FA. Teaching evidence-based medicine skills can change practice in a community hospital. J Gen Intern Med. 2004;20:340–3. doi: 10.1111/j.1525-1497.2005.04045.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Schilling K, Wiecha J, Polineni D, Kahalil S. An interactive Web-based curriculum on evidence-based medicine: design and effectiveness. Fam Med. 2006;38(2):126–32. [PubMed] [Google Scholar]
- 34.Vogel EW, Block KR, Wallingford KT. Finding the evidence: teaching medical residents to search MEDLINE. J Med Libr Assoc. 2002;90(3):327–30. [PMC free article] [PubMed] [Google Scholar]
- 35.Alper BS, Vinson DC. Experiential curriculum improves medical students' ability to answer clinical questions using the Internet. Fam Med. 2005;17(8):565–9. [PubMed] [Google Scholar]
- 36.Bradley P, Oterholt C, Herrin J, Nordheim L, Bjorndal A. Comparison of directed and self-directed learning in evidence-based medicine: a randomised controlled trial. Med Educ. 2005;39:1027–35. doi: 10.1111/j.1365-2929.2005.02268.x. [DOI] [PubMed] [Google Scholar]
- 37.Brettle A. Information skills training: a systematic review of the literature. Health Info Libr J. 2003;20(suppl 1):3–9. doi: 10.1046/j.1365-2532.20.s1.3.x. [DOI] [PubMed] [Google Scholar]
- 38.Ross R, Verdieck A. Introducing an evidence-based medicine curriculum into a family practice residency—is it effective? Acad Med. 2003;78(4):412–7. doi: 10.1097/00001888-200304000-00019. [DOI] [PubMed] [Google Scholar]
- 39.Smith CA, Ganschow PS, Reilly BM, et al. Teaching residents evidence-based medicine skills. J Gen Intern Med. 2000;15:710–5. doi: 10.1046/j.1525-1497.2000.91026.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Thom DH, Haugen J, Sommers PS, Lovett P. Description and evaluation of an EBM curriculum using a block rotation. BMC Med Educ. 2004;4:19. doi: 10.1186/1472-6920-4-19. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Baldwin C, Niebuhr V, Sullivan B. Meeting the computer technology needs of community faculty: building new models for faculty development. Ambulatory Pediatrics. 2004;4(1 suppl):113–6. doi: 10.1367/1539-4409(2004)004<0113:mtctno>2.0.co;2. [DOI] [PubMed] [Google Scholar]
- 42.Steiner V, Hartmann J, Ronau T. MedReach: building an Area Health Education Center medical information outreach system for northwest Ohio. J Med Libr Assoc. 2002;90(3):317–22. [PMC free article] [PubMed] [Google Scholar]
- 43.Kronick J, Blake C, Munoz E, Heilbrunn L, Dunikowski L, Milne WK. Improving on-line skills and knowledge. A randomized trial of teaching rural physicians to use on-line medical information. Can Fam Physician. 2003;49:312–7. [PMC free article] [PubMed] [Google Scholar]
- 44.Alper BS, White DS, Ge B. Physicians answer more clinical questions and change clinical decisions more often with synthesized evidence: a randomized trial in primary care. Ann Fam Med. 2005;3(6):507–13. doi: 10.1370/afm.370. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Ebell M. Information at the point of care: answering clinical questions. J Am Board Fam Pract. 1999;12(3):225–35. doi: 10.3122/jabfm.12.3.225. [DOI] [PubMed] [Google Scholar]
- 46.Carney P, Poor D, Schifferdecker K, Gephart D, Brooks WB, Nierenberg DW. Computer use among community-based primary care physician preceptors. Acad Med. 2004;79(6):580–90. doi: 10.1097/00001888-200406000-00017. [DOI] [PubMed] [Google Scholar]
- 47.Kollisch D, Gephart D, Brooks WB, Gagne R, Allen C, Donahue D. Impact of a preceptor education board and computer network to engage community faculty at Dartmouth Medical School. Acad Med. 1999;74(1 suppl):S70–S74. doi: 10.1097/00001888-199901000-00035. [DOI] [PubMed] [Google Scholar]
- 48.Davis D, O'Brien MA, Freemantle N, Wolf FM, Mazmanian PE, Taylor-Vaisey A. Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. 1999;292(9):867–74. doi: 10.1001/jama.282.9.867. [DOI] [PubMed] [Google Scholar]
- 49.Khan KS, Coomarasamy A. A hierarchy of effective teaching and learning to acquire competence in evidence-based medicine. BMC Med Educ. 2006;6:59. doi: 10.1186/1472-6920-6-59. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Hays W. Statistics. fourth edition Harcourt Brace Jovanovich College Publishers; Fort Worth: 1988. [Google Scholar]
- 51.Ajzen I, Fishbein M. Understanding attitudes and predicting social behavior. Prentice-Hall; Englewood Cliffs, NJ: 1980. [Google Scholar]
- 52.Casebeer L, Bennett N, Kristofco R, Carillo A, Centor R. Physician Internet medical information seeking and on-line continuing education use patterns. J Contin Educ Health Prof. 2002;22(1):33–42. doi: 10.1002/chp.1340220105. [DOI] [PubMed] [Google Scholar]
- 53.Coomarasamy A, Khan KS. What is the evidence that postgraduate teaching in evidence-based medicine changes anything? A systematic review. BMJ. 2004;329(7473):1017. doi: 10.1136/bmj.329.7473.1017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Davis D, Evans M, Jadad A, et al. The case for knowledge translation: shortening the journey from evidence to effect. BMJ. 2003;327:33–5. doi: 10.1136/bmj.327.7405.33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Ebell MH, Shaughnessy A. Information mastery: integrating continuing medical education with the information needs of clinicians. J Contin Educ Health Prof. 2003;23(suppl 1):S53–S62. doi: 10.1002/chp.1340230409. [DOI] [PubMed] [Google Scholar]
- 56.Jadad AR. Promoting partnerships: challenges for the Internet age. BMJ. 1999;319:761–4. doi: 10.1136/bmj.319.7212.761. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Mangrulkar RS. Targeting and structuring information resource use: a path toward informed clinical decisions. J Contin Educ Health Prof. 2004;24(suppl 1):S13–21. doi: 10.1002/chp.1340240504. [DOI] [PubMed] [Google Scholar]
- 58.Shrauger J, Osberg TM. The relative accuracy of self-prediction and judgments by others in psychological assessment. Psychol Bull. 1981;90(2):322–51. [Google Scholar]