Abstract
We conducted a prospective observational study to (1) determine usage and construct validity of a method to gauge the cognitive impact of information derived from daily e-mail, and (2) describe self-reported impacts of research-based synopses (InfoPOEMs) delivered as e-mail. Ratings of InfoPOEMs using an Impact assessment scale provided (a) data on usage of the impact assessment method, (b) reports of impact by InfoPOEM and by doctor and (c) data for analysis of construct validity of the scale. Participants were family physicians or general practitioners who rated at least five InfoPOEMs delivered on e-mail. For each InfoPOEM rated, 0.1 continuing education credit was awarded by the College of Family Physicians of Canada.
Use of the impact assessment scale linked to a daily InfoPOEM was sustained during the 150-day study period. 1,007 participants submitted 61,493 reports of ‘cognitive impact’ by rating on average 61 InfoPOEMs (range 5–111). ‘I learned something new’ was most frequently reported. ‘I was frustrated as there was not enough information or nothing useful’ was the most frequently reported negative type of impact. The proportion of reports of ‘No Impact’ varied substantially across individual InfoPOEMs. Impact patterns suggested an 8 or 9-factor solution.
Our Impact assessment method facilitates knowledge transfer by promoting two-way exchange between providers of health information and family doctors. Providers of health information can use this method to better understand the impact of research-based synopses. Sustaining current practice and increasing knowledge about new developments in medicine are important outcomes arising from research-based synopses delivered as e-mail, in addition to practice change.
Introduction
To keep up-to-date, many family doctors read synopses of original research on e-mail. 1 Synopses read on e-mail may raise awareness of new developments, contribute to continuing medical education and improve professional practice. 2 While behavior change interventions in medicine have received much attention, we know relatively little about the cognitive impact of research synopses, when delivered on e-mail to the doctor. 3 Indeed, e-mail alerts are one type of computer-mediated communication, and research on the impact of this type of communication in medicine is in its infancy. 4
In our previous work, we developed and tested a method to gauge the impact of information that doctors retrieved from medical databases, using a pop-up questionnaire containing ten items of ‘cognitive impact’. 5,6 In the context of information retrieval, we defined ‘cognitive impact’ in accordance with a generic ‘Acquisition-Cognition-Application’ model where health professionals absorb, understand and integrate information items. 7 While the context of reading e-mail differs from information retrieval, a systematic approach to understand the ‘cognitive impact’ of research-based synopses delivered as e-mail would contribute to research on research utilization. A valid outcome measure of the impact of research-based synopses delivered as e-mail could help doctors to earn credit for this activity, while providing feedback for authors.
We conducted the following study to explore the usage and validity of a method for systematically assessing the self-reported cognitive impact of e-mail alerts. In addition to taking a systematic approach, we conceptualize the ‘cognitive impact of information items’ to be a multidimensional construct as opposed to a one-dimensional construct (e.g., Impact? ‘Yes’ or ‘No’). 8
Methods
Synopses of original research, called InfoPOEMs, were first delivered on e-mail to 12,800 members of the Canadian Medical Association (CMA) in 2005. Designed for practitioners, an InfoPOEM (Patient Oriented Evidence that Matters) is a one page synopsis of research relevant to primary care, delivered daily, Monday to Friday.
In this study, all CMA members who received InfoPOEMs via e-mail as of September 2006 were eligible to participate. On September 15th and October 3rd 2006, the CMA e-mailed an invitation to participate to all addresses on their list. After completing a demographic questionnaire and providing informed consent online, CMA members who read InfoPOEMs could begin rating them by clicking a link in the top left corner of the e-mail page containing each InfoPOEM. This link connected the reader to a ten-item impact assessment scale developed in our previous work (▶). The reader could then report one or more than one item of impact (‘check all that apply’), with one exception. When “No Impact” was selected, no other item of impact could be chosen. Each rating of one InfoPOEM was a self-report of cognitive impact comprised of a single item or a combination of items of impact.
Figure 1.
Study Overview.
A participant was defined as a practicing family physician or general practitioner who submitted at least five ratings of InfoPOEMs from September 8 2006—February 4 2007. All 111 InfoPOEMs e-mailed during the study period were eligible for rating. Reports of impact were collected by the CMA, and forwarded weekly to the principal investigators. For each rated InfoPOEM, participants certified by the College of Family Physicians of Canada earned 0.1 Mainpro M1 credit.
We calculated descriptive statistics to assess usage of the impact assessment method. The frequency of ‘method use’ was calculated as the number of ratings during the study period. We counted the number of different impact patterns, defined as one or more than one item of impact reported by at least one participant for one InfoPOEM. To examine the construct validity of the 10-item impact scale, we applied principal component analysis (PCA) with varimax rotation. We identified the best solution for reported impact patterns and compared it to a proposed theoretical factor structure. 5 Our global construct is the cognitive impact of health information (e.g., an InfoPOEM read by a medical doctor), and is captured by ten items. We evaluated the robustness of the PCA solution to the dependence in ratings by using exploratory multilevel factor analysis (MFA). 9 We performed one MFA accounting for dependence due to multiple ratings from each participant and another MFA accounting for dependence due to multiple ratings for each InfoPOEM.
The study protocol was approved by the McGill University Faculty of Medicine Institutional Review Board.
Results
The 1,007 participants rated an average of 61 InfoPOEMs (range 5–111). This group submitted 61,493 reports of ‘cognitive impact’ (▶). Nine hundred and thirty eight participants reported they were currently doing Family Practice/General Practice (93.1%). Their average age was 46 years; 598 (59.4%) were men, 409 were women and 393 (39%) were certified by the College of Family Physicians of Canada. As compared to the population of Canadian family physicians, participants were more likely to report utilization of electronic reminder or warning systems in their practice (▶). 10 Reported usage of electronic health records was congruent with the general population of Canadian family physicians.
Figure 2.
Number of weekly ratings of InfoPOEMs on e-mail submitted by Canadian family physicians.
Table 1.
Table 1 Reported Use of Information Technology
| Number (%) of ‘Yes’ Answers | ||
|---|---|---|
| “In Your Main Patient Care Setting, I …” | Participants (n = 1,007) | 2004 National Physician Survey (n = 11,041) |
| use electronic patient health records | 172 (17.1) | 1,811 (16.4) |
| use electronic reminder systems for recommended patient care | 201 (20.0) | 1,049 (9.5) |
| use electronic warning systems for adverse prescribing and/or drug interactions | 197 (19.6) | 1,568 (14.2) |
Reports of ‘Cognitive Impact’: All InfoPOEMs
Use of the method was sustained throughout the study (▶). A single item of impact was reported in 47,496 ratings (77.2%). The ten most frequent impact patterns accounted for 89.4% of all ‘cognitive impact’ reports (▶). Not surprisingly, ‘I learned something new’ was most frequently observed (35.2% of all impact patterns). The most frequently reported negative impact pattern was ‘I was frustrated as there was not enough information or nothing useful’ (1.8% of all reports). The most frequently reported combination of two items of impact was ‘I learned something new’ AND ‘My practice will be improved’ (9.4% of all reports). In total, 85 different impact patterns were observed.
Figure 3.
Percentage of participants who submitted at least one rating of an InfoPOEM per study week.
Table 2.
Table 2 Frequency Distribution of Top Ten Reported Patterns of Impact
| Pattern of Impact | Number | % of Total |
|---|---|---|
| I learned something new | 21,657 | 35.2% |
| No Impact | 10,508 | 17.1% |
| This information confirmed I did (will do) the right thing | 5,886 | 9.6% |
| I learned something new AND My practice will be improved | 5,793 | 9.4% |
| I was reassured | 3,435 | 5.6% |
| My practice will be improved | 2,356 | 3.8% |
| I recalled something because of this POEM | 1,948 | 3.2% |
| This information confirmed I did (will do) the right thing AND I was reassured | 1,634 | 2.7% |
| I was frustrated as there was not enough information or nothing useful | 1,120 | 1.8% |
| I learned something new AND I was reassured | 639 | 1.0% |
| 54,976 | 89.4% |
Reports of ‘Cognitive Impact’ by InfoPOEM
One hundred and four (10.3%) participants reported at least once ‘I disagree with this information’ while 81 (8.0%) reported at least once ‘I think this information is potentially harmful’. There was substantial variation in the occurrence of ‘No Impact’ across individual InfoPOEMs. For example, considering all reports received for each InfoPOEM delivered in September 2006, the proportion of ‘No Impact’ spanned a range from a low of 4% (Orthotics not effective for plantar fasciitis—delivered September 13, 2006) to a high of 58% (Maternal shoe size not related to infant birth weight—delivered September 19, 2006).
Impact Patterns and Types of ‘Cognitive Impact’ by Participant
Patterns of impact varied greatly by participant. For example, one participant rated 102 POEMs and submitted reports corresponding to 28 different impact patterns. Only two people responded in such a way as to suggest they were not truly reflecting on the impact of InfoPOEMs. For example, one participant rated 101 POEMs and reported ‘I learned something new’ by itself, 101 times. The propensity to report the item ‘My practice will be improved’ varied the most between doctors (intraclass correlation = 0.2).
Factor Analysis
The PCA showed that an 8-factor solution explained 89.8% of total variance and was the best fit for the data. Two items were grouped together: ‘This information confirmed I did (will do) the right thing’ and ‘I was reassured’. ‘No impact’ and ‘I learned something new’ were both correlated with the same factor, but with opposite signs. This can be explained by the fact that ‘No impact’ could not be selected simultaneously with any other item, yielding a relatively strong negative correlation with the most frequently reported item ‘I learned something new’.
The remaining six items of impact measured different dimensions of the construct under consideration, namely ‘cognitive impact’ of information. A similar 8-factor solution was a good fit for the data when using MFA accounting for dependence in ratings due to multiple ratings from each participant. However, in a separate MFA accounting for dependence due to multiple ratings for each InfoPOEM, the presence of several moderately large residual correlations for the 8-factor solution suggested a 9th factor may be needed. ‘This information confirmed I did (will do) the right thing’ and ‘I was reassured’ would then measure different dimensions.
Discussion
In this study of the cognitive impact of research-based synopses delivered as daily e-mail to practicing doctors, our method for systematically rating their impact was frequently used. This is perhaps not surprising, as 14.9% of Canadian family physicians reported using online continuing medical education courses in 2004. 10
Our results may be interpreted in accordance with three types of research-based information use: Instrumental, Conceptual and Legitimating. 11–16 Instrumental use involves changing action in specific ways based on research results. Conceptual use involves using research results for general enlightenment, while legitimating use involves using research results to reinforce a predetermined position. Based on this framework, we found that conceptual use was frequently reported (I learned something new). However, research-based synopses delivered as daily e-mail may also have instrumental effects on practice (My practice will be improved) or they may legitimate practice (This information confirmed I did (will do) the right thing). While legitimating current practice does not change decision-making, it can be associated with an important effect or influence on the doctor, such as validating a decision already made. Thus, (1) sustaining current practice, and (2) increasing knowledge about new developments in medicine, are important outcomes that information providers should expect from this type of computer-mediated communication, in addition to practice change. 17 The main limitation of self-reported impact is the degree to which these reports reflect actual change in knowledge, attitude and behaviour. For example, when doctors report ‘My practice will be improved’ linked to an e-mail alert, we may or may not be able to document an observable change in behaviour or patient outcome. 18 In addition, we did not measure the amount of time participants took between the reading and the rating of their InfoPOEMs. In the absence of CME credit, use of the method may be less impressive.
As a result of using our method to systematically assess the cognitive impact of InfoPOEMs, we see potential to improve the targeting of synopses to the practice needs of family doctors. For example, our participants were in a position to use information derived from InfoPOEMs, yet they reported ‘No impact’ 17% of the time. Abstracting services strive to provide synopses of research that are easy to read and filtered for relevance. At present, this push of research findings to a heterogeneous group of doctors is not targeted to individual needs. If InfoPOEMs were further targeted to the needs of individuals, the impact of this type of ‘push’ communication could be further enhanced. In practical terms, this means that authors and information providers should strive to deliver more relevant information to minimize the risk of information overload. Psychologically, as the proportion of material that is regarded as personally relevant is increased, greater attention to the message should result. 19
In this study, doctors reported a high degree of conceptual use of InfoPOEMs on e-mail. Moving into the context of patient care, awareness of synopses of primary research may facilitate their direct instrumental use. For example, one trial recently demonstrated that e-mail alerts stimulated information retrieval by clinicians. 20 In the intervention arm of this trial, doctors receiving e-mail alerts conducted more searches in the McMaster PLUS database, as compared to those who received only passive guides to evidence-based literature. Exactly how the effect of e-mail alerts stimulated searches at some later time was not reported. The small effect of alerts on information retrieval (0.77 more logins/month/user) is compatible with the notion that clinicians retrieved items of information they had previously encountered as alerts. Only one other randomized trial has examined the effect of e-mail alerts in medicine. In this trial, weekly synopses of clinical research did not influence the attitudes of academic internists with respect to Evidence-Based Medicine or self-reported use of evidence in practice. 21 The authors recommended “further work to develop and validate more outcome measures” and suggested “future interventions should include interactive components with auditing and feedback”. Proponents of linking push communication of new knowledge (e.g., e-mail alerts) with information retrieval of accumulated knowledge in medicine have successfully stimulated debate. 4,22,23 However, we know little empirically about the use of synopses in medicine, due to a lack of research in real-world settings. If sustained changes in awareness, knowledge or attitude arise from reflecting on synopses of primary research, we believe it will be possible to link conceptual use arising from e-mail delivery with subsequent instrumental use in clinical practice. Indeed, InfoPOEMs may be retrieved using commercial search engines at the moment-of-need.
Conclusion
Our findings support usage of our method linked to research-based synopses such as InfoPOEMs delivered as daily e-mail. Our study reveals that ‘cognitive impact’ is a multi-dimensional construct that can be systematically assessed by practicing doctors. Future use of our assessment method could help us understand how to maximize the impact of information delivery for continuing medical education. Providers of health information can use this method to obtain reader feedback to understand the impact of research-based synopses in the context of ‘push’ communication or information retrieval. 24
Acknowledgments
The authors thank Yuejing Meng for expert assistance with data collection and analysis, Tara Bambrick for help with revisions, and John Aikman for administrative support.
Footnotes
This study was funded by the Canadian Institutes of Health Research (CIHR). Dr. Pluye holds a CIHR New Investigator Award. The study was supported by Practice Solutions Web Services, a subsidiary of the Canadian Medical Association. Practice Solutions Web Services made data collection possible through cma.ca. All authors, except Dr. Mercer, declare no competing interest. Dr. Mercer is employed by Practice Solutions Web Services.
References
- 1.Smith R. A POEM a week for the BMJ BMJ 2002;325(7371):983Nov 2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Haynes RB. Of studies, syntheses, synopses, and systems: the “4S” evolution of services for finding current best evidence ACP J Club 2001;134(2):A11-A13. [PubMed] [Google Scholar]
- 3.Grol R, Wensing M, Eccles M. Improving Patient Care: The Implementation of Change in Clinical PracticeElsevier Limited; 2005.
- 4.Herring SC. Computer-Mediated Communication on the Internet Ann Rev Inform Sci Technol 2002;36:109-168. [Google Scholar]
- 5.Pluye P, Grad RM, Stephenson R, Dunikowski L. A new impact assessment method to evaluate knowledge resources AMIA Annu Symp Proc 2005:609-613. [PMC free article] [PubMed]
- 6.Pluye P, Grad RM, Mysore N, Knaapen L, Johnson-Lafleur J, Dawes M. Systematically Assessing the Situational Relevance of Electronic Knowledge Resources: A Mixed Methods Study J Am Med Inform Assoc 2007;14(5):616-625Sep 1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Saracevic T, Kantor KB. Studying the value of library and information services. Part I. Establishing a theoretical framework. J Am Soc Inf Sci 1997;48(6):527-542. [Google Scholar]
- 8.Pluye P, Grad RM, Dunikowski L, Stephenson R. Impact of clinical information-retrieval technology on physicians: a literature review of quantitative, qualitative and mixed-method studies Int J Med Inf 2005;74(9):745-768. [DOI] [PubMed] [Google Scholar]
- 9.Reise SP, Ventura J, Nuechterlein KH, Kim KH. An illustration of multilevel factor analysis J Pers Assess 2005;84(2):126-136. [DOI] [PubMed] [Google Scholar]
- 10. National Physician Survey Ref Type: Internet Communication. 2005. 5-4-2006.
- 11.Weiss C. Using Social Research in Public Policy MakingLexington: Lexington Books; 1977.
- 12.Pelz DC. Some expanded perspectives on use of social science in public policyIn: Yinger M, Cutler SJ, editors. Major Social Issues: A Multidisciplinary View. New York: Free Press; 1978. pp. 346-357.
- 13.Knorr KD. Policymakers’ Use of Social Science Knowledge: Symbolic or InstrumentalIn: Weiss C, editor. Using Social Research in Public Policy Making. Lexington, MA: Lexington Books; 1977. pp. 165-182.
- 14.Estabrooks CA. The Conceptual Structure of Research Utilization Res Nurs Health 1999;22:203-216. [DOI] [PubMed] [Google Scholar]
- 15.Hivon M, Lehoux P, Denis JL, Taillez S. Use of health technology assessment in decision making: Coresponsibility of users and producers? Int J Health Technol Assess 2005;21(2):268-275. [PubMed] [Google Scholar]
- 16.Landry R, Amara N, Pablos-Mendes A, Shademani R, Gold I. The knowledge-value chain: a conceptual framework for knowledge translation in health Bull World Health Organ 2006;84(8):597-601. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Hakkennes S, Green S. Measures for assessing practice change in medical practitioners Implementation Science 2006;1(29). [DOI] [PMC free article] [PubMed]
- 18.Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of Physician Self-assessment Compared With Observed Measures of Competence: A Systematic Review JAMA: J Am Med Assoc 2006;296(9):1094-1102Sep 6. [DOI] [PubMed] [Google Scholar]
- 19.Kreuter MW, Holt CL. How Do People Process Health Information?. Applications in an Age of Individualized Communication. Curr Dir Psych Sci 2001;10(6):206-209. [Google Scholar]
- 20.Haynes RB, Holland J, Cotoi C, McKinlay J, Wilczynski N, Walters LA, et al. McMaster PLUS: A cluster randomized trial of an intervention to accelerate use of evidence-based information from digital libraries J Am Med Inform Assoc 2006;13(6):593-600. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Mukohara K, Schwartz, MD. Electronic delivery of research summaries for academic generalist doctors: a randomised trial of an educational intervention Med Educ 2005;39(402):409. [DOI] [PubMed] [Google Scholar]
- 22.Slawson D, Shaughnessy A. Teaching Evidence-Based Medicine: Should We Be Teaching Information Management Instead? Acad Med 2005;80(7):685-689. [DOI] [PubMed] [Google Scholar]
- 23.Shaughnessy AF, Slawson DC. Are we providing doctors with the training and tools for lifelong learning? BMJ 1999;319:1280. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Grad RM, Pluye P, Meng Y, Segal B, Tamblyn R. Assessing the impact of clinical information-retrieval technology in a Family Practice Residency J Eval Clin Pract 2005;11(6):576-586. [DOI] [PubMed] [Google Scholar]