Survey of user preferences from a comparative trial of UpToDate and ClinicalKey

INTRODUCTION

In December of 2011, the A. T. Still Memorial Library at the A. T. Still University of Health Sciences was invited to be a beta test site for ClinicalKey, a major new point-of-care and research platform recently introduced by Elsevier. The advance marketing for ClinicalKey implied its use as a point-of-care-product for physicians. Over the past several years, library staff had been exploring an institutional subscription to UpToDate, so we decided to conduct a side-by-side trial of the two products and to survey participants about their experiences and preferences if the library were able to subscribe to only one of the two. Survey questions were designed to ascertain not only the participants' overall preference, but to provide detailed information to the library staff concerning the basis of this preference, including ease of access and use and the relative value of the two products in the students' didactic and clinical years.

ClinicalKey <http://info.clinicalkey.com/access/> is a new product launched by Elsevier in April 2012. It brings together the content from a number of Elsevier's medically related products, including MD Consult, First Consult, Procedures Consult, over 500 clinical journals, and more than 950 medical electronic books. It is being marketed to physicians as “Faster, more relevant clinical answers, so you can spend less time searching and more time caring for patients,” in an apparent attempt to compete with UpToDate as a point-of-care clinical tool. UpToDate <http://www.uptodate.com/home/about-us> is a point-of-care physician evidence/decision tool developed in 1992. In 2008, it was purchased by Wolters Kluwer, a large global information services and publishing company. It provides detailed, clinically related information written by physicians for physicians to use in clinical settings. It has been heavily marketed directly to physicians and is considered by many physicians as the key evidence-based, point-of-care online tool. There have been numerous studies on the value of UpToDate in clinical settings 1–5. This brief communication adds to this literature by presenting the results of a side-by-side trial of these two products based on reports by undergraduate medical students and their faculty.

METHODS

In February 2012, the library initiated trials of UpToDate and the beta version of ClinicalKey. Students and faculty of the university's two osteopathic medical schools (Kirksville College of Osteopathic Medicine [KCOM] and School of Osteopathic Medicine in Arizona [SOMA]) and its physician's assistant program were informed of the trials and invited to participate in a survey to provide the library with feedback for its consideration in choosing to subscribe to one or both of the products.

On March 23, the announcement of the availability of the survey as well as the uniform resource locator (URL) for access was sent to the 94 people who had indicated an interest in participating in the trial. Over 300 responses to the survey were received as word-of-mouth information on the survey circulated among students and faculty of the 2 medical schools. The survey was open from March 23 until April 9. The survey (Appendix, online only) included background questions concerning participants' program affiliation, status (faculty or student), and current activities (didactic or clinical), as well as several questions concerning use of, satisfaction with, and preference for one or the other product. The Likert-type response metric for satisfaction questions ranged from 1 = very dissatisfied to 5 = very satisfied. Responses for the questions concerning recommendations to students ranged from 1 = always recommend UpToDate to 5 = always recommend ClinicalKey; 3 indicated no preference. For the question concerning preference for purchase, responses ranged from 1 = strongly recommend UpToDate to 5 = strongly recommend ClinicalKey; 3 indicated no preference. The survey was posted on SurveyMonkey <http://www.SurveyMonkey.com>. The university's institutional review board determined the survey was exempt from review.

ANALYSIS

Survey responses were downloaded from the SurveyMonkey database and imported into SPSS, version 20 (SPSS, an IBM Company, Chicago, IL), for analysis. Means (standard deviations) and counts (percentages) are reported, as appropriate. Mann-Whitney tests were used for between-subjects comparisons. Wilcoxon tests were used for within-subject comparisons, and stepwise multiple regression analyses were conducted to evaluate the contribution of individual items to the global satisfaction score. A P-value of 0.05, 2-tailed, was used to designate statistical significance.

RESULTS

The survey was completed by 23 faculty members and 292 students. Among the 292 students who responded to the survey, 208 (71.2%) were in the classroom or didactic portion of their program, while 84 (28.8%) were in clinical rotations. Twelve (66.7%) of the 18 faculty who responded to the question about their status were teaching in the classroom, while 6 (33.3%) were working with students in a clinical setting.

Respondents scored UpToDate (4.7±0.70) slightly higher than ClinicalKey (4.2±1.1) on overall satisfaction (P<0.001). The direction of mean differences was consistent across satisfaction questions. The largest difference favoring UpToDate was seen for ease of use and efficiency in answering specific questions. The smallest difference was seen in satisfaction with comprehensiveness of the programs.

Product recommendations to colleagues

While 56.9% of respondents indicated that they would always or typically recommend UpToDate to students, 30.0% indicated they would always or typically recommend ClinicalKey (P<0.001). However, examination of response patterns showed that those in a clinical rotation or teaching in the clinic strongly favored UpToDate, while those in a didactic venue favored ClinicalKey.

In response to a question concerning which product they would prefer if the library could purchase only one, 53.6% of respondents indicated (Figure 1) that they recommended or strongly recommended UpToDate, while 30.8% recommended or strongly recommended ClinicalKey (P<0.001). Responses, again, differed depending on the current status of the student or faculty.

Figure 1.

Frequency of responses to “How often would you recommend UpToDate versus ClinicalKey?”

KCOM = Kirksville College of Osteopathic Medicine

SOMA = School of Osteopathic Medicine in Arizona.

Stepwise linear regression analysis was conducted on the satisfaction scale items (see question 7 in the online only appendix for a listing of the nine characteristics included among the satisfaction question) to determine which aspects of ClinicalKey and UpToDate were most strongly associated with overall satisfaction. The 9 items accounted for 75% of the total variance in overall satisfaction for UpToDate and 87% of the variance for ClinicalKey, clearly indicating that user evaluation of the products accounted for user preferences that were indicated in the other questions in the survey. User friendliness, comprehensiveness, and ease of use weighed most heavily in overall satisfaction for both platforms.

Question 8 asked, “Thinking only of the products you have used in column one below, please indicate your preference for that product versus ClinicalKey and UpToDate.” The other point-of-care products listed were ACP PIER/StatRef, First Consult/MD Consult, Access Medicine, DynaMed/EBSCO, and Essential Evidence Plus/Info Poems. Respondents to the ClinicalKey comparison (n = 210) preferred ClinicalKey between 73% and 79% of the time, except for the comparison to First Consult/MD Consult, for which ClinicalKey was preferred only 55% of the time. Subsequent discussion with faculty and students indicated that they preferred the way First Consult/MD Consult was formatted over the ClinicalKey format.

The questions on previous use of UpToDate indicated that almost half of the respondents had used it previously, and 72.8% of these respondents had used it at least once a week.

The last question in the survey was an open-ended question directing respondents to “Please provide specific reasons for your satisfaction or dissatisfaction with UpToDate and ClinicalKey. Include examples, where possible.” The comments to this question reflected findings from the closed-ended questions. Fifty respondents indicated a preference for UpToDate, thirty-eight for ClinicalKey, and twelve equally for both. The provided reasons showed a strong preference for UpToDate in clinical settings but ClinicalKey in the didactic phase of medical education.

Sample comments

Representative samples of the relevant comments are presented below.

UpToDate comments:

We need Up-to-Date for students. It's essential for training to make us better 3rd/4th year students, as nearly every site uses it and expects students to be proficient with it early on. Many schools have it, so we should ensure that our students aren't a step lower than others on rotations.

One of the things I prefer about UpToDate is that it condenses all the most recent journal findings into a concise, quickly accessible article. In terms of point of care resources, UpToDate delivers recommendations in a timely manner. ClinicalKey requires a little too much sifting; often you will have to dig through several journal or Medline articles, some of which aren't even available to view.

ClincalKey comments:

Clinical Key is amazing!!!! It is so easy to use. Searching for articles couldn't be easier, and unlike with PubMed, you don't have to worry about finding “Free full text” articles, as all the articles that come up in Clinical Key are free full text. Access to the online versions of SO MANY textbooks is invaluable. It will save students so much money and time, and allows us to access resources that we would not normally access due to lack of funds or time. PLEASE purchase clinical key. It makes medical education SO much easier.

Clinical Key provided access to textbooks I wish I could have afforded and have access to throughout the year. After using it, I believe that it would be extremely useful when I am on rotations. I believe it is a great tool and will help the KCOM community using it.

Comments by respondents who liked both products:

A bit like comparing apples and oranges. UptoDate is fantastic for quick reference on dz and treatment options. Clinical Key has access to text for first and second years that is phenomenal. So I think it depends on where you are in your rotations.

I have found that many rotations expect me to have access to UptoDate which is best for answering clinical questions as they come up on rotations. First Consult is great for a concise explanation of a topic that I am studying or reviewing and I use it almost daily (especially since it is available on my phone). Clinical Key fills a huge need for a good way to search for journal articles (this has been difficult to say the least in the past). All three products fill different but important roles.

Respondents liked UpToDate's organization and depth in answering clinical questions. The ClinicalKey open-ended responses indicated that it was seen as a strong tool for information support by students in the didactic or classroom phase of the training. They liked the fact that it contained a strong collection of medical text and reference books, including many of their textbooks, combined with full-text journal content and other useful content such as Procedures Consult and a large collection of medical videos and images that can be easily searched simultaneously. The respondents favorably noted the fact that much of the e-book content can be downloaded in a portable document format (PDF) that supports comments and highlighting of content.

CONCLUSION

The response to both products was positive. UpToDate was preferred in the clinical setting. The open-ended responses indicated that not only was it heavily used in the clinical setting, but that students were expected to use it and were at a competitive disadvantage if they did not have access to it. While the breadth and depth of the full-text content of ClinicalKey was well received, its use as a point-of-care product was limited by the fact that the survey respondents did not find First Consult, which represents the major point-of-care content in ClinicalKey, to be at all comparable to UpToDate. If Elsevier wants to seriously challenge UpToDate's position as the dominant point-of-care clinical information tool for physicians, it appears they will need to expand and upgrade First Consult.

The library decided to subscribe to both products to serve different purposes. To enable the library to subscribe to both of these products in a year with a relatively tight budget, the library cancelled a significant portion of its print and digital journal collections. Over the past ten years, the university, like other academic health sciences centers, has been shifting to greater use of academic technology in its instruction and greatly expanding its distance programs and number of distance-based students. There has also been an increasingly strong emphasis in clinical settings on a more evidence-based model of care. The addition of these two strong and well-received clinical tools has enabled the library to shift its resources to better support these changes.