Use and Perceived Benefits of Handheld Computer-based Clinical References

Abstract

Objective

Clinicians are increasingly using handheld computers (HC) during patient care. We sought to assess the role of HC-based clinical reference software in medical practice by conducting a survey and assessing actual usage behavior.

Design

During a 2-week period in February 2005, 3600 users of a HC-based clinical reference application were asked by e-mail to complete a survey and permit analysis of their usage patterns. The software includes a pharmacopeia, an infectious disease reference, a medical diagnostic and therapeutic reference and transmits medical alerts and other notifications during HC synchronizations. Software usage data were captured during HC synchronization for the 4 weeks prior to survey completion.

Measurements

Survey responses and software usage data.

Results

The survey response rate was 42% (n = 1501). Physicians reported using the clinical reference software for a mean of 4 years and 39% reported using the software during more than half of patient encounters. Physicians who synchronized their HC during the data collection period (n = 1249; 83%) used the pharmacopeia for unique drug lookups a mean of 6.3 times per day (SD 12.4). The majority of users (61%) believed that in the prior 4 weeks, use of the clinical reference prevented adverse drug events or medication errors 3 or more times. Physicians also believed that alerts and other notifications improved patient care if they were public health warnings (e.g. about influenza), new immunization guidelines or drug alert warnings (e.g. rofecoxib withdrawal).

Conclusion

Current adopters of HC-based medical references use these tools frequently, and found them to improve patient care and be valuable in learning of recent alerts and warnings.

Introduction

Information technology (IT) has become increasingly important in patient care. ¹ The accelerated adoption of IT in healthcare is due to several factors including the need for rapid access to information in the context of an exponential growth in medical knowledge, especially pharmacotherapy, advancements in the capabilities of medical IT devices (hardware) and the development of more sophisticated applications (software). ^2,3 Handheld computers (HCs), or personal digital assistants (PDAs), are particularly appealing to clinicians because they are relatively easy to use, portable, fast, affordable, readily updated, and can store large quantities of information. ⁴

To date, the most common functions for HCs in clinical settings are to provide electronic clinical reference guides such as pharmacopeias for drug information, patient tracking tools, billing-capture tools, portable databases for conducting research and educational resources such as medical textbooks. ⁵ The role of HCs in patient care is evolving as the technology permits greater use of wireless connectivity and as more patient data are stored in digital formats. Newer HC capabilities include electronic prescribing, real-time electronic medical records access and point-of-care evidence-based literature searches.

We previously conducted a survey in 2000 of HC early-adopters who were then using a new HC clinical reference drug guide. ⁶ Since then, the growth of HC use in clinical practice has increased substantially with an estimated 57% of physicians in 2004 using HCs in their practice. ⁷ We conducted a new study of HC reference application for several reasons: HCs are increasingly being used in clinical practice; users have undoubtedly become more experienced in HC use since 2000; there has been great expansion in HC software programs for different clinical needs; and we are now able to measure actual user behavior.

Methods

Data were collected from 2 sources; an online survey to assess physicians’ experience with HC clinical references and downloaded tracking data from HCs to capture actual application usage patterns. The tracking data for HC usage behavior was collected during synchronization through a desktop computer to the Epocrates website when the participants updated their HC software. We compared individual survey findings to the captured usage data for each participant.

Clinical Reference Applications

Epocrates Inc. is a provider of handheld and Web-based clinical reference tools. Our group previously studied a free drug reference guide (Epocrates Rx) that was compatible only with Windows PCs and HCs using the Palm OS platform. For this study, we evaluated a suite of software that requires an annual subscription and can also be used with Macintosh PCs and HCs with the Microsoft Pocket PC platform. The clinical reference software we studied has 4 applications: an enhanced version of the drug reference guide or pharmacopeia, an infectious disease reference guide, a medical diagnostic and therapeutic reference guide and a messaging feature that is transmitted along with software updates during synchronizations.

The current version of the free drug reference guide includes a database of over 3,300 drug monographs that provides drug dosing recommendations, contraindications, adverse reactions, cost information, mechanism of action, drug–drug interactions (DDIs) and access to health insurance plan formularies. The enhanced subscription drug reference guide (Rx Pro) also includes over 400 alternative/herbal medicines with drug interactions, an embedded weight-based dosing calculator and integration with the infectious disease treatment guide (ID). Use of the diagnostic and therapeutic reference guide (Dx) requires an additional fee and includes content from an existing source (Griffith’s 5-Minute Clinical Consult) and is also integrated with the pharmacopeia. The messages include alerts from several federal agencies such as the Food and Drug Administration (FDA) and the Centers for Disease Control (CDC). Examples of alerts include drug recall alerts such as rofecoxib (Vioxx) withdrawal or public health information such as recent influenza vaccination guidelines.

Study Design and Study Participants

During February 2005, we conducted a survey of 29,000 physicians with subscriptions to the complete set of clinical reference applications and who also had previously participated in previous Epocrates’ surveys. A randomized sample of 3600 subscribers (12.4%), generated by using the random selection function in Microsoft Access, received an email invitation to participate in our survey. Participants completed an online survey and allowed the collection of synchronized tracking data that provided user behavior for the prior 4 weeks. Participants had a 2-week window of opportunity to participate in the study followed by a 4-week period to synchronize their HCs.

The survey had 5 sections: demographic information including physician specialty and years in practice; experience with healthcare technology including other HC applications; frequency of HC clinical references applications’ use; perceived clinical benefits of point-of-care clinical references, such as improved patient care or reduced medication errors; perceived benefits of the message alert feature and the perceived benefits of more sophisticated HC-based clinical applications such as portable electronic health records or electronic prescribing with HCs. Survey questions regarding the Dx application were restricted to physicians who also subscribed to this application (n = 878, 24%). Seven-point Likert scales were used for assessing the degree of agreements with statements; 1 = strongly disagree, 4 = neither disagree nor agree and 7 = strongly agree. Physicians were asked to provide recent examples of HC clinical reference applications use resulting in improved or safer patient care. Physicians were also asked to provide their “wish list” for technologic improvements to HCs or other wireless portable computers that would further enhance their delivery of healthcare.

Data Collection

Anonymous study identifiers linked the survey responses to synchronized usage tracking data for each participating physician. Survey data were collected electronically through a Web-based survey tool. Following agreement to participate in the study, usage data was collected for each participating physician for the 4-week period that preceded the completion of the survey and most recent HC synchronization. No other personal HC data was downloaded. Patient level information was not included and investigators were blinded as to the identification of the users.

The “key press” histories while using the clinical reference guide were collected, and included: identification of drugs searched in the pharmacopeia, use of embedded tools such as the drug-drug interaction (DDI) tool, type of infection or disease searched for in the infectious disease and diagnostic and therapeutic guides, respectively. User behavior included the frequency of looking up drugs, infections or other diseases in the applications and the subcategories and features within each application. The clinical reasons for actual HC reference application use were not assessed. Human research committee approval was obtained prior to conducting the study.

Statistical Analysis

Survey questions regarding the frequency of use of the infectious disease and diagnostic and therapeutic resources for the prior 4 weeks were compared to the actual application usage at the physician level. The pharmacopeia is much more frequently used and therefore we asked physicians to estimate the average daily number of drug lookups during the prior 4 weeks. In order to compare the estimated reported daily use frequency to the measured 4-week download history, we divided the number of unique drug lookups by the days of recorded HC use for each physician. The days of recorded use were a proxy for the number of patient care days for each clinician in the prior 4-weeks. This adjustment did not account for clinical workdays when the clinical reference software was not used at all. The Chi-square statistic and the 2-sample t-test were used to evaluate differences in the demographic characteristics of the survey respondents and non-respondents. Correlation of reported and actual use was compared using the Spearman rank correlation. Data were stored in Microsoft 2003 Access and Excel databases. Statistical analysis was conducted using SAS V.8.

Results

Characteristics of the Survey Participants

A total of 1501 (42%) physicians responded to the survey, including 453 who also subscribed to Dx (52%). Evaluation of the demographic characteristics of the respondents (▶) showed that the respondents were predominantly male, mid-career physicians and nearly all practiced outpatient care in all or part of their practice. Respondents’ demographics were similar to non-respondents except for practice specialty (▶). We did not have practice location data for the non-respondents. When compared to the most recent national demographic data (2003), survey respondents were more likely male (86 vs. 74%) and practiced in family/general practice (27 vs. 12%) and the internal medicine subspecialties (20 vs. 6%) but less commonly practiced surgery (6 vs. 13%). ⁸ Respondents had a mean of 3.9 years of experience with the HC clinical references (▶). In addition to Epocrates software, other HC medical references were used by 2/3 of respondents as well as other IT tools such as e-mail to communicate with patients (31%). Overall, 39% (n = 577) of respondents reported using HC clinical references during more than half of clinical encounters.

Table 1.

Table 1. Demographics of Survey Recipients

	Survey Respondents	Non-Respondents	p-value
Number (%)	1501 (42)	2097 (58)
Age in years mean (SD)	46.9 (8.6)	46.5 (8.2)	0.16
Male gender n (%)	1290 (86)	1843 (88)	0.09
Years in practice after training mean (SD)	15.8 (8.6)	14.5 (8.4)	<0.0001
Practice specialty n (%)			<0.0001
Family or general practice	403 (27)	544 (26)
Internal medicine—general	232 (15)	378 (18)
Internal medicine—subspecialties	304 (20)	504 (24)
Pediatrics	135 (9)	114 (5)
Obstetrics—gynecology	109 (7)	111 (5)
Surgery	92 (6)	63 (3)
Other	226 (15)	383 (18)
Practice location n (%)
Inpatient	91 (6)	NA
Outpatient	708 (47)	NA
Mixed	702 (47)	NA

Table 2.

Table 2. Handheld Computer (HC) and Related Information Technology Experience among Respondents

Years of HC ownership mean (SD)	5 (1.4)
Years with HC clinical reference software experience mean (SD)	3.9 (1.1)
Use e-mail with patients n (%)	461 (31)
Use of other HC software n (%)
Other medical references	992 (66)
Evidence-based decision tools	532 (35)
Patient tracking	286 (19)
Billing and coding	287 (19)
Wireless connectivity to an EMR or other real-time patient data source	141 (9)
Electronic prescribing with the HC	127 (8)
Proportion of clinical encounters associated with the use of HC clinical reference software n (%)
≤25%	510 (34)
26–50%	404 (27)
51–75%	266 (18)
≥76%	321 (21)

Abbreviations: HC=handheld computer or personal digital assistant; EMR=electronic medical record.

Clinical Use of Clinical Reference Applications

Among physicians who participated in the survey, 1249 (83%) synchronized their HCs during the 4-week data collection period. The mean frequency for using the pharmacopeia was 6.3 (SD 12.4) unique drug lookups per day. Regarding the other references, the diagnostic and therapeutic guide, infectious disease guide and the DDI checking tool were used a mean of 8.4 (SD 14), 4 (SD 5.2) and 2.9 times (SD 6.4) respectively during the span of the prior 4 weeks.

The correlation of reported use to actual measured use was poor for the frequency of unique drug lookups in the pharmacopeia (r = 0.36) but better for the DDI tool (0.50) and the diagnostic and therapeutic reference tool (0.49; ▶). It is likely that the recall for pharmacopeia use was less reliable because of the more frequent drug lookups over the course of the prior 4 weeks and the survey asking the user to estimate the average daily use within that timeframe.

Table 3.

Table 3. Use of Handheld Computer Clinical Reference Software: Self-reported Estimates Compared to Actual Usage Patterns

Reported Average Use of Clinical Reference	Actual Tracked Use of Clinical Reference ∗
	Respondents ∗∗n (%)	Mean n (SD)	Median n	Spearman Correlation
Pharmacopeia use for drug information lookups per day∗∗∗				r = 0.36; p < 0.0001
None	2 (0.2)	3.3 (3.2)	3.3
1–2	283 (23)	3.7 (9.2)	2
3–5	402 (32)	5.1 (8.5)	2.7
6–10	250 (20)	7.7 (14.5)	3.8
More than 10	312 (25)	9.1 (16.1)	4.2
Total	1249	6.3 (12.4)	2.9
Pharmacopeia use of the drug-drug interaction tool in the prior 4 weeks				r = 0.50; p < 0.0001
None	175 (14)	0.13 (1.3)	0
1–2	321 (26)	0.80 (2.0)	0
3–5	276 (22)	2.5 (4.7)	1
6–10	199 (16)	3.3 (4.6)	2
More than 10	274 (22)	7.5 (10.7)	4
Total	1245	2.9 (6.4)	0
Diagnostic and therapeutic information resource use in the prior 4 weeks				r = 0.49; p < 0.0001
None	29 (10)	2.1 (1.9)	1
1–2	70 (25)	4.1 (4.3)	3
3–5	72 (26)	6.9 (7.6)	4
6–10	58 (21)	9.1 (10.1)	6
More than 10	46 (16)	18.7 (27.5)	12
Total	275	8.4 (14)	14

^∗ Actual usage patterns were collected by synchronization of the handheld computer to track the prior 4 weeks of clinical reference software data use.

^∗∗ Of the 1501 survey respondents, 1249 (83%) had synchronized their HCs in the prior 4 weeks and included 275 (61%) of the 453 subscribers to the Dx guide.

^∗∗∗ On average, the clinical reference software was used on 13 of the 28 days. For each physician, the daily use was calculated by dividing their 28-day total usage by the number of days he/she used any of the clinical references.

The medication classes most commonly looked up while using the pharmacopeia were antimicrobials (12%), cardiovascular drugs (12%), dermatologic drugs (9%), analgesics (7%), hematologic/oncologic drugs (7%) and endocrine/metabolism drugs (7%). The organ systems most commonly looked up with ID were ophthalmologic (15%), central nervous system (13%), cardiac (13%) and gastrointestinal (13%). Among the 652 disease categories in Dx reference, the most commonly looked up diseases were influenza (8%), migraine (6%), diverticular disease (6%), Crohn’s disease (6%), pharyngitis (5%), atrial fibrillation (5%), pertussis (5%), essential hypertension (5%) and psoriasis (5%).

Perceived Patient Benefits from Using Clinical Reference Applications

Physicians were surveyed as to their perceptions of how frequently during the prior 4 weeks the HC clinical reference guides helped to improve the quality and safety of their patient care. The most frequently beneficial guides were the pharmacopeia (mean 5.6 episodes) followed by the Dx (4.2) and ID reference (4.1). Physicians who used the clinical references more frequently also reported more frequent benefits in improved care. Physicians who reported using the pharmacopeia more than 10 times a day (n = 312; 25%) reported that the guide resulted in safer patient care a mean of 8.3 episodes in the prior 4 weeks. Similar associations between frequency of use and perceived benefits were seen with the Dx and ID guides. Specific examples of reported HC use resulting in improved or safer care are provided in ▶.

Table 4.

Table 4. Examples of Reported Improved Medication Prescribing Resulting from Using the Handheld Computer Clinical Reference Software

Category	Examples of responses
Prevention of drug–drug interactions	The pharmacopeia revealed a potentially life-threatening interaction between a quinolone and amiodarone.
	I prescribed a different antibiotic after finding a significant interaction between methotrexate and trimethoprim/sulfamethoxazole.
Drug dosing recommendations	Helped me insure the appropriate intravenous to oral steroid conversion in a patient with Crohn’s disease.
	Using the glomerular filtration rate (GFR) calculator led to a 50% reduction in the allopurinol dose for a patient.
Prevention or early recognition of an adverse drug event	A patient developed pancreatitis on nitrofurantoin and with no other risk factors. Pancreatitis was listed as a complication and the drug was stopped.
	I discovered that ipratropium is contraindicated in patients with a peanut allergy. This saved me from sending home a peanut sensitive asthmatic child on ipratropium.
Pregnancy and lactation restrictions	I did not prescribe ciprofloxacin to a breast-feeding mother after finding it is a Category C drug.
	Assisted in safely prescribing anti-malaria prophylaxis for pregnant patients.
Education	I have used the pharmacopeia extensively in an academic intensive care unit to determine if specific medications may be contributing to observed adverse effects…medication changes were made promptly….decreasing morbidity. This allowed better teaching and more rapid dissemination of knowledge among team members.
	I use the pharmacopeia regularly to educate patients as to drug risks and interactions when patients take over the counter and herbal/alternative products.

Physicians were asked to judge the value of the message alerts delivered to their HCs during synchronization. Using a 7-point Likert scale, physicians were asked to judge if alerts were valuable in improving patient care or providing important up-to-date information for their patients or staff. Urgent notification of public health warnings (5, SD 1.5) and drug alert warnings (5, SD 1.5) were judged higher than patient safety news (4.8, SD 1.5) or CDC announcements such as bioterrorist alerts (4.4, SD 1.5).

Potential Future Enhancements to Using Handheld Computers in Clinical Practice

Physicians were asked to assess the clinical usefulness of potential future enhancements in their HCs with the caveat that the technology would be able to provide security for patient confidentiality and privacy. Using a 7-point Likert scale, physicians agreed that having patient-specific data (e.g., diagnoses, allergy lists and laboratory test results) connected with wireless integration to the HC-based clinical reference guides would improve productivity and efficiency, reduce medical errors, improve communication about patients between clinicians and improve patient outcomes (mean 5.4–5.7). The respondents also agreed that similar benefits would be accrued by also having point-of-care access to guidelines, online medical texts and article search and retrieval capabilities such as Pubmed (mean 5.2–5.5) and that further integration of HCs with computerized order entry, electronic prescribing and/or decision support (DS) would enhance care (mean 5.4–5.8). Examples of responses to a wish list of future HC and wireless technology enhancements to improve the delivery of patient care are provided in ▶.

Table 5.

Table 5. Potential Benefits of Enhanced Handheld Computer Technology: The Clinicians’ Wishlist for Future Improvements

Category	Examples of responses
Increased point-of-care availability of clinical data	A simple electronic medical records program that would be accessible from several locations (office, nursing home, hospital).
	Patient charts that can be accessed by all providers taking care of a particular patient to help decrease duplication of tests and to increase knowledge of all current medical problems and medications.
Enhanced wireless connectivity	Ability to communicate directly to HCs used by the majority of specialists with whom I routinely consult.
	Continuous link to online health information such as “Up To Date”.
Electronic prescribing and wireless computerized order entry	Electronic prescribing with direct transmission to patient’s pharmacy. Prescription writing integrated with a drug reference guide.
Voice and handwriting recognition	Voice recognition for order entry and dictation and a link to my office EMR. Handwriting recognition so notes could be hand written but would appear in type.
More intelligent prescribing	Software that would let you enter diagnoses and help select medications for patients with multiple medical problems.
	A diagnostic program that would link a possible adverse drug event and the drug interaction program.
Improved hardware	Larger screen, longer battery life, faster, lighter, better graphics and hack proof.
	Wearable computer: size of the HC; power of the tablet PC; wireless; with virtual display; voice recognition; natural language processing; integrated medical/health record; clinical reference guides, Medline search and artificial intelligence capabilities.

Abbreviations: HC=handheld computer; EMR=electronic medical record.

Discussion

One of the greatest challenges in modern medicine is keeping up to date with the growing medical knowledge base, and evaluations have demonstrated that clinicians have many unanswered questions. ⁹ Tools such as HC references may begin to fill this gap. The current study found that two-fifths of physicians reported using these references in more than half of encounters. Physicians self-reported that this helped them deliver higher quality and safer care.

Since our prior survey in 2000 of users of a then free HC pharmacopeia, ⁶ the current users of the subscription clinical reference guide have more experience with the applications (3.9 vs. less than 1 year) and have been in practice longer (15.8 vs. 7.8 years). It is not surprising that older physicians are later adopters of information technology tools such as HC reference guides. Two other possible explanations may account for the difference in practice experience between the two studies. The current software includes more features than the earlier free pharmacopeia software and may be more attractive to clinicians many years out of training. In addition, the current study evaluated software requiring an annual fee and may be more affordable to physicians who are longer out of training.

A 2004 survey by Forrester Research found that physicians use HCs 5 times more than the general population and that physicians who were female, older, surgeons and practice in small offices are less likely to use HCs. ⁷ Our survey found similar demographic patterns among responders. The high use of HCs during patient encounters was especially notable. Patients are becoming more comfortable with physicians using computers during encounters ¹⁰ and patients also have favorable attitudes on the use of HCs during these encounters. ¹¹ There are now many HC clinical reference applications on the market including at least 6 infectious disease references, 13 general drug information references, and specialized drug references for nursing, cardiology, pediatrics, geriatrics, psychiatry and oncology. ¹² Our findings of high HC utilization are in contrast to a 2002 survey of Mayo Clinic internal medicine physicians that found infrequent daily use of HC pharmacopeias and medical references among attending physicians. ¹³

The increased use of HCs by physicians is following similar recent diffusion patterns seen with other types of healthcare IT such as electronic medical records. ¹⁴ A recent RAND study ¹⁵ found that certain attributes of innovations are more closely associated with the exponential growth of healthcare IT: the perceived practice advantages these technologies provide clinicians; the compatibility of technologies, such as HCs, with existing workflow and healthcare systems; and acceptable complexity or relative ease of using and seeing other clinicians use these technologies in daily practice. This is what would be predicted by Rogers’ theory of diffusion of innovations. ¹⁶

Several studies have suggested that HC applications can improve patient care. Burdette and colleagues evaluated 6 commercial HC infectious disease resources and found that 5 of the programs each provided 95% or more of the expert recommendations regarding treatment of suspected infectious diseases. ¹⁷ Direct wireless access to the clinical data repository in a hospital’s information systems with HCs is thought to reduce one of the proximal causes of medical errors, limited access to current patient information including communications from colleagues. ¹⁸ Grasso and colleagues, in an earlier study of the accuracy of discharge medication lists, found that HCs improved medication reconciliation by reducing transcription errors. ¹⁹

Two recent studies evaluated HC use in providing decision support (DS) in very different clinical settings. A recent Australian study found that HC-based DS improved antibiotic prescribing in critical care. ²⁰ The HC DS included local antibiotic use guidelines, historical data of unit-specific antibiograms providing infection susceptibility patterns, up-to-date microbiology laboratory test results and a program to predict the risk assessment of antibiotic treatments for suspected ventilator-associated pneumonia. Berner and colleagues demonstrated that the use of an HC-based DS tool for predicting non-steroidal anti-inflammatory drug-related gastrointestinal risk assessments and treatment recommendations resulted in safer treatment decisions. ²¹

The tremendous amount of information required by prescribers for the risks of all possible drug–drug combinations is beyond the capabilities of human memory. ²² Drug–drug interactions are an important cause for admission of elderly patients hospitalized for drug toxicity. Examples of DDIs resulting in severe adverse drug events include hypoglycemia from glyburide and co-trimoxazole, digoxin toxicity from digoxin and clarithromycin and hyperkalemia from potassium-sparing diuretics and ACE inhibitors. ²² The DDI tool embedded within the pharmacopeia can potentially reduce such preventable events and respondents in our survey found this tool to be especially valuable in providing safe care.

Future growth of the penetration of HC applications into clinical practice will need to address some of the real and perceived barriers to using HCs. Many of the barriers (e.g., ease of use and value) can be overcome by institutional support such as assistance with purchasing decisions of particular applications, training and help desk support, providing mobile access and integrating hospital information systems into HC based applications. In fact, the overwhelming majority of HC use in clinical practice to date has taken place without institutional financial or technology support. ²³

Unlike most other types of information technology tools used in hospitals and other healthcare settings, individual physicians who purchased the HCs and software themselves have largely determined HC use. ²³ The lack of widespread institutional support has been blamed as one of the contributors to the previous slow adoption of HCs. ²⁴ Organizations are starting to take a more active role in how HCs are used in their institutions, especially with real-time patient-level data becoming more accessible with wireless HCs. An example of such institutional support is Columbia University Medical School where students are given HCs which are loaded with more than 30 applications and reference resources. ²⁵

However, there are several concerns that organizations must address in order to support HCs as part of their information technology infrastructure. When asked to provide a “wish-list” for future HC improvements, our survey respondents raised several concerns. Concerns included worry about security, especially around confidentiality of patient data, economic concerns such as future costs for software and HC maintenance and technical concerns such as device limitations and integration with other rapidly changing technologies. ²³

This study had several limitations. While the size of the survey response was robust, the proportion of respondents was only fair and may have resulted in a biased sample of the survey population. For example, respondents may have been more frequent HC users than non-responders. An additional limitation is that it is possible that some of the current survey respondents participated in our prior survey. However, only 128 (8.5%) of the respondents and 166 (7.9%) of the non-respondents were using the Epocrates pharmacopoeia at the time of the March 2000 survey. We were unable to compare the software use patterns for 252 (17%) of the survey respondents. Estimates of daily pharmacopeia use for each physician were based on their 4-week total divided by the number of days of use of any of the clinical reference software. It is possible that physicians saw patients on days when the software was not used and may have resulted in an overestimation of the average daily usage. In addition, the 4-week sample of actual usage may have been less than the perceived use recorded in the survey either because the 4-week period was not representative of HC use just prior to the survey and/or clinicians were unable to reliably recall usage frequencies for such a long stretch of time. Multiple drug or disease lookups may have been used for an individual patient or a single drug lookup may have been used in the care of several patients. Therefore, we cannot determine the rate of the HC software use per patient. The study was conducted for a single software suite and may not be generalizable to other HC-based clinical reference guides. Several steps were taken to reduce potential bias resulting from participation of investigators working for Epocrates. The Epocrates co-authors were blinded to the actual physician sampling (see earlier). In addition, the non-Epocrates authors designed the survey and methodology for assessing the HC usage data and had final editorial decisions.

In conclusion, HC-based clinical reference applications were used frequently by survey respondents and judged to be helpful in delivering safer patient care. As the technology permits more integration of patient-specific information and decision support tools improve, it is likely that these IT devices will become more integrated into the routine workflow of patient care. Future studies are needed to assess the impact of HC reference software on patient outcomes.