Abstract
Objective
The authors developed and evaluated a rating scale, the Attitudes toward Handheld Decision Support Software Scale (H-DSS), to assess physician attitudes about handheld decision support systems.
Design
The authors conducted a prospective assessment of psychometric characteristics of the H-DSS including reliability, validity, and responsiveness. Participants were 82 Internal Medicine residents. A higher score on each of the 14 five-point Likert scale items reflected a more positive attitude about handheld DSS. The H-DSS score is the mean across the fourteen items. Attitudes toward the use of the handheld DSS were assessed prior to and six months after receiving the handheld device.
Statistics
Cronbach's Alpha was used to assess internal consistency reliability. Pearson correlations were used to estimate and detect significant associations between scale scores and other measures (validity). Paired sample t-tests were used to test for changes in the mean attitude scale score (responsiveness) and for differences between groups.
Results
Internal consistency reliability for the scale was α = 0.73. In testing validity, moderate correlations were noted between the attitude scale scores and self-reported Personal Digital Assistant (PDA) usage in the hospital (correlation coefficient = 0.55) and clinic (0.48), p < 0.05 for both. The scale was responsive, in that it detected the expected increase in scores between the two administrations (3.99 (s.d. = 0.35) vs. 4.08, (s.d. = 0.34), p < 0.005).
Conclusion
The authors' evaluation showed that the H-DSS scale was reliable, valid, and responsive. The scale can be used to guide future handheld DSS development and implementation.
Introduction
Optimizing patient safety is a key goal of health services research and clinical service delivery. Reports of the Institute of Medicine emphasize the importance of information technology in achieving patient safety. 1 Specifically, clinical decision support software (CDSS) has been shown to have a positive effect on patient safety, and, in particular, on the reduction of medication errors, in acute care hospital settings. 2–5 Evidence for the effect of CDSS in the ambulatory setting is limited, 2 although preliminary research suggests a benefit. 4,6–8 Ambulatory settings provide many challenges to the use of CDSS including the need for clinician mobility, the short time available for patient visits, tight budgets, and systems that are not easy to integrate into the workflow.
Personal Digital Assistants (PDAs) or handheld devices with clinical decision support software (H-CDSS) are being proposed and marketed for use in the health care setting.
The use of PDAs, in general, has been rapidly increasing in health care. 4–6,8–15 For example, Criswell and Parchman surveyed all U.S. family medicine residency programs and reported that two-thirds of the U.S. residents were using PDAs—indicating rapid adoption of PDAs in family practice residency programs. 11 There are many medical decision support tools available for use with PDAs. Although there are few studies of CDSS on PDAs, there are reports that the more commonly used applications are drug references, medical references, personal scheduling, and medical calculators. 4,6–10,12–13
A recent survey of 2,130 pediatricians, 63% response rate, reported that 35% used PDAs at work and 40% used PDAs for personal reasons. Those using the PDAs thought that using the PDA increased efficiency and reduced medical errors. The applications that were more often used were drug references, medical calculations, and personal scheduling. 12 Yen-Chiao et al. reported similar applications in a survey designed to identify how physicians use the PDAs, what applications they use, and why applications are not used. 13
Studies have shown that there are barriers to use of these devices and that there is still resistance to use of these devices for clinical purposes. Key barriers include screen size, time to use, size, weight, ease-of-use, cost, and availability of medical applications, among others. 5,7–10,13–16,19–22,24 Other important issues that must be considered are that the CDSS should be optimized for the workflow and type of conditions seen in the ambulatory as well as the inpatient setting, including the time pressures, and that it should not have a negative affect on physician–patient interaction. Given the potential use of these devices for clinical decision support in both ambulatory and inpatient settings, it is important to investigate the effects of interventions to reduce the barriers to use in a systematic way. Many assessments of barriers to use of PDAs have been qualitative in nature and of necessity often involve small samples of respondents. For instance, Lu et al. conducted interviews with 20 physicians using the Critical Incident Technique (CIT) to identify barriers to PDA use and reported quotes that identified the barriers. 13 Not only has there been little research on methods to reduce barriers to the use of handheld CDSS in the health care setting, there has also not been a validated scale specifically designed to measure clinician attitudes and perception of handheld CDSS. Our determination of the need for a scale targeted to handheld DSS was that scales related to general computer use are unlikely to address the unique features of these devices and that a simple-to-administer tool can be used easily and quickly with a large sample of participants.
Our goal was to develop and evaluate a rating scale to assess attitudes toward the use of handheld decision support systems in ambulatory and inpatient settings. The psychometric properties of the scale, Attitudes toward Handheld Decision Support Software (H-DSS), were tested among a group of physicians-in-training before and after they were provided with a PDA. This study was conducted in the context of a randomized controlled trial looking at the impact of the DSS. 25
Methods
The study protocol, consent procedure, and instrument were approved by the University of Alabama at Birmingham Institutional Review Board.
In assessing the quality of the scale, we adapted the criteria developed by an international committee for development of health services research tools. 26 These criteria are similar to other guidelines for establishing the reliability and validity of survey instruments.
The following criteria were considered the most relevant to evaluate the proposed scale:
1 Conceptual model—e.g., the concept to be measured including the justification for item content.
2 Reliability—e.g., the degree to which the scale is free from random error; consistency of measurement.
3 Validity—the extent to which the scale measures what it purports to measure.
4 Responsiveness—the ability of the scale to detect changes over time.
The methods used to develop the scale, which we are calling the Attitudes toward Handheld Decision Support Scale, and to assess each of these criteria are described below.
Scale Content (Conceptual Model ): To identify barriers to the use of CDSS/PDA we performed an extensive search of the extant literature, followed up references in identified articles, and used the input of our team to identify articles. We identified a list of barriers that were developed into Likert-type questions. 5,7,8,15,17–24 For example, one potential barrier to adopting CDSS/PDA is the perception that these computers are too complex to use in a clinical setting. As a result, a specific question was developed to assess this notion (see Table 1 available online only at www.jamia.org). Based on the other identified barriers, further questions were developed for use in the survey. Following the preliminary construction of the scale, comments from the research team were solicited. The team members included experts in survey methods, informatics, and questionnaire development as well as clinicians, all of whom participated in modifying the format, ordering of items, clarifying the intent of words, and improving the overall layout of the scale.
The scale was then pilot tested for clarity and readability with a convenience sample of nine internal medicine resident physicians who were at an affiliated but different residency program from those involved in the subsequent validation study. Following the completion of the scale, the pilot group was instructed to carefully review the scale for ambiguities and ease-of-use, and to make comments regarding its use. These comments were used to revise the preliminary scale.
The final attitude scale consists of 14 items rated on a five-point Likert scale (see Table 1 available online only at www.jamia.org). The items measure the strength of the residents' attitudes toward handheld computers and the use of CDSS, with higher scale values for individual items representing more positive attitudes (e.g., perception of fewer barriers) to use of handheld decision support software. The H-DSS score is the mean across the fourteen items.
Study Sample
We evaluated the scale using a sample of 82 Internal Medicine residents in post-graduate years one, two, and three at an urban medical center in the southeastern United States, who volunteered to participate in a study examining the use of handheld CDSS. Recruitment materials included posters, mailings, and announcements regarding the study. Residents were recruited from a pool of 126 residents and all residents were offered the opportunity to participate in the study. Those agreeing to participate were given a PDA. The residents had ambulatory clinics that primarily serve lower income populations and are affiliated with a primary care residency program in an urban university. The residents' inpatient experiences were at various urban acute care hospitals.
Psychometric Assessment
To assess reliability, validity, and responsiveness we administered the scale twice to the same group of subjects with approximately a six month interval between administrations. In addition to the H-DSS items, we also asked other questions related to subjects' experience using a PDA including whether they had previously used a PDA, how much they used it in different settings, and what types of medical programs they had on the PDA. During the first administration, the questions on amount and type of use were completed by only those residents who currently used PDAs (n = 51) and on the second administration, all residents completed the questions (n = 77).
In addition to the H-DSS items, we also collected data on the respondents' age, gender, ethnic group, post-graduate year, primary care interest, and self-reported sophistication of computer use, which was reported on a five-point scale with 1 = Very sophisticated and 5 = Very unsophisticated.
After the first administration of the scale, residents were provided with a PDA (Palm OS) and were permitted to load any programs they wanted and use the PDA however they desired. We also asked questions on usage of the PDA (Table 2 available online only at www.jamia.org) and we analyzed the data on the second administration. The questions inquired about the residents' frequency of using the PDA outside the medical setting and accessing inpatient and outpatient medical information, and the total number of medical programs used.
Statistical Analysis
Our analysis was guided by our study objectives, to assess the reliability, validity, and responsiveness of the scale.
Reliability—In order to assess internal consistency of the overall H-DSS scale, Cronbach's Alpha reliability coefficient was computed for each administration. Reliabilities of 0.70 or greater are considered acceptable for internal consistency. 26 A factor analysis using principal components extraction with varimax rotation was also performed to confirm the latent structure of the scale.
Validity—Content validity was assessed by the panel of researchers as part of the development process as described above.
In terms of construct-related validity, we hypothesized that those residents with a higher score on the scale (e.g., a more positive attitude toward use of the PDA) would use the PDA more than those who perceived more barriers. This aspect of validity was assessed using Pearson correlations of the H-DSS score with each of the PDA usage questions, including how often the PDA was used for non-medical, inpatient and outpatient use, and total number of medical programs used. For concurrent validity, we used the second administration scores and usage data, since that was when all residents had an opportunity to use the PDA and answer those questions. For predictive validity, we then examined the correlation of the first administration H-DSS score with the usage questions at follow-up.
In order to further assess the construct validity of the scale we examined a multivariate linear regression model in which the outcome was the score on the first administration of the attitude scale. The independent factors in the model were several resident characteristics and self-reported sophistication of computer use.
Responsiveness—To assess responsiveness of the scale, we hypothesized that most of the concerns about using the handheld would be reduced by having actual experience using it. If the attitude scale were a responsive measure of perception of barriers, we hypothesized that compared to the baseline assessment, the second administration of the scale would show a decrease in perceived barriers (i.e., the scores would increase). Only residents who completed both scales were used in the analysis of responsiveness. Paired sample t-tests and a two-factor Analysis of Variance (ANOVA) were used to test for changes in the mean H-DSS scale score.
To more closely study the changes in attitudes over time, descriptive data on individual items were examined. For all items, and for each administration, the proportion of subjects who rated the item as “Strongly agree” or “Agree” was computed and reviewed.
A two-sided p-value of 0.05 was used for all statistical tests. SPSS 13.0 (SPSS Inc., Chicago) was used to analyze the data.
Results
Resident Characteristics
Seventy-five of the 82 residents completed both administrations (▶). Of those who completed the study, 77% (n = 58) were male. Although the majority of U.S. residents is also male, our male population was somewhat higher than the U.S. resident population (77% vs. 61%). 27 Residents were equally distributed among the three training years. Sixty-three percent (n = 47) of the completers indicated owning a PDA at the time of the first scale administration. The characteristics of those who completed both administrations were similar to those who did not (data not shown).
Table 3.
Table 3. Characteristics of Participants (n = 75) Completing Both Administrations of the H-DSS
| Participant Characteristic | |
|---|---|
| Male (percent) | 77.3% |
| Ethnicity (percent) | |
| White, non-Hispanic | 74.7% |
| Other | 25.3% |
| Plan to practice primary care (percent) | 20.0% |
| Mean age (standard deviation) | 28.6 (2.44) |
| Post-graduate year (PGY) (percent) | |
| PGY-1 | 30.7% |
| PGY-2 | 34.7% |
| PGY-3 | 34.7% |
| Previous PDA owners (percent) | 62.7% |
| Self-report of computer sophistication (percent) | |
| Very sophisticated | 8.0% |
| Sophisticated | 32.0% |
| Neither sophisticated nor unsophisticated | 46.7% |
| Unsophisticated | 9.0% |
| Very unsophisticated | 4.0% |
Reliability
The internal consistency reliability for those completing both of the administrations of the H-DSS was α = 0.73; internal consistency reliability for all participants completing the H-DSS at pretest and posttest were α = 0.71 each. The factor analysis showed that a single component solution (eigenvalue = 3.79 compared to the next best of 1.50) was by far the best fit for the scale. Scale items were designed for residents in training. However, we made minor revisions and found similar reliability with a small sample (n = 15) of primary care physicians.
Validity
Data on self-report of usage of PDA are reported in ▶. In addition to these three questions, residents were asked to indicate which programs they used and the total number of programs used was calculated. The mean number of programs used was 3.68 (standard deviation = 2.00) with a range of 0–9.
Table 4.
Table 4. Usage of Handheld Computers in Various Settings after Six Months of Use
| Questions: In a typical day, how often do you use your handheld computer: | Percentage of Respondents Selecting Each Response |
||||
|---|---|---|---|---|---|
| Less than one time | 1 or 2 times | 3–5 times | Six or more times | No answer | |
| For any purpose, outside the medical setting? | 22.7% | 36.0% | 21.3% | 10.7% | 9.3% |
| To access medical information in regard to your hospitalized patients? (inpatient day) | 4.0% | 8.0% | 30.7% | 48.0% | 9.3% |
| To access medical information in regard to the patients seen in your office? (clinic day) | 10.7% | 37.3% | 30.7% | 9.3% | 12.0% |
▶ shows the correlations between the three PDA usage questions during the second administration and the H-DSS scale score during each administration. As can be seen, all of the correlations are positive demonstrating low to moderate concurrent and predictive validity of the H-DSS scale. Evidence for predictive validity was somewhat weaker than for concurrent validity. That is, of the correlations reflecting predictive validity (correlations between baseline H-DSS score and usage six months later) two of the four are statistically significant, while the correlations among data collected at the same time (i.e., concurrent validity) tended to be higher (3 of 4 are statistically significant).
Table 5.
Table 5. Correlations of H-DSS Scores with Residents' Reported Usage of PDA
| Question on the use of PDA in a typical day at the six month administration | Correlations ∗ between H-DSS Score at Two Different Times † and PDA Usage at Six Months‡ |
|
|---|---|---|
| Baseline H-DSS score | Score after 6 months of use | |
| How often do you use your handheld computer for any purpose, outside the medical setting? | 0.18 (p = 0.133) | 0.36 (p = 0.002) |
| How often do you use your handheld computer to access medical information in regard to your hospitalized patients? | 0.46 (p < 0.001) | 0.55 (p < 0.001) |
| How often do you use your handheld computer to access medical information in regard to the patients seen in your office? | 0.26 (p = 0.037) | 0.48 (p < 0.001) |
| Total number of medical programs used | 0.21 (p = 0.076) | 0.20 (p = 0.087) |
∗ Correlations are between H-DSS score at baseline and at six months with self-reported PDA usage at six months.
† H-DSS score is the mean across the fourteen items on the scale.
‡ PDA usage is defined as the mean response to the three usage questions (above) on a scale of 1–4 with 4 being the highest usage.
Previous users of PDAs were associated with a more positive attitude. An independent-sample t-test for difference in means was used to determine whether the H-DSS scores differed between those who had previously owned a PDA vs. those who did not own one. The mean H-DSS score of previous owners on the first administration was significantly higher than for non-users (4.06, s.d. = 0.38 vs. 3.86, s.d. = 0.25, p = 0.012). This finding further supports the validity of the scale.
The regression model indicated that primary care (Beta = 0.26, p = 0.03) and more self-reported computer sophistication (Beta = 0.32, p = 0.007) were significantly associated with higher H-DSS scores. Variables included in the model that were not significantly associated with scale scores included age, sex, ethnicity, and year of residency.
Responsiveness
Overall the H-DSS scores at the first administration were high (mean = 3.99, s.d. = 0.35), indicating that the attitudes were positive. Nevertheless, the scores for the second administration (mean = 4.08, s.d. = 0.34) did show a statistically significant increase (paired t-test = 2.87, p < 0.005). Although the increase in scores was small, the standard deviation was also small, demonstrating that the H-DSS scale is very sensitive to detecting changes. These data provide support for the responsiveness of the attitude scale.
Consistent with our hypothesis, we found that the change was primarily a result of a decrease in perceived barriers in the residents who had not previously owned a PDA and are now “new users” (see ▶).
Table 6.
Table 6. Changes in Attitude Scale Scores for New PDA Users Compared to Previous PDA Users
| Users | N | Mean (s.d.) barrier score at baseline | Mean (s.d.) barrier score after six months | t | p |
|---|---|---|---|---|---|
| Previous PDA users | 47 | 4.06 (0.38) | 4.14 (0.35) | 1.6 | 0.12 |
| New users of PDA | 28 | 3.86 (0.25) | 3.99 (0.30) | 2.88 | 0.008 |
To provide more detail on the specific changes over time, we examined the individual items. ▶ shows the proportion of residents who “Agreed” or “Strongly agreed” with each of the items on the scale. For the group as a whole, the largest changes over time were in items 2, 3, and 5, which related to usability of the handheld.
Table 7.
Table 7. Proportion of all Responders Who Strongly Agreed or Agreed with the Barrier Statements of the H-DSS (n = 75).
| Barrier Statements on the H-DSS† | First Admin. | Second Admin. |
|---|---|---|
| 1. Handheld computers are simple enough to use in the clinic setting. | 0.92 | 0.95 |
| 2. Handheld screens are adequate to use effectively. | 0.85 | 0.96 |
| 3. Most programs on handheld computers are simple to use. | 0.75 | 0.95 |
| 4. Many medical programs available for handheld computers are worthwhile to use. | 0.89 | 0.87 |
| 5. There is enough time to use a handheld computer in the clinic. | 0.72 | 0.83 |
| ∗ 6. There are few patients for whom I would need a decision support program. | 0.30 | 0.35 |
| 7. It would be beneficial to use a computer for decision support. | 0.71 | 0.77 |
| ∗ 8. I have reservations using a handheld computer in front of the patient. | 0.23 | 0.15 |
| 9. I don't think my attending would mind if I used a handheld computer in the clinic. | 0.92 | 0.93 |
| ∗ 10. Handhelds are too large and heavy to carry around with me. | 0.01 | 0.01 |
| ∗ 11. I have too many things to hold in addition to working with the handheld computer. | 0.04 | 0.05 |
| 12. Handheld computers are reasonably priced for use in the medical setting. | 0.64 | 0.60 |
| 13. I am generally comfortable using computers. | 0.85 | 0.85 |
| 14. I am interested in learning more about using handheld computer decision support systems in the clinic. | 0.95 | 0.91 |
∗ Agreement reflects greater perception of barriers. Lower proportion of agreement reflects more positive attitudes.
† Statements are designed for education setting. A modified version has been developed for use with physicians in practice.
When analyzing the H-DSS items of the new users that changed at least 10%, we found that the new users showed positive changes in attitude towards item 2, the adequacy of the PDA screen (from 70% to 100% indicating agree or strongly agree), and item 3, the handheld's ease of use (from 80% to 90%), whereas the previous users showed positive changes on the PDA's ease of use in the clinics and using it in front of patients (data not shown).
Discussion
We developed a scale to measure attitudes toward handheld decision support tools. We found that our scale had acceptable psychometric properties including measures of reliability, validity, and responsiveness.
Subsequent to our development of the scale (2001–2003), Venkatesh et al. analyzed a variety of instruments related to adoption and use of information technology. 28 The theoretical model, based on their data, identified four dimensions that predict technology use. These dimensions are (1) performance expectancy, e.g., the extent to which the user believes that using the system will be helpful in his or her work; (2) effort expectancy, e.g., the ease of use of the system; (3) social influence, or the degree to which important others believe the user should use the system; and (4) facilitating conditions, or the perception that the user has the knowledge to use the system and that the environment facilitates its use. Although Venkatesh et al.'s conceptual framework was developed after our scale was completed, the areas they identified as predictive of use of information technology are similar to ours. 28
While the focus of our article is on the psychometric properties of the scale, the results on PDA use are generally consistent with other studies in that we found that the majority of the residents already used PDAs. 5,10–12,20 Unlike in other studies 7,10 , very few (1%) of the residents in this study had concerns about the size or weight of the PDA. This may reflect improvements in the technology since the time of the earlier studies. Recent studies have identified barriers such as battery life, privacy and security, and breakage. 10,12 Despite the initial generally positive attitudes, our scale demonstrated responsiveness and the ability to detect changes over time. Future research might examine scales with a less enthusiastic population or other health professionals.
D'Alessandro et al. studied the barriers to using computer-based digital health sciences libraries (DHSL) among rural physicians. This study was conducted five years before ours, and was assessing PC, rather than handheld, use. 20 They suggested that barriers to use of the PC, inconvenient location and lack of time and training, may be overcome by using PDAs as point of care tools. More recently, McAlearney et al. 14 conducted focus groups to describe physicians' use of PDAs in the clinical setting. They grouped the users into 4 categories (non-user, niche-user, routine user, power user) and found differences in perceptions of use and barriers to use of the PDA. Non-users tend to not see the benefits of the PDA, whereas power users tend to find using the PDA intuitive. Although Ho et al. 5 reported that residents used PDAs for medical purposes, other types of decision support tools in addition to medical calculators or drug information references are still not widely used. A scale such as the one we developed to address attitudes prior to and after the introduction of the PDA tools could provide useful data on the effect of the intervention.
Limitations
This scale validation study was conducted on a sample of internal medicine residents from a single residency program with certain demographics. As such, this may limit the generalizability of the scale—a problem common to studies of this type. In addition, the potential for recall bias is present in the self-report data on usage of the PDA.
Conclusion
The scale on attitudes toward using handheld computers for clinical decision support was developed and assessed using standard criteria for scales of this type. The scale was reliable, valid, and responsive. As more CDSS are developed for use on handheld computers, assessment of the impact of these systems on perceptions of barriers will need to be done. With the increasing use of handheld CDSS our attitude scale provides an important, simple-to-use scale for that assessment.
Footnotes
This research was supported in part and the PDAs were funded by grant R18-HS 11820 from the Agency for Healthcare Research and Quality.
References
- 1.Committee on Quality of HealthCare in America, Institute Of Medicine Crossing the Quality Chasm. A New Health System for the 21st Century 2001. Washington DC: National Academy Press; 2001.
- 2.Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomesa systematic review. JAMA 2005;293:1223-1238. [DOI] [PubMed] [Google Scholar]
- 3.Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors JAMA 1998;280:1311-1316. [DOI] [PubMed] [Google Scholar]
- 4.Rothschild JM, Lee TH, Bae T, Bates DW. Clinician use of a palmtop drug reference guide J Am Med Inform Assoc 2002;9:223-229. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Ho W, Forman J, Kannry J. Portable digital assistant use in a medicine teaching program Proc AMIA Symp 2001:1031.
- 6.Tamblyn R, Huang A, Jacques A, et al. The medical office of the 21st century (MOXXI)effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. CMAJ 2003;169:549-556. [PMC free article] [PubMed] [Google Scholar]
- 7.Sittig DF, Jimison HB, Hazlehurst BL, et al. Techniques for identifying the applicability of new information management technologies in the clinical settingan example focusing on handheld computers. Proc AMIA Symp 2000:804-808. [PMC free article] [PubMed]
- 8.Speedie S, Pacala J, Vercellotti G, Harris I, Zhou X. PDA support for outpatient clinical clerkshipsmobile computing for medical education. Proc AMIA Symp 2001:632-636. [PMC free article] [PubMed]
- 9.McLeod TG, Ebbert JO, Lymp JF. Survey assessment of personal digital assistant use among trainees and attending physicians J Am Med Inform Assoc 2003;10(6):605-607. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Barrett JR, Strayer SM, Schubart JR. Assessing medical residents' usage and perceived needs for personal digital assistants Int J Med Inf 2004;73(1):25-34. [DOI] [PubMed] [Google Scholar]
- 11.Criswell DF, Parchman ML. Handheld computer use in U.S. family practice residency programs J Am Med Inform Assoc 2002;9(1):80-86. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Carroll AE, Christakis DA. Pediatricians' use of and attitudes about personal digital assistants Pediatrics 2004;113(2):238-242. [DOI] [PubMed] [Google Scholar]
- 13.Lu YC, Lee JK, Xiao Y, Sears A, Charters K. Why don't physicians use their personal digital assistants? Proc AMIA Symp 2003:405-409. [PMC free article] [PubMed]
- 14.McAlearney AS, Schweikhart SB, Medow MA. Doctors' experience with handheld computers in clinical practicequalitative study. BMJ 2004;328(7449):1162. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Schiffman RN, Yischon L, Navedo DD, Freudigman KA. User satisfaction and frustration with a handheld, pen-based guideline implementation system for asthma Proc AMIA Symp 1999:940-944. [PMC free article] [PubMed]
- 16.Fischer S, Stewart TE, Mehta S, Wax R, Lapinsky SE. Handheld computing in medicine J Am Med Inform Assoc 2003;10(2):139-149. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Churchill BE, Jimison H, Lyman JA, Mailhot MF, Quick EA, Simpson DA. Current usage patterns and attitudes toward handheld computers in clinical care. Abstract in converging information technology and health care Proc AMIA Symp 2000. [PMC free article] [PubMed]
- 18.Ruland CM. Clinicians' use of a palm-top based system to elicit patient preferences at the bedsidea feasible technique to improve patient outcomes. Proc AMIA Symp 1999:1150Nov. [PMC free article] [PubMed]
- 19.Marshall M, Sumner S. Family practice clerkship encounters documented with structured phrases on paper and hand-held computer logs Proc AMIA Ann Symp 2001:428-432. [PMC free article] [PubMed]
- 20.D'Alessandro DM, D'Alessandro MP, Galvin JR, Kash JB, Wakefield DS, Erkonen WE. Barriers to rural physician use of a digital health sciences library Bull Med Libr Assoc 1998;86(4):583-593. [PMC free article] [PubMed] [Google Scholar]
- 21.Lindgren KN, Bissell DM, Lin CT. Handheld computer use in third year medical clerkships Proc AMIA Symp 2001.
- 22.Manning B, Gadd CS. Introducing handheld computing into a residency programpreliminary results from qualitative and quantitative inquiry. Proc AMIA Symp 2001:428-432. [PMC free article] [PubMed]
- 23.Goldblum, OM (2001) Electronic Prescribing: Criteria for Evaluating Handheld Prescribing Systems and an Evaluation of a New, Handheld, Wireless Wide Area Network (WWAN) Prescribing System. Proceedings of the American Medical Informatics Association. Dermatol Online J. 2001 Feb;7(1):1.. [PubMed]
- 24.Seckman CA, Romano CA, Marden S. Evaluation of clinician response to wireless technology Proc AMIA Symp 2001:612-616. [PMC free article] [PubMed]
- 25.Berner ES, Houston TK, Ray MN, et al. Improving ambulatory prescribing safety with a handheld decision support systema randomized controlled trial. J Am Med Inform Assoc 2006;13(2):171-179. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Scientific Advisory Committee of the Medical Outcomes Trust Assessing health status and quality-of-life instrumentsattributes and review criteria. Qual Life Res 2002;11:193-205. [DOI] [PubMed] [Google Scholar]
- 27.Brotherton SE, Rockey PH, Etzel SI. US graduate medical education, 2003–2004 JAMA 2004;292(9):1032-1037. [DOI] [PubMed] [Google Scholar]
- 28.Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technologytoward a unified view. MIS Quart 2003;27(3):425-478. [Google Scholar]