Abstract
An advanced PDA support system for the triage was designed to evaluate users’ acceptance of this system compared with a traditional terminal system. Davis’ Technology Acceptance Model was used to evaluate users’ acceptance. All 72 ER nurses in a 2700-bed medical center were invited for the study. The results showed that the PDA system was easier to operate than the terminal one, but had worse interface. The subjects showed significantly greater willingness to accept the terminal system instead of the PDA system. The comparative acceptance of PDA, compared with that of the old system, might still be marginally too low if its interface couldn’t be improved or no other unique practical benefits could be verified.
INTRODUCTION
The emergency department (ED) has been the major place for people to get urgent, life-saving or non-office-hour medical care. The quality emergency care begins with a well-designed triage to balance the efficiency of quick spot check and thorough, comprehensive examination1.
The Personal Digital Assistant ( PDA) appears to be a valuable tool because it can support point-of-care and information processing2. Since the PDA is still new in the medicine3, it would be practically important to develop a PDA support system for the triage and to evaluate users’ acceptance in order to examine its possibility of success 4–7.
The purposes of this study were (1) to design a custom-made, usability-engineered, wireless, intelligent, and comprehensive triage PDA support system and (2) to evaluate the ED nurses ’ acceptance of this new system in terms of Davis ’ Technology Acceptance Model (TAM)5,6, compared with a traditional terminal -based alternative support system. The TAM model proposed that two factors, perceived ease of use and perceived usefulness of system, could reliably and validly predict user’s acceptance of new information technology 4, 7–12.
METHODS
Background
The development and evaluation of systems was implemented in the ED of a 2700-bed medical center in Taipei, Taiwan. There were 72 nurses in the ED, including 20 certified triage nurses. They served around 300 patients per day. The triage used to be done in a semi-structured manner on a paper form.
Development of the PDA System
The Palm-based handspring Visor Prism PDA, installed with a Chinese system, was chosen for testing because of its color mode and expandability with a 802.11b wireless module (Intel/Xircom). A PC with Windows 98 was set up as a server to backup data locally, to print results in a networked printer in the triage area (paper medical record was still mandatory by our law), to display results on the monitor, and to upload data to IBM DB2 v7.2 mainframe database in IBM OS/390 through a ODBC connection. The NS Basic for Palm 2.1 was used to design the application. The VB 6.0 was used for the server programs. Access 2000 was used for the database management in PC. A task force, compos ed of one senior ED triage nurse, one experienced system analyst and programmer, and 2 experienced hospital IT staff, was teamed up. Their suggestions were also collected to improve the system. An intensive prototyping approach was used to assure that the system was developed based on real user needs. All major designs were well understood and approved by the senior ED triage nurse.
The PDA support system was designed in a structured way that 203 items of the most popular chief complaints, 47 items of the demographic data and 20 items of the vital signs were organized in 18 main screens under 4 categories, as shown in Table 1. Another auxiliary 23 items were designed to further simplify the data entry. Due to the difficulty of entering Chinese characters using the Graffiti® input approach, which was mainly designed for English data entry, a special PDA information representation approach was developed13. For example, hand writing data entry was replaced by the method of clicking the predefined pushbuttons ; and user should be able to switch to any screen in no more than 2 clicks. The official triage guideline published by the Ministry of Health in Taiwan was used to design a table-based expert system to support the triage judgment.
Table 1.
The information structure for the comprehensive triage PDA support systems.
| Main Category | PDA Screen |
Information Items |
Auxilliary Items |
||
|---|---|---|---|---|---|
| Data Items and Chief Complaints | # | Data Items | # | ||
| A. Basic | A1. Personal | Sex, ID, etc. | 4 | ||
| A2. Allergy/Delivered By and From | Food Allergy, Medicine Allergy, Delivered by ERT, etc. | 15 | Allergen | 10 | |
| Delivered From | 13 | ||||
| A3. Past History | Hypertension, DM, etc. | 17 | |||
| A4. Accompanied by | Family, Friends, etc. | 11 | |||
| B. Vital Sign | Blood Pressure, Pulse, etc. | 20 | |||
| C. Traumatic Chief Complaints and Symptoms | C1. Wound Mechanism | Snake- bit, dog- bit, etc. | 21 | ||
| C2. Car Accident | Drinking driving, etc. | 19 | |||
| C3. Cause | Suicide, violence, etc. | 11 | |||
| C4. Wound Type | Tearing, etc. | 7 | |||
| C5. Wound Location | Head, Fore-arm, etc. | 11 | |||
| D. Non-Traumatic Chief Complaints and Symptoms | D1. Symptoms | Chest Pain, etc. | 26 | ||
| D2. Urinary | Lost Control, etc. | 16 | |||
| D3. Feature | Vague Sight, etc. | 11 | |||
| D4. Nerve | Bewildered, etc. | 18 | |||
| D5. Skin | Wet and cold, Pale, etc. | 18 | |||
| D6. Mental/Poison | Over-dosage, etc. | 20 | |||
| D7. Gastro-intestinal | Qualm, Colic, etc. | 16 | |||
| D8. Bone and Others | Arthritis, etc. | 9 | |||
| 4 | 18 | 270 | 23 | ||
Design for the Traditional Terminal Alternative System
A terminal-based alternative system, using the same information structure in Table 1, was developed to work with hospital’s IBM OS/390 mainframe. The system was run in PC installed with PCOM 3270 terminal emulator having interface that was not GUI. This was a simple data-entry system that no expert system was built in.
Subjects
All 72 ED nurses were formally invited in this study. They were given 4 formal runs of training courses to get familiar with the PDA system and the correct operation of the system. The first 2 4-hour runs of testing and orientation of the PDA protocol system were given in 6/2001 and 12/2001. All subjects were asked to practice 20 real paper cases using PDA support system after the training. The third run of training was implemented in a simulated wireless environment in 4/2002 after the system was technically validated. The last 4-hour run was a detailed training of the final version of system. Since all ED nurses were very familiar with the terminal environment, only 2 runs of training for the terminal system were given in 3/2002 and 4/2002.
Questionnaire Design and Analysis
A questionnaire composed of 36 questions was developed referring the work done by Davis6: 19 questions were related to the perceived ease of use, 16 related to the perceived usefulness of system, and 1 related to the willingness to use. All were used to measure user’s acceptance. Among the 19 questions of ease of use, 6 were related to the interface factor, 12 related to the operation of system, and 1 was the overall score for this category of questions. Among the 16 questions of usefulness, 8 were related to the content usability, 7 related to the efficiency, and 1 was the overall score. Four experts, specialized in nursing care, technology management, health care management and information management, examined the content validity of the questionnaire. A 7-point Likert scale was used for all questions: 1 meant the worst, 7 the best, and 4 the medium. Twelve out of 72 randomly selected ED nurses were asked to test the reliability of the questionnaire. The survey was made in 5/2002. The SPSS 10.0 for Windows was used for analysis.
RESULTS
The PDA System
The structure of PDA system is shown in Figure 1, in a format of snow crystal, which emphasized the information structure of system, the feature for user to switch to any other screen in 2 clicks, and the potential of expanding the system with more screens in a well-organized manner.
Figure 1.
The snow -crystal like presentation of screen structure of PDA system.
The sample screen shot of the PDA system is shown in Figure 2, which was the screen of Wound Mechanism (C1 in Table 1) as the main category of traumatic chief complaints. In Figure 2, the screen name was displayed on the top left in the reversed color. Clicking the corresponding pushbutton in the central chief complaint area entered the chief complaint . The recommended triage level was 1 based on formall guideline. The user could switch to any screen within the system by first clicking the bottom left button to change the main category, and then clicking the bottom right button to the specific screen. The user could enter the Result Summary screen, Figure 3, and conclude the case at any time by clicking the acuity score.
Figure 2.
The screen shot of the Wound Mechanism screen (C1 in Table 1).
Figure 3.
The screen shot of the Result Summary screen that displays 17 clicked chief complaints.
The Terminal Alternative System
Figure 4 shows one of the screens of the alternative terminal system. The interface was not GUI. However, the ED nurses were very familiar with this design because it was the typical screen of the hospital information system. Nine screens, with larger Chinese font size, were designed in this system.
Figure 4.
The typical screen shot of the terminal triage support system, compared with the screen size of the PDA in the lower right corner.
The Evaluation of User’s Acceptance
The Cronbach’s α’s for the questions related to the perceived ease of use for the PDA system, the perceived usefulness of PDA system, the perceived ease of use for the Alternative system, and the perceived usefulness of Alternative system were 0.91, 0.93, 0.89, 0.92, respectively. These results are compatible to previous reports 5, 10.
The response rate was 94% (68 out of 72) , but only the results from 63 ER nurses were used for the analysis due to the data quality problem. The average age for the respondents was 31.2 years with an average of 7.6 years of experiences in the ED. 95% were females. 78% used computers for their work several times a week and 27% used computers often and everyday. Only 22% were familiar with the PDA system, compared with 71% with the terminal system. 36% agreed that the speed of the PDA system was acceptable, compared to 98% for the terminal system.
The results of survey are shown in Table 2. Though all the subjects agreed that the PDA system was easier to operate than the terminal one, it had worse interface and overall ease of use. The usefulness was not significantly different between both systems. The subjects were more willing to use the terminal system.
Table 2.
User’s acceptance of the PDA vs. Terminal Systems (n=63).
| Question Category | Number of Questions in Category | Average(SD) Score Within Category for PDA | Average(SD) Score Within Category for Terminal | Mean(SD) of (PDA-Terminal) Category Score Difference | t^ |
|---|---|---|---|---|---|
| Ease of Use | 19 | 4.08(0.46) | 4.52(0.45) | −8.30(8.89) | −7.41** |
| Interface | 6 | 4.15(0.77) | 4.98(0.65) | −5.00(5.00) | −7.94** |
| Easy to operate | 12 | 4.40(0.48) | 4.28(0.41) | 1.46(5.71) | 2.031* |
| Overall Ease of Use | 1 | 4.27(1.12) | 4.76(0.87) | −0.49(0.8) | −4.88** |
| Usefulness | 16 | 4.29(0.72) | 4.38(0.65) | −1.49(9.49) | −1.25 |
| Useful contents | 8 | 4.47(4.01) | 4.51(0.64) | −0.29(4.8) | −0.47 |
| Improved efficiency | 7 | 4.01(0.92) | 4.17(0.82) | −1.13(4.76) | −1.88 |
| Overall Usefulness | 1 | 4.63(1.07) | 4.71(1.02) | −0.08(0.63) | −1.00 |
| Willingness to accept | 1 | 4.54(1.18) | 5.10(0.98) | −0.56(1.28) | −3.45** |
Pair t-test, degree of freedom = 62, two - tail test;
p<0.5,
p<0.1
DISCUSSIONS AND CONCLUSIONS
The evaluation of user’s acceptance of PDA, compared with other alternatives, is important for us to choose the best strategy to develop useful tools in clinical settings 3.
We had made much effort to design a technically advanced triage PDA support system that was equipped with the 802.11b wireless module to enhance its mobility and a built-in expert system to support the triage judgment. An information representation approach was even developed to improve the interface of PDA13. The function to support comprehensive triage was also embedded into the system to assure the clinical usability of system. Detailed training courses were given to the ED nurses to make them more familiar with the PDA system. On the contrary, far less attention was given to the terminal alternative system, which was supposed to be given away soon. However, the study results showed very interestingly imbalanced rewards.
Though the subjects might agree that the PDA system was easier to operate than the terminal one — thanks to our efforts to improve its usability, the “obsolete” interface of terminal system was still supposed “better” than that of PDA such that the overall perceived ease of use favored the terminal system. Both system showed no significant difference in usefuleness, as they used the same information structure in the same design to support comprehensive triage. The terminal system had no guideline-based expert system. Althought his expert system was purposely designed to “support” the triage nurses to make “better” judgment, apparently they did not appreciate it yet. On the whole, the subjects showed a higher willingness to accept the alternative terminal system. The technical advantages invested in the PDA system did not win their appreciation though they were interested in PDA and eager to try it.
The screen of the terminal system was quite plain and was not GUI, as shown in Figure 4, compared to that in the PDA system as shown in Figure 2 and 3. However, the PDA system was defeated by the terminal system in all aspects regarding the interface, for example, font size, screen size, screen content arrangement, data entry (typing vs. handwriting), and screen maneuvering. These are indeed the features, as well as the weakness, of PDA. The traditional terminal system is favored partly because the subjects were more familiar with it.
However, if we examine the subjects’ acceptance of the PDA system alone, the study showed no discouraging results. The subjects still expressed very positive attitude to almost all questions on the ease of use and usefulness of the PDA system.
Health professionals might continue to show interests in PDA3 to invent more creative uses. However, the comparative acceptance of PDA, compared with the traditional alternative system, might still be marginally too low due to its interface constraints. Therefore, more creative design or improvement of its interface, such as a speech recognition technique to enter data in natural language, apparently needs to be done. To make the PDA practically accepted , benefits to improve patient care and health quality might need to be well examined and verified.
REFERENCES
- 1.Gilboy N, Travers D, Wuerz R. Re-evaluating triage in the new millennium: a comprehensive look at the need for standardization and quality. Journal of Emergency Nursing. 1999;25:468–473. doi: 10.1016/s0099-1767(99)70007-3. [DOI] [PubMed] [Google Scholar]
- 2.Chang P, Wu SC, Chou P, He CM. Empirical study of the use of PDA in community medicine. Taiwan Journal of Public Health. 2002;20:27–35. (in Chinese) [Google Scholar]
- 3.Fischer S, Stewart TE, Mehta S, Wax R, Lapinsky SE. Handheld Computing in Medicine. J Am Med Inform Assoc. 2003;10:139–149. doi: 10.1197/jamia.M1180. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Adams DA, Nelson RR, and Todd PA: Perceived usefulness, ease of use, and usage of information technology: A replication. MIS Quarterly June, 1992:227–247.
- 5.Davis FD: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, September, 1989: 319–340.
- 6.Davis FD. User acceptance of information technology: System characteristics, user perceptions and behavioral impacts. International Journal of Man-Machine Studies. 1993;38:475–487. [Google Scholar]
- 7.Dillon TW, McDowell D, Salimian F, Conklin D. Perceived ease of use and usefulness of bedside-computer systems. Computers in Nursing. 1998;16:151–156. [PubMed] [Google Scholar]
- 8.Davis FD, Bagozzi RP, Warshaw PR. User acceptance of computer technology: A comparison of two theoretical models. Management Science. 1989;35:982–1003. [Google Scholar]
- 9.Fenech T. Using perceived ease of use and perceived usefulness to predict acceptance of the World Wide Web. Computer Networks and ISDN System. 1998;30:629–630. [Google Scholar]
- 10.Hendrickson AR, Masssey PD, and Cronan, T P: On the test-retest reliability of perceived usefulness and perceived ease of use scales. MIS Quarterly, June, 1993:227 –230.
- 11.Segars AH and Grover V: Re-examining perceived ease of use and usefulness: A confirmatory factor analysis. MIS Quarterly, December, 1993:517–525.
- 12.Subramanian GH. A replication of perceived usefulness and perceived ease of use measurement. Decision Sciences. 1994;25:863–874. [Google Scholar]
- 13.Chang P, Wu SC, Tzeng YM, Sang YY. The 3+1 snow-crystal-like information representation approach for the PDA-based application. The Journal of China Association for Medical informatics. 2002;15:17–32. (in Chinese) [Google Scholar]
- 14.Chang P, Tzeng YM, and Sang YY: The development of wireless PDA support systems for the compreh ensive and intelligent triage of the emergency nursing. The Taiwan Journal of Nursing Aug. 2003. (accepted, in Chinese)