Abstract
The transition from paper to electronic documentation systems in acute care settings is often gradual and characterized by a period in which paper and electronic processes coexist. Intermediate technologies are needed to “bridge” the gap between paper and electronic systems as a means to improve work flow efficiency through data acquisition at the point of care in structured formats to inform decision support and facilitate reuse. The purpose of this paper is to report on the findings of a study conducted on three acute care units at Brigham and Women’s Hospital and Massachusetts General Hospital in Boston, MA to evaluate the feasibility of digital pen and paper technology as a means to capture vital sign data in the context of acute care workflows and to make data available in a flow sheet in the electronic medical record.
Keywords: Digital pen and paper technology, electronic medical record, vital sign data, acute care, nurses
Background and Significance
The benefits of the electronic medical record (EMR) with regard to patient safety are well established and have made the process of computerization a principal focus in healthcare settings (1–7). While a fully functional EMR is often the goal of automation, financial, staff and workflow constraints typically preclude simultaneous implementation of a complete system. Therefore decisions are made regarding prioritization. Both Brigham and Women’s Hospital (BWH) and Massachusetts General Hospital (MGH) have a robust outpatient EMR, the Longitudinal Medical Record (LMR). However, the inpatient record is paper-based. Both sites have electronic physician order-entry, enterprise-wide data, drug and allergy repositories and discharge applications. In addition, BWH has an electronic medication administration record in place (EMAR). All other clinical documentation at both sites is recorded in a paper record. During a recent scoping and prioritization of inpatient electronic documentation at these sites, multidisciplinary teams rated flow sheet data acquisition as the highest priority for electronic conversion across care enhancement, compliance, efficiency, and quality perspectives. Physician team members reported that existing flow sheets were illegible, often unavailable during rounds and that inadequate access to these data interfered with their ability to fully evaluate patient status and write orders. Team members identified transition to monitors and devices to automatically capture flow sheet data as desirable, but acknowledged that the substantial financial and time investments necessary to acquire these devices precluded immediate acquisition and adoption. Manual entry of vital signs and other flow sheet data was proposed as a possible interim solution. However, barriers exist to manual entry including lack of access to hardware within vital sign assessment and documentation workflow patterns and variable competence of paraprofessional staff (whom are often responsible for documenting vital sign data) with electronic applications. In addition, manual data entry represents an increased potential for errors as additional steps are added to an already labor-intensive process. Overcoming these barriers would require considerable investment in training, equipment and time.
The feasibility of the digital pen and paper technology was proposed as a bridge solution (e.g. could be implemented within 6–12 months to capture vital sign data typically recorded on bedside flow sheets and to make those data available on electronic flow sheets in the EMR until automated monitoring devices were in secured. A paucity of published research on the use of digital pen and paper technology in health care settings exists. We were unable to locate any published research where digital pen technology was employed for vital sign data acquisition purposes in acute care settings. Despont-Gros et. al found that the digital pen is easy to integrate in a component-based architecture based on web services (REF). Lind and Karlsson used the digital pen to capture patient reported data and found the technology to be accepted by patients for reporting symptoms from home to providers in palliative homecare settings (8). Yen and Gorman pilot tested usability of the digital pen in a labor and delivery acute care setting. Nurses were randomized to document using digital pen or conventional pen for clinical documentation. Data were not captured for viewing or reuse in the EMR, but solely to test usability of the digital pen. They found that while nurses’ attitudes towards the digital pen and its potential was positive, the design of the digital pen (e.g. larger and heavier than ballpoint pen) had poor usability and interfered with clinical workflow processes (9).
Research Methods
Our research question was, “Is the digital pen and paper technology a feasible short-term or “bridge” solution for vital sign data acquisition on acute inpatient units?” The goals for the study were as follows:
- Test the following hypotheses:
- The digital pen and paper technology will provide a feasible and workflow-friendly, bridge solution for capturing vital sign data in acute care settings.
- The technology will be associated with improved satisfaction with the vital sign data collection and reviewing process.
- Usability of the digital pen (as described by Yen and Gorman) will be improved through pre-implementation workflow process evaluation and mapping (e.g. preemptive measures taken to improve usability based on evaluation will overcome poor usability features of digital pen).
To evaluate feasibility of the digital pen and paper technology from multiple perspectives including reliability as a data acquisition method, staff acceptance and cost effectiveness.
The digital pen and paper technology requires that a unique dot pattern is printed on the form background to facilitate stroke capture by the miniature camera that sits inside the digital pen. After data is recorded on the paper flow sheet, the pen is docked and data are uploaded in a structured format to the digital pen server, processed and then sent to the Partners data repository. A user interface was developed within the inpatient EMR to facilitate access to vital sign data when viewing patient results.
Interdisciplinary focus groups were conducted at each site to evaluate current-state vital sign data collection and evaluation processes and observation was employed for validation. Process maps were generated using Visio and feedback was elicited. The data gathered from the pre-implementation workflow analysis was employed to assist the research team with closely mimicking workflow and taking measures to minimize poor usability features identified in the work of Yen and Gorman (9). For example, lanyards were attached to the pen so caregivers could wear the pen while taking vitals, minimizing the risk that the pen would fall out of a pocket or be dropped. Docking stations were placed strategically on workstations out on the unit, so that pens were accessible and could be easily docked and retrieved. User feedback was employed to identify the best location for the electronic flow sheet in the EMR so vital signs could be accessed in the context of other work routinely done in the EMR. A vital sign tab was placed in the Partners web shell under “results”, so that the electronic flow sheet was easily accessible to all providers and the location was consistent across sites.
The paper flow sheets in use at both sites were large, tri-fold flow forms designed to capture all vital sign and assessment data recorded by nursing staff over a 24-hour period. Vital sign flow sheets were designed for use during the pilot period and included a sub-set of the tri-fold assessment data. Based on user feedback, the following data elements (plus attributes) were included at both sites: temperature, blood pressure, pulse, respirations, oxygen flow method, oxygen saturation, and weight. In addition, the BWH flow sheet included finger stick glucose.
Implementation
Pre-implementation education was provided at both sites on all shifts for nursing staff (nurses, nursing assistants, unit secretaries) and via electronic mail for physician staff. While the digital pen is easy to use, nursing staff was trained to print forms and troubleshoot the system. Where possible, the digital pen was integrated into the existing workflow. The pre-implementation workflow was that the forms were collated and placed in the paper record on the evening shift. Consistent with the pre-established routine, digital vital sign flow sheet forms were printed each evening and put into the paper record. To facilitate the process, a webpage was built that used services to pull patient census for each patient care unit. Secretarial staff were able to batch print flow sheets for each patient on evening shift at the same time that they typically collated new forms and placed them in the paper record. Individual forms could also be printed for patients admitted or transferring to a study unit. The system automatically pulled patient data onto the flow sheet so that once the forms were printed, they could be placed directly into the record in front of the larger tri-fold. The health information management committees from both hospitals approved the forms for use during the study period as a subset of the larger 24-hour flow sheet. Nurses and physicians were instructed on how to access electronic flow sheet in the EMR and were told that because we were testing reliability of data acquisition, the paper flow sheet remained the “official” medical record and treatment decisions were to be based on the paper record.
Study Design
A prospective interventional study was conducted involving consecutive patients on two inpatient surgical units at BWH and one inpatient medical unit at MGH for a total of six weeks at each site. A random generator was used to select a 25% sample of all vital sign instances (e.g. occasions on which one or more vital sign values were recorded on a particular patient) to review data acquisition. Two members of the research team collected data and entered into an Access database, classifying data into the following categories:
Accurate
Data missing: No value recorded on paper flow sheet
Data missing: Handwriting recognition
Data missing: Data recorded out of box
Data missing: Value recorded is out of range (we programmed the system to accept only interpreted values considered possible in “live” humans on acute, non-intensive care units).
Data missing: Unable to determine
Data missing: <10% confidence in HWR accuracy
Data inaccurate: Hand Writing Recognition (HWR)
Data inaccurate: overwritten (value recorded and then changed)
Other
A third member of the research team checked a random sample of 5% of categorized data. Inter-rater reliability analysis revealed Cronbach’s alpha of .96 and intraclass reliability of .91. The majorities of discrepancies were related to data coded as “Data missing: unable to determine” and “other”. Rater three rechecked all data from these categories. In all cases, data were reclassified into one of the 7 remaining categories. In addition, data classified in the two “data inaccurate” categories were found to be overlapping. Rater one rechecked modifications and consensus was reached for each value. Data classification categories were then collapsed into 7 categories as follows:
Accurate
Data missing: No value recorded on paper flow sheet
Data missing: Handwriting recognition
Data missing: Data recorded out of box
Data missing: Value recorded is out of range
Data inaccurate: HWR
Data missing: <10% confidence in HWR accuracy
Web Survey
A web-based survey was developed to assess interdisciplinary satisfaction with the vital sign data collection and evaluation process. The survey included 20 questions (plus demographics) developed based on pre-implementation focus groups and observation. The questions were on a seven-point Likert scale with one representing “strongly agree”, six representing “strongly disagree” and seven representing “not applicable”. A link was sent out to all nursing and resident physician staff before implementation to measure baseline satisfaction and during the final week of the study to measure satisfaction associated with the digital pen intervention at each site. The Wilcoxson Signed-Rank Test was employed to test for median differences in satisfaction from pre-implementation to post-implementation period.
Results
Reliability of data acquisition
The digital pen and paper technology was conducted at BWH from December 5, 2005 through January 13, 2006 and at MGH from January 23 through March 3, 2006. There were a total of 3,596 vital sign data entry instances at both sites. Reliability of data acquisition was tested on a random sample of 25% of vital sign data entry instances. A total of 899 vital sign entry instances were reviewed (4,892 separate vital sign data values), classified and analyzed. Overall, data acquisition was 91.9% accurate. The 8.1% error was broadly related to inaccurate handwriting recognition (3.5%) resulting in incorrect values in the EMR and missing data (4.6%) resulting in incomplete data in the EMR. Missing data were classified as follows: system unable to recognize handwriting (2.3%), data recorded out of the box on paper flow sheet (.7%), data recorded was out of range (.2%), system was <10% confident in HWR value (1.4%). Data acquisition was significantly more accurate at MGH (95.2%) vs. BWH (89.3%); (Mann-Whitney test; Z=−9.198; p.000).
Web Survey
Eighty-one (81) respondents answered the web-based survey (43 pre-digital pen; 38 post-digital pen); 38.3% from BWH; 61.7% from MGH. The majority of respondents were female (90.1%), age 21–30 (53.1%), nurses (91.4%), working in direct care positions (95%) and on day shift with rotation to night shift (35.8%). The Wilcoxson Signed-Rank test was performed on each question for all participants and median scores were compared for pre-post implementation periods across both sites and at each hospital site. While trends in median improvement in overall satisfaction were noted across sites (median score 39.1 pre-digital pen/43.1 post-digital pen), the trend is not significant (p=.435). Significant median improvement scores were noted in responses to the following questions:
| Median pre/post | P value | |
|---|---|---|
| The current location of vital sign flow sheets makes it easy to check vital signs before I give medications. | 34.9/44.9 | .041 |
| Flow sheet data can be easily accessed when needed for patient care. | 35.4/46.4 | .027 |
| Flow sheets are available when I need to record data. | 31.2/50 | .000 |
| Patient weight values are available when I need them. | 31.7/40.6 | .001 |
Cost
Cost per year is based on estimates from the digital pen vendor (Digital Pen Systems) and includes the following:
License fee (one-time set-up fee per site): $40,000.
Yearly license fee per pen: $1200. Includes support, maintenance, upgrades, and application design.
Vendor professional engineering services (applies only for sites with less than 50 pens in use; first year only): $20,000 per month to offset vendor development cost to layout the forms, configure the system, test and install.
Printer: $4,000
Total cost per unit (based on 6 pens per unit):
First year/first unit: $51,200 (assuming professional engineering services not necessary because forms are developed and tested;
Each additional unit: $11,200 (6 pens/1 printer).
Each additional year: $7,200 per unit.
The number of pens needed per unit is dependent on the patient care model as well as the number of patients per unit. At BWH, nursing assistants take the routine vitals signs and RNs take vital signs less frequently. Therefore three pens were adequate for each 24-bed unit (one for each nursing assistant and one extra for nurses). At MGH, all nursing staff takes vital signs. Eight pens were needed for the 20-bed medical/telemetry unit.
Discussion
Our findings only partially support our hypotheses. Our first hypothesis was that digital pen and paper technology is a feasible bridge solution for capturing vital sign data in acute care settings. Our findings did not support this hypothesis. While data acquisition accuracy improved over the course of the study (due to a combination of improved system performance and improved handwriting by clinical staff) our results suggest that data do need to be verified before they are transferred to the repository. Because BWH has an EMAR in place, structured, coded vital sign data would provide value in terms of triggering decision support during the medication administration process, but the higher rate of inaccuracy at that site would require annotations that would be burdensome for nurses on busy acute care units. At MGH, the majority of the clinical documentation is in the paper record. Therefore verifying vital signs in the EMR would likely be an unreasonable disruption to the patient care workflow. In addition, during this pilot study, the AMCs began moving towards a more aggressive timeline to implement a full inpatient electronic medical record. The time required to make modifications exceeded our original definition of “bridge solution” . The plan to implement automated monitoring systems within the next few years at the AMCs meant that by the time limitations were addressed, the “bridge solution” would no longer be needed.
Some early implementation issues may have had an impact on data acquisition accuracy at BWH. The feasibility study went live first at BWH in early December 2005. The first two and one-half weeks of the 6-week study were characterized by multiple hardware and software difficulties that may account for the significant differences in reliability of data acquisition between the two sites. During the first two weeks of the study, the printer installed for printing the digital forms was defective and frequent jamming occurred, causing delays in printing forms for new patients. The printer was replaced during week two. In addition, the digital pen software had some programming errors that resulted in system down time and interfered with printing. Restarting the server corrected the problem, but caused workflow disruptions for nursing and secretarial staff who were required to contact the research team to correct the problem. System downtime also caused problems with data uploads because data appears in the electronic flow sheet based on the time a pen is docked, not the time data are recorded. The decision was made to use docking time, rather than the time the data was recorded because when the HWR of recorded times on the flowsheet were inaccurately interpreted (e.g. HWR was not a legitimate time) errors were generated in the electronic medical record. Therefore the time the pen was docked was used instead. This implementation choice became a limitation of our study. If the system was down on night shift, a pen used on the night shift could potentially be docked and uploaded after the data from the day shift. Moreover, in some cases there were problems noted with the way in which data uploaded to electronic flow sheets causing data that were correctly interpreted by the digital pen system to appear inaccurate in the electronic flow sheet. For example, in some cases, an entire vital sign entry instance would upload multiple times. In other cases individual values would upload twice into the system. This caused subsequent single vital sign entry instances to appear in multiple columns. An additional problem noted occurred with infrequent pen docking where multiple vital sign entry instances were uploaded simultaneously (e.g. vitals taken at 3 AM, 4 AM and 6 AM; pen docked once at 6:05 AM). Data uploading from the digital pen server to the Partners server would become shuffled and display inaccurately in the flow sheet. Values taken at one time would display in a column with vital sign data taken at a different time (see figure 1). The final three and one-half weeks of the study at BWH went smoothly, but the system was viewed as “unreliable” due to the number of problems encountered early on and the relatively short duration of the study.
During the second phase of the study at MGH, there were minimal issues noted with the digital pen software (two brief server downtimes). Data acquisition errors due to multiple uploads into the electronic flow sheet were resolved. Data shuffling with infrequent pen docking continued to be a minor issue at MGH, but was addressed by asking staff to dock the pens frequently. Nursing staff were educated regarding the types of data acquisition errors noted at BWH and initial training included demonstrating ways to avoid error (e.g. writing clearly, in the designated box, avoiding “script” writing and voiding errors and recording corrected values in a new column).
Our second and third hypotheses were that the digital pen and paper system would be associated with improved satisfaction with the vital sign data collection and reviewing process and that usability of the digital pen will be improved through pre-implementation workflow process evaluation and mapping. We did see significant improvement in satisfaction in several key areas. An important motivation for conducting this study was to improve vital sign data access and here we noted significant improvement in staff satisfaction. In addition, we found the digital pen to be workflow friendly. The pace and acuity level on inpatient units leaves little room for workflow disruptions and adoption of technology in this setting is challenging. However, over the course of the study, the pens were used on all shifts every day, even though all staff always had access to non-digital pens. We interpreted this as evidence that the pens were acceptable to caregivers and did not represent an unreasonable workflow burden.
Conclusion
Intermediate technologies have the potential to assist in the transition from paper to electronic documentation systems. The digital pen and paper technology is promising with regard to staff acceptance and integration into existing workflow processes. The ongoing license cost of the technology requires that the return on investment is carefully calculated and that workflow processes are evaluated to determine an adequate number of pens to support workflow without adding unnecessary costs. In addition, continued work is needed to maximize data acquisition accuracy to best support the transition from paper to electronic systems without introducing significant burden on nursing workflow.
Acknowledgments
The authors wish to thank the clinical staff at BWH and MGH for their involvement and support of this research study, Jason Thornton RN for assistance with data collection and Partners Information Systems Research Council for funding of this project.
References
- 1.Kuperman GJ, Teich JM, Gandhi TK, Bates DW. Patient safety and computerized medication ordering at Brigham and Women's Hospital. Jt Comm J Qual Improv. 2001;27(10):509–21. doi: 10.1016/s1070-3241(01)27045-x. [DOI] [PubMed] [Google Scholar]
- 2.Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc. 2001;8(4):299–308. doi: 10.1136/jamia.2001.0080299. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. Jama. 1998;280(15):1311–6. doi: 10.1001/jama.280.15.1311. [DOI] [PubMed] [Google Scholar]
- 4.Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc. 1999;6(4):313–21. doi: 10.1136/jamia.1999.00660313. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.King WJ, Paice N, Rangrej J, Forestell GJ, Swartz R. The effect of computerized physician order entry on medication errors and adverse drug events in pediatric inpatients. Pediatrics. 2003;112(3 Pt 1):506–9. doi: 10.1542/peds.112.3.506. [DOI] [PubMed] [Google Scholar]
- 6.Turner CL, Casbard AC, Murphy MF. Barcode technology: its role in increasing the safety of blood transfusion. Transfusion. 2003;43(9):1200–9. doi: 10.1046/j.1537-2995.2003.00428.x. [DOI] [PubMed] [Google Scholar]
- 7.Despont-Gros C, Bœuf C, Geissbuhler A, Lovis C. The Digital Pen and Paper Technology: Implementation and Use in an Existing Clinical Information System. Studies in Health Technology and Informatics. 2005;116:328–333. [PubMed] [Google Scholar]
- 8.Lind L, Karlsson D. A system for symptom assessment in advanced palliative home healthcare using digital pens. Medical Informatics & the Internet in Medicine. 2004;29(3–4):199–210. doi: 10.1080/14639230400005966. [DOI] [PubMed] [Google Scholar]
- 9.Yen P, Gorman P. Usability Testing of Digital Pen and Paper System in Nursing Documentation. AMIA 2005 Proceedings. [PMC free article] [PubMed] [Google Scholar]