Accuracy and Speed of Electronic Health Record Versus Paper-Based Ophthalmic Documentation Strategies

Abstract

PURPOSE:

To compare accuracy and speed of keyboard and mouse electronic health record (EHR) documentation strategies with those of a paper documentation strategy.

DESIGN:

Prospective cohort study.

METHODS:

Three documentation strategies were developed: (1) keyboard EHR, (2) mouse EHR, and (3) paper. Ophthalmology trainees recruited for the study were presented with 5 clinical cases and documented findings using each strategy. For each case-strategy pair, findings and documentation time were recorded. Accuracy of each strategy was calculated based on sensitivity (fraction of findings in actual case that were documented by subject) and positive ratio (fraction of findings identified by subject that were present in the actual case).

RESULTS:

Twenty subjects were enrolled. A total of 258 findings were identified in the 5 cases, resulting in 300 case-strategy pairs and 77 400 possible total findings documented. Sensitivity was 89.1% for the keyboard EHR, 87.2% for mouse EHR, and 88.6% for the paper strategy (no statistically significant differences). The positive ratio was 99.4% for the keyboard EHR, 98.9% for mouse EHR, and 99.9% for the paper strategy (P < .001 for mouse EHR vs paper; no significant differences between other pairs). Mean ± standard deviation documentation speed was significantly slower for the keyboard (2.4 ± 1.1 seconds/finding) and mouse (2.2 ± 0.7 seconds/finding) EHR compared with the paper strategy (2.0 ± 0.8 seconds/finding). Documentation speed of the mouse EHR strategy worsened with repetition.

CONCLUSIONS:

No documentation strategy was perfectly accurate in this study. Documentation speed for both EHR strategies was slower than with paper. Further studies involving total physician time requirements for ophthalmic EHRs are required.

Advances in information technology have provided opportunities for instant data access and communication. Through these technologies, the electronic health record (EHR) has become a critical strategy for medical care reform in the United States, with the goal of improving quality and safety of delivery, while decreasing costs.^1–4 These systems have potential to allow efficient storage, retrieval, and analysis of medical data that would be impossible with traditional paper-based records.⁵ To promote EHR adoption, the Health Information Technology for Economic and Clinical Health Act (HITECH) of 2009 offers financial incentives for hospitals and health providers who demonstrate so-called meaningful use of these systems in their practices.^6,7

However, EHR adoption in the American healthcare system has been limited, particularly in office-based practices.^8,9 Among ambulatory practices across all specialties in 2008, only 4% reported having an extensive EHR, whereas 13% reported having a basic system.¹⁰ Among ophthalmology practices, the EHR adoption rate has been similarly low. A survey of American Academy of Ophthalmology members in 2008 revealed that only 12% of practices had implemented an EHR system.¹¹ Potential barriers to EHR adoption in ophthalmology have been attributed to the complex medical and surgical components of the field, unique workflow, and documentation requirements with heavy emphasis on graphical representation of examination findings, high clinical volume, and heavy reliance on ancillary imaging devices with data transfer challenges.¹² A recent survey conducted by a major research firm among physicians across numerous specialties found that ophthalmologists had the lowest level of satisfaction with their EHR systems.¹³

Despite this ongoing shift toward EHR adoption, virtually all practicing ophthalmologists learned to document ophthalmic examination findings onto standardized paper-based templates. One key challenge is to translate these paper templates to computer-based EHR forms that are sufficiently accurate and efficient. An additional challenge is that most ophthalmologists are accustomed to documenting examination findings using hand-drawn diagrams,¹⁴ which may vary even among different ophthalmologists and often are difficult to implement with EHR systems. Most EHR documentation strategies rely on keyboard and mouse-based input methods, which inherently are more restrictive compared with hand-drawn sketches on paper. For example, mouse-based input typically requires selection of common prepopulated examination findings via menu widgets, whereas keyboard-based input provides textbox widgets for free text entry. Although these selection-based EHR methods potentially allow for standardized nomenclature for information exchange,^15,16 they do not permit spatial representation of data, which is traditionally important in the ophthalmic examination. These limitations of EHR input methods may affect the quality and speed of clinical documentation.

The influence of EHR data entry strategies on ophthalmic documentation is not widely documented. Better understanding of this relationship will permit future design of more efficient EHR data entry strategies by ophthalmologists.¹⁷ Overcoming the challenges of EHR data entry will result in longer-term benefits such as potential for data retrieval, data analysis (e.g., serial intraocular pressure, large-scale registries), and quality improvement. This pilot study was designed to evaluate the accuracy and speed of keyboard and mouse EHR documentation strategies, compared with a traditional paper strategy, for ophthalmic examination documentation. Ophthalmologists were presented standardized clinical cases, and the documentation process was analyzed using time-motion research methods.

METHODS

this prospective cohort study was approved by the Institutional Review Boards at Columbia University Medical Center and Oregon Health & Science University. Data collection and analysis were performed at these institutions.

SUBJECTS AND CLINICAL CASES:

Eligible study subjects were defined as ophthalmology residents or fellows and were recruited from the New York City area. The decision was made to restrict subjects to trainees because it was believed that this would produce a more homogeneous group for analysis, with less potential confounding from variability in age, experience, or computer skills compared with a group that included senior ophthalmologists. Written informed consent was obtained from all subjects before participation.

Five representative clinical cases were selected for this study from a publicly available website (http://webeye.ophth.uiowa.edu/eyeforum/cases.htm) that displayed grand rounds-type ophthalmology scenarios. The text of each case was read verbally and was audiotaped by one of the authors (P.J.T.) for standardized presentation to study participants. Finally, each case was reviewed in detail, and each discrete examination finding was identified by consensus of all authors for subsequent data analysis. For example, an examination significant for microcystic edema and guttatae in the cornea and 1+ anterior chamber cell was considered to have 4 distinct examination findings: edema, guttatae, 1+, and cell. Negative examination findings were included if the absence of features was described explicitly in the case, but were not considered if their absence was not presented explicitly in the case. The total number of examination findings was tabulated for each of the 5 cases.

DEVELOPMENT OF REPRESENTATIVE DOCUMENTATION STRATEGIES:

Three documentation strategies were developed for this study: (1) a keyboard EHR strategy (Figure 1), which provided a textbox-based widget for entry of all examination findings; (2) a mouse EHR strategy (Figure 2), which consisted of a series of prepopulated drop-down widgets containing common ophthalmologic findings for selection; and (3) a paper strategy (Figure 3), which allowed for hand-drawn sketches for diagramming the ophthalmic examination. To isolate the documentation strategies tested and to minimize variability (network lag, proprietary data storage formats, extraneous menus, etc.) found in commercial EHRs, the decision was made to create custom EHR strategies for this study.

FIGURE 1.

Representative interface for keyboard electronic health record (EHR) documentation strategy of the ophthalmic examination developed by authors for this study. Textboxes are used for keyboard input of ophthalmic examination findings. Tab key or mouse is used to advance to the next box. OD = right eye; OS = left eye; SLE = slit lamp examination.

FIGURE 2.

Representative interface of a mouse electronic health record (EHR) documentation strategy of the ophthalmic examination developed by authors for this study. Checkboxes (e.g., presence of guttae) and pull-down menus (e.g., severity from 1+ to 4+) with common ophthalmic examination findings are used for selection. Comment box is used for free text entry of additional findings. Scrollbar (right side) is used to advance the interface. OD = right eye; OS = left eye.

FIGURE 3.

Paper-based strategy for documentation of the ophthalmic examination used for this study. Preprinted templates allow for spatial arrangement of ophthalmic examination findings to be hand written or hand drawn. A/C = anterior chamber; ACIOL = AC intraocular lens; Conj = conjunctiva; Ext = external structures; OD = right eye; OS = left eye; PCIOL = posterior chamber intraocular lens.

The 2 EHR strategies were created using a software development kit (Visual Basic 2008; Microsoft, Redmond, Washington, USA) and were based on common data entry methods familiar to the authors from commercial ophthalmology EHR systems. The paper strategy was derived from a standard paper-based template used by several faculty providers in the Columbia University Department of Ophthalmology, which was not used routinely for clinical documentation by any of the subjects recruited for this study.

PRESENTATION OF CASES:

Each subject was given a 5-minute standardized tutorial on the use of each documentation strategy by the authors (P.C., P.J.T.). Additionally, subjects were asked to rate their comfort level with computers (novice, intermediate, advanced), whether they had used computers during their premedical training, and whether they were using EHRs in clinical practice. Each subject was presented with 5 clinical cases and was asked to document pertinent examination findings using each of the 3 strategies, resulting in 15 case-strategy pairs per subject. Each audiotaped case was presented immediately before documentation with the individual strategy and was repeated for each strategy. The subjects were instructed not to begin documenting until the presentation of the case had been completed. If requested by the subject, portions of the cases were repeated verbally for comprehension. To avoid any systematic bias from order of presenting the 3 cases and 5 strategies to each subject, all cases were presented sequentially along with each strategy in the same order for each subject. Each subject was presented with case 1 using keyboard EHR, case 2 using mouse EHR, case 3 using paper, case 4 using keyboard EHR, case 5 using mouse EHR, case 1 using paper, case 2 using keyboard EHR, and so forth. Documentation time was recorded using a digital timer controlled by the authors.

DATA ANALYSIS:

Accuracy of documentation was assessed based on sensitivity and positive ratio (PR) of documentation compared with what was truly present in the actual audiotaped case presented to subjects. Sensitivity for each case-strategy pair was calculated as the number of findings identified by the subject that were truly present in the actual case, divided by the total number of actual findings in the case. For example, an examination with 4 findings significant for 2 quadrants of dot blot hemorrhages with clinically significant macular edema in the left eye described by the subject as dot blot hemorrhages with clinically significant macular edema would have a sensitivity of 2 of 4 (50%). PR for each case-strategy pair was calculated as the number of findings identified by the subject that were truly present in the actual case, divided by the number of positive findings reported by the subject. For example, an examination case that was truly significant for 2+ nuclear sclerosis (2 true findings: nuclear sclerosis, 2+ modifier) in which the subject identified as 2+ nuclear sclerosis with cortical changes (3 reported findings: nuclear sclerosis, cortical changes, 2+ modifier) would have PR of 2 of 3 (67%).

Speed of documentation for each case-strategy pair was defined as the time required to document each examination finding and was calculated by dividing the documentation time of the entire case by the number of findings identified by the subject. For example, a documentation time of 3 seconds for 2 examination findings would result in a calculated speed of 1.5 seconds/finding.

Data were stored using an electronic spreadsheet program (Excel 2003; Microsoft). A 2-way mixed-effects analysis of variance was performed to characterize the variability in speed and accuracy of each interface. Because the 5 cases and 3 documentation strategies were all performed in the same order by each of the 20 subjects, evaluation of the learning effects within each strategy was performed using a univariate linear regression model. Analysis was performed using statistical software (SPSS software version 15.0; SPSS, Inc, Chicago, Illinois, USA). Statistical significance was considered when P < .05.

RESULTS

STUDY SUBJECTS AND CASES:

Twenty subjects from 5 ophthalmology training programs in New York City were recruited for this study. Among this group, 17 were residents (postgraduate years 2 through 4), whereas 3 were fellows (postgraduate years 5 or 6). Nineteen (95%) of the 20 subjects reported having taken premedical course-work that involved the use of computers, and all (100%) reported at least an intermediate proficiency with computers. All (100%) of the subjects had learned to document the ophthalmic examination using paper and diagrams and were using paper charting in their ophthalmology clinic. Fifteen (75%) reported that their hospitals were in the process of implementing an EHR for their ophthalmology clinics, but no subjects were currently using EHRs in their practices.

Each case averaged 51.6 findings (range, 48 to 56). There were a total of 258 ophthalmic examination findings identified in the 5 cases, with 300 case-strategy pairs, and 77 400 total possible findings documented by the subjects.

DOCUMENTATION ACCURACY:

Sensitivity for documentation of examination findings was 89% (range, 80% to 98%) for the keyboard EHR strategy, 87% (range, 80% to 94%) for the mouse EHR strategy, and 89% (range, 82% to 98%) for the paper strategy. There were no statistically significant differences in sensitivity among these 3 strategies. PR for documentation was 99.4% for the keyboard strategy, 98.9% for the mouse strategy, and 99.9% for the paper strategy (P < .001 between mouse EHR vs paper strategies, no statistically significant differences between keyboard EHR vs mouse EHR or keyboard EHR vs paper strategies).

DOCUMENTATION SPEED:

With cases analyzed individually, the paper strategy was faster than both keyboard EHR and mouse EHR strategies in 4 of the 5 (80%) cases (Table). With all 5 cases analyzed together, mean documentation speed 6 standard deviation was 2.4 ± 1.1 seconds for the keyboard strategy, 2.2 ± 0.7 seconds for the mouse EHR strategy, and 2.0 ± 0.8 seconds for the paper strategy. Documentation speed of the paper strategy was significantly greater than either of the EHR strategies (P < .001 for paper vs keyboard EHR, P < .001 for paper vs mouse EHR).

TABLE.

Speed of Ophthalmic Documentation in 5 Clinical Cases by 20 Ophthalmologists Using 3 Strategies: Keyboard-Based Electronic Health Record, Mouse-Based Electronic Health Record, and Paper-Based Writing

	Documentation Strategy
	Keyboard EHR	Mouse EHR	Paper
Mean documentation time (sec)
Case 1 (48 findings)	127.1	102.0	88.4
Case 2 (56 findings)	82.3	78.1	61.8
Case 3 (52 findings)	96.1	94.6	106.2
Case 4 (53 findings)	162.2	125.0	124.1
Case 5 (49 findings)	77.5	98.3	71.8
All cases	108.8	99.6	90.4
Documentation speed (sec/finding)
Case 1	3.1 ± 1.0	2.4 ± 0.5	2.2 ± 0.5
Case 2	1.6 ± 0.6	1.6 ± 0.5	1.2 ± 0.4
Case 3	2.1 ± 0.6	2.3 ± 0.7	2.4 ± 0.7
Case 4	3.5 ± 1.1	2.7 ± 0.8	2.6 ± 0.9
Case 5	1.7 ± 0.6	2.2 ± 0.6	1.5 ± 0.5
All casesa	2.4 ± 1.1	2.2 ± 0.7	2.0 ± 0.8

EHR = electronic health record.

^aFor documentation speed comparisons among all cases, P < .001 for keyboard EHR vs paper and mouse EHR vs paper EHR strategies.

LEARNING EFFECTS:

For each of the 3 documentation strategies, the 5 cases were always performed in the same order by each of the 20 subjects. A univariate linear regression analysis of the documentation time showed no statistically significant learning effects because the 5 cases were documented using the keyboard EHR (P = .58) and paper (P = .14) strategies. There was a statistically significant increase in time because the 5 cases were documented using the mouse EHR (P < .02), showing that subjects worsened in speed over time with that strategy (Figure 4).

FIGURE 4.

Graphs showing the learning effects of (Top) keyboard electronic health record (EHR), (Middle) mouse EHR, and (Bottom) paper-based strategies for ophthalmic examination documentation. There were no statistically significant learning effects with keyboard EHR (Top, P = .58) or paper (Bottom, P = .14). There was a statistically significant worsening in documentation time as more cases were documented using the mouse EHR strategy (Middle, P < .02).

DISCUSSION

this study evaluated the accuracy and speed of ehr documentation of the ophthalmic examination compared with a traditional paper documentation strategy. The study design aimed to simulate typical clinical scenarios in which ophthalmologists document clinical findings after performing the entire examination. Key findings from this study were that: (1) no documentation strategy was perfectly accurate, although the paper strategy was slightly more accurate than the mouse EHR strategy; (2) both EHR documentation strategies were significantly slower than the paper strategy; and (3) documentation speed of the mouse EHR strategy worsened with repetition.

None of the documentation strategies in this study were perfectly accurate. However, the paper strategy had a significantly greater PR than the mouse EHR strategy, meaning that more findings were documented by the mouse EHR strategy that were not truly present in the case. There are inherent differences between paper and EHR strategies that may explain these findings. In particular, menu-based interfaces seen in the mouse EHR strategy require the user to select examination findings from a prepopulated database through drop-down widgets and checkboxes. This layout may be more conducive to false-positive documentation, because the ophthalmologist may be forced to select examination findings that otherwise would not necessarily have been mentioned in a paper strategy. These mouse-based data entry strategies are very common in current EHR systems, and extraneous findings can be removed manually. However, overdocumentation in EHRs for this and other reasons raises concerns about quality of care as well as accuracy of billing.¹⁸ Furthermore, spatial representation of examination details that may be conveyed intuitively in a paper strategy must be detailed explicitly with EHR strategies. This can be seen in fundus diagrams where details must be described through text in a keyboard EHR. Finally, although not tested in this study, the copy-and-paste and copy-and-forward features of EHRs can contribute to false-positive (decreased PR) documentation if used indiscriminately.^19–21 Nonetheless, our findings suggest that EHR documentation strategies are comparable with regard to accurate representation of ophthalmic findings, compared with a paper strategy.

Overall, both EHR strategies in this study were slower than paper documentation. Several studies evaluating documentation time of EHR use in the ambulatory setting also have reported slower documentation speed after transition from paper.^22,23 In ophthalmology, documentation efficiency is critical, given the complex workflow and high patient volume.¹² Our experimental data show that documentation of the ophthalmic examination is significantly slower using EHR strategies. For instance, an extra 0.41 seconds/finding was required on the keyboard EHR compared with the paper strategy. This translates to an additional 21.16 seconds/case (with each case in this study averaging 51.6 total findings). In a clinic with 50 patients, this difference would amount to an additional 18 minutes solely for documentation of examination findings using a keyboard EHR strategy. Other studies involving EHR documentation time have had varied results. One analysis on nursing documentation showed no loss of time in EHR charting,²⁴ whereas another time-motion study of ophthalmologists showed that EHR documentation time was 6.8 minutes slower per patient than with paper (Chiang MF, written communication, December 8, 2012). Ideally, any losses in physician productivity from increased EHR documentation time should be offset by gains from improved communication, quality, and safety of care.^4,5 Similarly, improvements in productivity from EHRs should not come at the expense of decreased quality or safety.

There were no statistically significant learning improvements seen in any of the 3 study documentation strategies. Although we may expect that documentation time would improve as users become more familiar with the strategy, some published reports on productivity have shown that there is a lag time after EHR implementation.^25,26 Physician perception of this apparent lag time is similar. In a survey of ophthalmologists, Chiang and associates reported that 34% believed that productivity was increased after 6 months of implementation of an EHR, whereas 30% believed no change had taken place, 15% believed a decrease had taken place, and 21% were unsure.¹¹ Our findings with the keyboard EHR and paper strategies did not demonstrate any statistically significant patterns. In fact, documentation speed worsened significantly with use in the mouse EHR strategy (P = .003). Possible reasons for this finding include subject fatigue from repetitive testing in a study environment or subject fatigue from repeated use of the mouse point-and-click interface. The impact of fatigue has been demonstrated from electrophysiologic and neuropsychological perspectives ²⁷ Additional research to evaluate the efficiency of a mouse EHR may be warranted to validate this finding.

This study evaluated only keyboard and mouse as input methods for an ophthalmic EHR. Neither of these methods was found to be an ideal strategy. With the recent advent of voice recognition and touchpad interfaces in mobile computing, these input strategies may be suited for ophthalmology. In medicine, radiologists and pathologists long have implemented voice recognition in dictation of reports with some success.^28–32 Ophthalmologists currently implementing scribes for examination documentation may find voice recognition to be helpful if such technology could be incorporated into an EHR. Studies involving efficiency of touch screen interfaces have yielded mixed results,^33,34 although some commercial EHR platforms already have begun to implement this technology. We expect to see further research with these interfaces as these technologies mature.

There are several additional study limitations. First, examination findings were presented sequentially using EHR simulations created by the authors, followed by subject documentation. In practice, ophthalmic charting often is nonlinear and generally is performed with commercially available EHRs. A typical examination may involve documenting the anterior segment examination, reviewing diagnostic tests (e.g., visual field or tonometry), documenting the posterior examination, and subsequently viewing images (e.g., optical coherence tomography). That said, it may be difficult to separate contributing and conflicting effects from those additional real-world factors. We believe that this experimental study design more clearly isolates and permits analysis of effects caused by the EHR interfaces themselves. Second, it has been reported that learning the EHR is a factor that hindered ongoing EHR use.^35,36 Although the subjects reported experience with computer interfaces, they received only a 5-minute tutorial on the EHR strategies. Speed and accuracy may change with ongoing use of EHRs, and this warrants additional research. Third, many ophthalmic EHRs implement either stylus-based input methods or a mouse-based interface as a means to allow graphical annotation of the ophthalmic examination results. These functions were not explored in this study. However, it is also unclear whether hand-sketched graphical diagrams of the ophthalmic examination allow for greater objective representation of findings over descriptive annotations, especially with the availability of other imaging methods in most ophthalmic practices. Fourth, our study evaluated documentation strategies in isolation, which is unlikely to be seen in clinical practice. Most EHRs implement a combination of keyboard and mouse strategies for documentation; certain EHR widgets (e.g., textboxes, checkboxes) suitable for 1 examination finding (e.g., tonometry) may be inefficient for others (e.g., ocular motility). The intention of this study was to evaluate fundamental principles of documentation while minimizing variability (e.g., network lag, ancillary menus, etc.) found in commercial EHRs. We believe that our study system captured the essential features of a modern EHR and permitted analysis of basic documentation principles. Finally, this study involved only 5 clinical cases. Larger samples might have produced more precise estimates. However, we believed that additional testing time beyond the 1 hour/subject required for this study might have introduced error because of fatigue and might have decreased the number of subjects willing to volunteer.

This pilot study aimed to evaluate EHR documentation strategies for ophthalmic charting. Our findings suggest that although neither EHR documentation strategy was perfect, both offer similar accuracy and speed. Furthermore, although the EHR strategies are associated with slightly worse documentation time than the paper strategy, they are no less accurate. The true benefits of EHRs ultimately may lie more in retrieval and analysis of medical knowledge, rather than in entry and storage of data. Further studies in these EHR features are warranted to elucidate its role in the delivery and quality of patient care. These studies will have relevance for all practicing ophthalmologists.

Biographies

Patrick Chan is a cornea and refractive surgery fellow at the Cleveland Clinic Cole Eye Institute. He completed his medical education and ophthalmology residency at Columbia University College of Physicians and Surgeons. Prior to his medical career, he received an undergraduate degree in Computer Science from Duke University, and was a software developer and an intern at the National Aeronautics and Space Administration (NASA).

Dr Michael F. Chiang is Knowles Professor of Ophthalmology & Medical Informatics and Clinical Epidemiology at Oregon Health & Science University (OHSU), and is a Vice-Chair for Research in the Department of Ophthalmology at OHSU. His clinical practice focuses on pediatric ophthalmology. His research involves telemedicine, clinical information systems, and image analysis. His lab has been funded by the National Institutes of Health (NIH), and by several charitable foundations, since 2003. He received a BS in Electrical Engineering & Biology from Stanford, and an MD from Harvard Medical School and the Harvard-Massachusetts Institute of Technology Division of Health Sciences & Technology, and an MA in Biomedical Informatics from Columbia University where he was an NIH/NLM (National Library of Medicine) Fellow. He completed residency and pediatric ophthalmology fellowship training at the Johns Hopkins Wilmer Eye Institute. He is chair of the American Academy of Ophthalmology Medical Information Technology Committee, and serves on numerous other national committees. He is on the editorial boards for the Journal of the American Association for Pediatric Ophthalmology & Strabismus and the Journal of the American Medical Informatics Association (JAMIA), and is an Assistant Editor for JAMIA.