Using Adaptive Turnaround Documents to Electronically Acquire Structured Data in Clinical Settings

Abstract

We developed adaptive turnaround documents (ATDs) to address longstanding challenges inherent in acquiring structured data at the point of care. These computer-generated paper forms both request and receive patient tailored information specifically for electronic storage. In our pilot, we evaluated the usability, accuracy, and user acceptance of an ATD designed to enrich a pediatric preventative care decision support system. The system had an overall digit recognition rate of 98.6% (95% CI: 98.3 to 98.9) and a marksense accuracy of 99.2% (95% CI: 99.1 to 99.3). More importantly, the system reliably extracted all data from 56.6% (95% CI: 53.3 to 59.9) of our pilot forms without the need for a verification step. These results translate to a minimal workflow burden to end users. This suggests that ATDs can serve as an inexpensive, workflow-sensitive means of structured data acquisition in the clinical setting.

Introduction

Data within an electronic medical record (EMR) is most useful when stored in a structured format. We focus significant developmental resources within our own health care network to develop interfaces which efficiently capture and structure such information [1]. We have had some success in our efforts [2, 3], but are continually evaluating new ways to address the longstanding challenge of structured data entry.

Additional obstacles exist when attempting to directly acquire structured data from patients and support staff prior to clinical encounters. The body of research that focuses on non-practitioner data acquisition highlights some of these particular challenges. Not only are most computer-based systems expensive and difficult to maintain, but their relatively fixed location impedes workflow [4, 5]. More importantly, the “digital divide” that exists among novice computer users often hinders this form of direct data entry [6].

We attempt to address these challenges by utilizing computer-interpreted paper forms. Paper continues to be widely used because it is an excellent transactional medium. It’s familiar, easy to work with, fully enabled, portable, and cheap [7]. Our previous research [8] has demonstrated the potential utility optical character recognition (OCR) technology would have when extended to the patient population. Advances in both computer hardware and OCR software packages allow us to utilize the many positive transactional features of paper while ultimately storing handwritten responses in a structured electronic format. In this way, paper is used as a turnaround document, which has the sole purpose of capturing information for electronic storage.

The paper form that we have designed has grown into a truly adaptive interface with computer systems. Patient specific content and prompts can be dynamically generated by a rules engine and printed on paper forms in real-time. Hand entered responses can then be automatically read back into the computer system through forms-based OCR software and referenced back to the patient’s EMR.

As part of our pilot evaluation of this technology, we examined how these adaptive turnaround documents (ATDs) would function in the real-world clinical setting of a high volume outpatient pediatric clinic. In particular, we wanted to measure the utilization rates of the forms, the accuracy of the recognition software, and the extra work requirements for the end users.

Background

The Child Health Improvement through Computer Automation (CHICA) system is designed to provide preventative care decision support and easy access to pertinent clinical data in outpatient clinical settings [9]. This paper-based system ultimately provides pediatricians with a highly tailored encounter form which provides reminders based on the patient’s specific health status. This same form also serves as the documentation of the visit. We designed this system to generate an ATD for both the parents and support staff upon a child’s arrival to the clinic, so that we could deliver high quality, real-time data to the system for improved decision support.

Methods

We obtained approval for the study from the Indiana University Institutional Review Board which also serves as the IRB for Wishard Hospital and the Indiana University Medical Group (IUMG) clinics.

Form Design

Sensitive to the work flow demands of a busy pediatric practice, the pilot form design was based mostly on the input of its end users (figure 1). The document is divided into two main sections. The top section is dedicated to the nursing and support staff. There is a section for patient identification, and the remainder of this area consists of fields for numeric value entry. Each potential measurement has implied units and a series of large boxes to capture each digit of the recorded value. There is also an optional “required” flag for each field that alerts staff when particular measurements are necessary to either address routine well care requirements at a given age or to readdress previously abnormal values.

Figure 1.

A completed ATD. This pre-screening form was developed to better inform our pediatric decision support system at the point of care. The top section is specific to nurses and support staff. Vital signs and other measurements are recorded in their associated field on the form. The bottom section is directed toward the particular patient. Up to twenty questions tailored to the patient’s medical history are answered by filling in the corresponding bubble located to the left of each question.

The bottom section consists of a questionnaire for direct patient entry. The CHICA database contains a large set of patient questions which are represented as rules in Arden Syntax. This allows us to assign priority scoring and conditional logic to narrow the scope of potential questions that are asked. As a result, the parent or adolescent is given the twenty most clinically relevant questions upon arrival to the clinic. They are asked to denote their answers by filling in the corresponding “yes” or “no” answer bubble with a pen or pencil.

The form generation software encodes form- specific identifiers along the bottom of the page. These barcodes are used to link the particular ATD back to the corresponding patient database record once it’s scanned.

We created a template of this design using the “Designer” application within the Teleform software suite by Cardiff Software (Vista, California, http://www.cardiff.com)..

Data Collection

We conducted the study in the IUMG Pediatric Clinics, located in downtown Indianapolis. The pilot ATDs were automatically generated upon patient check-in once CHICA received an HL7 arrival trigger message. They were printed to a Hewlett Packard 4100mfp (http://www.hp.com) multifunction Printer / Digital Sender / Fax machine located within the clinic. This particular unit is convenient for our clinical setting as it serves as both the origin and destination of the ATDs. The Digital Sender module of the device was set to send 300 dots-per-inch Tagged Image Format files (TIFFs) to a central OCR server through our secure hospital network for processing and optical character recognition.

This server ran the Windows XP operating system, and contained copies of the Teleforms application and the Outlook 2002 mail client. The server receives email from the Digital Sender and an interface included within Teleforms automatically retrieves the TIFF from Outlook’s mailbox for processing. The OCR software is designed to read both computer-printed and handwritten entries contained within recognition zones that are defined through creation of form specific templates.

Data Sources

ATDs printed by the CHICA system over six days were completed by eight nurses and seven ancillary staff members. The support staff was informed of the system trial and received a thirty minute training session to explain the pilot study. They were instructed to only fill the pertinent fields for a given patient. If parents did not complete their section of the ATD, the support staff wrote a reason for incompletion somewhere on the form for later review. The primary author was available by phone call and through daily visits to the clinic to answer questions that arose during the trial and to note provider comments about the system.

Data Evaluation

To measure utilization rates, we tallied the total number of scans received by the server. We compared this number with our internal counts of printed forms during the available “uptime” of the CHICA system. We also separately tallied the number of completed sections by either the support staff or the patient and recorded these results. Prior to the trial we expected a significant percentage of Hispanic patients to be seen in the clinic. At the time of the pilot study, the multilingual version of the questions had not been written, so the nurses were instructed to denote “Spanish Speaking” on the form so that these could be evaluated separately.

To evaluate the software’s accuracy, we tallied the total counts of handwritten digits and potential parent questions on all submitted forms. Each form was then manually reviewed in the “Verifier” application by one of the authors (PB) to establish a gold standard for correct readings. All forms were then processed three times by the “Reader” OCR application to ensure consistency of results. Each computer interpretation of a handwritten value was counted as to whether it was correct, and whether it required human review. Separate tallies of character recognition (OCR) and parental mark recognition (OMR) were maintained.

Results

Utilization Rates

The system printed 224 forms over the six day trial. The OCR server contained a total of 221 unique ATD scans, which corresponds with a 98.7% form completion rate. Out of these 221 forms, 100% had handwritten vitals recorded. Approximately 44 (19.9% of total) forms were given to Hispanic patients and were intentionally left blank. Of the remaining 177 forms, 176 (99.4% of total) were completed by the patient or their parents.

OCR Accuracy

The 221 forms in this study contained a total of 1387 unique handwritten digits. The vast majority of these were correctly recognized by the software (98.6%, 95% confidence interval (CI): 98.3 to 98.9). The software was also reliable: all 1309 digits that were not identified to require verification were truly correct. Twenty recognition errors were recorded among the sample set (figure 2).

Figure 2.

Two examples of handwritten entries incorrectly evaluated by the Teleforms package. Note in the bottom example, an attempt to write two digits into one box.

OMR Accuracy

The 176 completed parent sections contained a total possible 3,131 questions for completion. Of these, 99.2% (95% CI: 99.1 to 99.3) were correctly evaluated. 174 of these answers were flagged for verification, and this number spanned over 51 forms (28.9% of total).

125 of the 221 ATDs (56.6% of total) required no verification step and their contents were directly input back into the patient’s EMR following computer interpretation. All of these results were consistent throughout the three sets of data created by the OCR software.

Nursing staff provided a good deal of feedback throughout the time of the pilot study. They noted that the system was easy to use, primarily due to their familiarity with scanning from previous research efforts [8]. It also was minimally invasive to their workflow. Both nurses and physicians had many suggestions for new features or new question content for the database.

Discussion

Compared to our previous pilot work, we were very pleased with the overall recognition rates. The CHICA system is designed to provide a provider encounter form only upon receipt of either pre-screening ATD data, or a manual request within the system’s user interface. Therefore, the support staff will verify and correct these forms as they are scanned. We feared this step had the potential to be error prone and clumsy for end users, given the higher verification rates required in the previous pilot study. But the computer correctly identified 57% of the forms as requiring no verification. The remaining 43% that required verification had an average of 1.7 fields to correct. Given the current process, verification takes on average between 5-10 seconds to complete, which we believe will be quite tolerable. On a day-to-day basis, this translates to a minimal added work burden. We are increasingly confident that this data entry method will be well accepted among its end users, especially given the fact that they already function in a similar fashion.

We were pleasantly surprised by the high accuracy of digit recognition when compared to our previous OCR evaluation. We counted 78 digits (5.6% of total) which required verification, 20 of which required correction.

Given this robustness of OCR, we expected the OMR to perform better. In theory, OMR should be extremely accurate. There are four potential answers for each question as our form was designed: yes, no, both, or none. Any mark within either of these bubbles should trigger its corresponding answer. However, the software, in its default setting attempts to measure minimal percentages that a given bubble is filled. As a result, certain patients fooled the software due to the way they chose to fill their bubbles. If some were completely filled, and some were checked, then the software would ask to verify those that were incompletely filled, adding user time to the verification process. We suspect that we will see significant improvements in the accuracy and decreases in the number of values requiring verification as we become more familiar with how the OMR recognition engine is properly configured.

We were also pleased to see how compliant patients were when asked to complete these forms. We feel this is due in large part to the nature of this particular patient population. These children were all scheduled for health maintenance exams and were almost always accompanied by their parents who have a vested interest in ensuring their child’s welfare.

Probably the greatest testament to the form’s success came from anecdotal physician comments. Multiple practitioners have approached us during the pilot to tell us that they have learned about guns within homes and abusive family situations, as a direct result of the patient questionnaire.

As we continue to complete the pieces of the CHICA system, the data acquired through this form will deliver a great deal of value to the practitioner. Vital signs and other growth parameters will be evaluated by a computer real-time to both check for abnormal values and provide physicians with new measurements, such as a body mass index and growth percentiles. The current measurements will also be compared with the patient’s historical values to identify abrupt growth trend changes.

There are a series of preventative care algorithms that are driven in large part by patient-reported risk factors. When given access to the EMR and answers to these risk factor questions, our decision support system will provide physicians with a patient-specific prioritized list of discussion topics and actionable messages before the patient visit begins.

The successes of our pilot evaluation have made us thoughtful of the generalizability of this technology. ATDs have potential uses in dozens of clinical scenarios. For a relatively small investment, practitioners, support staff, and patients themselves have the ability to enrich an EMR with structured clinical data.

Figure 3.

Two examples of incorrect OMR evaluations. In the example to the left, the software assumed that the left bubble was the intended answer. In the right example, the software did not register any response, as this particular parent circled all twenty bubbles.