Impact of document consolidation on healthcare providers’ perceived workload and information reconciliation tasks: a mixed methods study

Abstract

Background

Information reconciliation is a common yet complex and often time-consuming task performed by healthcare providers. While electronic health record systems can receive “outside information” about a patient in electronic documents, rarely does the computer automate reconciling information about a patient across all documents.

Materials and Methods

Using a mixed methods design, we evaluated an information system designed to reconcile information across multiple electronic documents containing health records for a patient received from a health information exchange (HIE) network. Nine healthcare providers participated in scenario-based sessions in which they manually consolidated information across multiple documents. Accuracy of consolidation was measured along with the time spent completing 3 different reconciliation scenarios with and without support from the information system. Participants also attended an interview about their experience. Perceived workload was evaluated quantitatively using the NASA-TLX tool. Qualitative analysis focused on providers’ impression of the system and the challenges faced when reconciling information in practice.

Results

While 5 providers made mistakes when trying to manually reconcile information across multiple documents, no participants made a mistake when the system supported their work. Overall perceived workload decreased significantly for scenarios supported by the system (37.2% in referrals, 18.4% in medications, and 31.5% in problems scenarios, P < 0.001). Information reconciliation time was reduced significantly when the system supported provider tasks (58.8% in referrals, 38.1% in medications, and 65.1% in problem scenarios).

Conclusion

Automating retrieval and reconciliation of information across multiple electronic documents shows promise for reducing healthcare providers’ task complexity and workload.

BACKGROUND

Information reconciliation is a complex and often difficult task that many healthcare providers perform on a daily basis. The current state of healthcare dictates that providers’ responsibilities are woven seamlessly with information technology, particularly, clinical information systems that they use to perform tasks. Fatigue and mental workload are associated with complexity and difficulty of the work.¹

Each time a patient transitions from one care setting to another, clinicians need to review previous medications and medical history, followed by a reconciliation of information differences across disparate records gathered from various sources. This reconciliation process requires considerable time and effort, and sifting through multiple documents for a single patient introduces the potential for inaccuracies and oversights. Prior studies have demonstrated^2,3 that such problems, especially in medication duplication, lead to medical error, with a direct impact on patient care. In general, complex tasks and higher cognitive and physical workload result in lower accuracy and reduced efficacy in the accomplishment of tasks.¹

Automating information reconciliation has the potential to reduce cognitive and physical workload, key barriers identified in medication reconciliation processes.⁴ Hospitals and health systems increasingly participate in health information exchange (HIE), in which data and information are shared across organizational boundaries to facilitate improved access to information.^5,6 A common form of HIE involves the exchange of documents encoded using the Health Level Seven (HL7) Clinical Document Architecture (CDA) standard, including Continuity of Care Documents or CCDs.

CCD is one of the CDA templates that contains only patients’ critical and summary medical information. Templates are commonly used to constrain CDA for specific use cases. Since duplicative and conflicting implementation guides (IG) for templates were published by different standards organizations, HL7 introduced the Consolidated CDA (C-CDA), which is an IG specifying a library of templates. C-CDA became the single source of truth for implementation of CDA templates, including CCDs. Exchanging CCDs and incorporating problem, medication, and allergy sections from CCDs into electronic health record (EHR) systems is required by the “meaningful use” incentive program to improve overall clinical effectiveness.^7,8

Although increased information sharing via CCDs is an improvement over data islands, the technical architecture of many HIE infrastructures has created an information reconciliation challenge for end users. HIE infrastructures typically receive a CCD after a patient encounter. Over time, HIE infrastructures naturally accrue multiple CCDs per patient. When information sharing occurs, all of these documents are often shared with a downstream provider, rather than providing a unified summary across multiple CCDs. Therefore, providers who access an HIE or EHR interface must sift through multiple documents that include potentially duplicative or conflicting information. In Figure 1, a screenshot of the U.S. Department of Veteran’s Administration Virtual Lifetime Electronic Record (VLER) application, which uses an HIE infrastructure to gather CCDs from non-VA providers, shows a list of available documents for a single veteran. Without efficient and effective methods for de-duplicating data across documents, providers are forced to review lengthy and redundant information. Cumbersome review and reconciliation of information have been observed in prior studies on HIE,^9,10 establishing the need for better tools.

Figure 1.

List of available medical records for a single veteran at VLER system.

The challenge of reconciling clinical content in CCDs has attracted both local and national attention. The Indiana Network for Patient Care (INPC), created by investigators at the Regenstrief Institute and operated by the Indiana Health Information Exchange (IHIE), is the nation’s largest inter-organizational clinical data repository. IHIE now serves over 25 000 physicians and over 10 million patients.^11,12

Information reconciliation across multiple CCDs for a single patient was named by IHIE as a “grand challenge” in the 2013 Hoosier Healthcare Innovation Challenge (HHIC). HHIC is a state-based software development competition that brings together healthcare and technology professionals to provide creative solutions for some of the most challenging problems in healthcare.¹³

As a part of the HHIC competition, we developed a prototype system to consolidate and de-duplicate multiple CCDs for a single patient.¹⁴ We further refined this system and tested its real-world performance in CCD consolidation compared to manual review.¹⁵ We were encouraged by the preliminary results of the system’s performance. Yet, we also wanted to characterize how the system’s performance influenced the perceived workload of providers. Our novel system is able to consolidate multiple CDA-based documents into a single document that can be rendered for review by an end user. As such, our hypothesis is that this system has potential to significantly reduce providers’ perceived workload when conducting information reconciliation tasks.

The purpose of this study is to evaluate the impact of the CDA-consolidation system on healthcare providers’ perceived workload while reviewing and reconciling medical documents. We also sought to determine the system’s effect on time spent, as well as the accuracy of information reconciliation by providers.

METHODS

Study design

To evaluate the system’s impact on perceived workload, we employed a simulation-based usability assessment in a laboratory setting. Real-world users were asked to interact with the system to conduct 3 commonly performed scenarios in outpatient settings (referrals, medications, and problems reconciliation). Following each simulated work task, users completed a standardized assessment of their perceived workload. At the end of all the scenarios, users were interviewed about their experiences with the system. Figure 2 summarizes the overall study design.

Figure 2.

Overall structure of the study.

The study seeks to measure the participants’ perceived workload, accuracy, and efficiency of information reconciliation, while reviewing 2 types of medical records: 1) multiple CCDs for a patient vs. 2) 1 consolidated CCD generated by the CDA-consolidation system. Therefore, participants completed each scenario twice: 1) before consolidation and 2) after consolidation. No system training was provided to the participants. Also, participants were unaware that the CCD in the second task of each scenario was already consolidated and reconciled. Therefore, they reviewed the reconciled CCDs with the intention of finding duplications. Figure 3 is a screenshot of a sample CCD that is presented to the participants in a web browser.

Figure 3.

Screenshot of a sample CCD that is presented to the participants in a web browser.

Overall, each participant conducted 6 tasks across 3 scenarios. In each scenario, the first task was to review multiple CCDs for the patient, and the second task had only 1 consolidated CCD. All of the participants conducted a referrals scenario first, then medications reconciliation, and finally the problems reconciliation scenario.

The study was approved by the Institutional Review Board (IRB) at Indiana University (1503069512). As the scenarios were constructed using de-identified medical record data, the IRB granted the study a waiver of HIPAA authorization. Because the participants were medical professionals, the IRB also waived the requirement of informed consent to participate in the scenarios.

Evaluation measurement

To evaluate perceived workload, we used the National Aeronautics and Space Administration Task Load Index (NASA-TLX), which is the most widely used research measure to assess task workload and effectiveness in humans.¹⁶ The NASA-TLX includes an overall index of mental workload as well as the relative contributions of 6 subscales: mental, physical, and temporal task demands, and effort, frustration, and perceived performance.¹⁷

At the end of each task, the time for task completion was recorded and then participants completed the NASA-TLX. The NASA-TLX captures self-rating on a scale of 100 points for mental, physical, and temporal demands as well as effort, performance, and frustration associated with each task. In total, each participant completed the NASA-TLX 6 times.

An overall NASA-TLX score was defined based on the formula ${\displaystyle {\sum }_{i=1}^6(r_i\times w_i)/\left(\underset{2}{\overset{6}{}}\right)}$. In this formula, “r” represents the self-reported score for each of the 6 measures in the NASA-TLX, and “w” is the weight of the measure defined based on the 15 paired-wise selection of measures. An open-source HTML and JavaScript version of the NASA-TLX application was used in this study.¹⁸

After completing the tasks and collecting NASA-TLX results, the following open-ended questions were asked in a face-to-face interview. The open-ended questions were designed to capture participants’ experiences and perceptions beyond the ones recorded through NASA-TLX. Although the focus was consolidation of duplicate information, we were further interested in understanding additional challenges that participants face while reviewing medical records. Therefore, the questions were tailored to address these requirements. Our intention was to ask more general and open-ended questions to allow participants to freely think about the processes of medical record review during their daily practice and speculate the impact of our system in their work.

1. What were the challenges you faced in finding information from the patient’s medical records in general?

2. If you had a tool that could bring together patient information from medical documents into a single view, would this be helpful to you? If yes or no, why?

3. How would you use such a tool in your practice? Would you use it in advance of the encounter or during the visit?

The open-ended interviews at the conclusion of study took approximately 20 minutes for each participant. Each session was audio recorded, transcribed, and then reviewed by the corresponding author. The main themes of the interviews were extracted and categorized.

The accuracy of information reconciliation was evaluated by reviewing the consolidated documents. The corresponding author collected the documents that were reconciled by the participants after each task and manually counted the duplications missed by each participant in order to calculate the accuracy of the manual reconciliation performed by the participant.

The system

The system is fully described elsewhere;^14,15 here, we briefly summarize its purpose and operation. The system is designed to operate as a web service in which multiple CDA documents pertaining to the same patient can be merged and de-duplicated. The system uses a set of rules to examine 3 key sections of the CDA documents important for clinical decisions in outpatient settings: the problem list, allergies, and medications. The system examines all of the entries across each document to generate a consolidated, single CDA document free of duplicate entries. The system utilizes semantic standards as well as simple free-text string comparisons to eliminate duplicate entries based on date and time stamps associated with each entry. Our prior evaluations of the system demonstrated high levels of accuracy in consolidating both simulated and real-world CDA documents.

As consolidation of CCD documents was the focus of this work, only CCD documents were reconciled using the system and reviewed by the participants in the study. Nevertheless, the system is capable of consolidating any CDA templates if the template-specific rules are defined in the system.

Clinical scenarios

We designed 3 transitions of care scenarios in which we could examine cognitive tasks associated with the review of the document sections consolidated by the system: 1) referral, 2) medication reconciliation, and 3) problems reconciliation. The participants were asked to read the scenarios and conduct specific tasks. The following scenarios were provided to the participants:

Referral scenario: As a health provider, you are reviewing electronic medical records (CCDs) of a patient that you want to refer to a specialist. To refer the patient to a specialist, you need to identify all of the important allergies in the patient’s medical record that are necessary for the specialist to know. You may identify these allergies based on reviewing the patient’s medications and problems list. Your task is to reconcile all of the patient’s allergies and then write a note to the specialist identifying those identified allergies.

Medication reconciliation: Patient-X is a new patient who arrived at your clinic today. Your office has requested and received clinical records from Patient-X’s previous provider. These records are ready for review prior to the visit. Your task is to review the CCDs in the documents and reconcile the patient’s medications prior to the clinical encounter.

Problems reconciliation: A new patient is coming to your clinic for an initial visit. His/her medical record is complex with multiple conditions, which your office received from the prior physician(s). Your task is to reconcile the complex problem list as part of preparation for seeing the new patient.

Participants

A convenience sample of 5 primary care physicians and 4 nurse practitioners was recruited (9 total participants) from IU Health, a health system with more than 1500 board-certified or board-eligible physicians practicing at over 200 locations statewide. Participants were invited to participate via email using a contact list provided by Regenstrief Institute. The participants were contacted in person or via email and asked if they were interested in participating. Through the screening process, we assured that they had experience using EHR systems and regularly conducted information reconciliation before or during a patient visit.

CCD sampling

The CCDs used in this study were sampled from the INPC. To collect CCDs for each scenario, we queried the database for patients with a minimum of 5 problems, 15 medications, and 4 documented allergies.

The CCDs in the database did not contain a global patient identifier. In order to select the relevant CCDs for a particular patient, we identified and matched patients based on the first name, last name, and birth date fields of CCDs. As there was a chance for incorrect patient matching, the CCD identification of each patient was double checked manually. To assess the effect of our consolidation system, we then selected patients with 3 or more CCDs.

For each scenario, 2 patients were randomly selected from our sample after applying the following filter. We ran the sample of CCDs through the consolidation system and then after reconciliation, we chose 2 patients for each scenario who had the same number of problems, medications, and allergies. For instance, for the referral scenario, 2 patients were randomly selected with a total of 8 reconciled problems, 5 reconciled allergies, and 16 reconciled medications. These 2 patients had a different number of entries in each of the 3 sections before reconciliation; however, after reconciliation, the number of entries was the same.

Because each scenario included 2 tasks, we randomly selected 6 patients with the aforementioned criteria. In each scenario, the second task was always to review the consolidated CCD.

Statistical analysis

Based on prior research^19,20 and our team’s clinical experience, we used 8 as the minimum sample size required to identify at least 22% reduction of time (mean value) needed for medical information reconciliation. We set alpha at 0.05 and sought to achieve power at 0.80.

Results from our scenarios were tabulated into a spreadsheet, and we compared variables (time and NASA-TLX subscales) using a paired t test analysis. A P-value of less than 0.05 was considered statistically significant.

RESULTS

Accuracy of information reconciliation

The following is a summary of the accuracy of the information reconciliation, compared with the manual work by the study participants and the automated system.

• 100% accuracy in referrals and problems reconciliation: All of the participants correctly reconciled the allergies and problems in multiple CCDs. The consolidated CCDs were already reconciled and there was no need for de-duplication of information. However, since the participants were unaware of this factor, they still attempted to locate duplications in consolidated CCDs.

• Mistakes in manual medication reconciliation: In the medication reconciliation scenario, participants made a few mistakes while attempting to reconcile multiple CCDs. Two of the participants failed to add 1 medication in the list of reconciled medications at the end of the task. Also, 3 of the participants were not able to reconcile the medication “Lorazepam” with its common brand name “Ativan,” but rather included both names in the final medication list. The medication reconciliation scenario was conducted correctly for the consolidated version of CCDs without any mistake.

Perceived workload

Based on the results of the paired t test analysis, the overall perceived workload of reviewing consolidated CCDs is significantly lower than reviewing multiple CCDs. However, some of the NASA-TLX measures did not change significantly in each scenario. In Table 1, the underlined P-values greater than .05 indicate the non-significant measures in each scenario.

Table 1.

Paired t test analysis results for a multiple vs. consolidated CCD review. Numbers with asterisks are P-values that are considered as significant. The lower value for performance indicates the higher performance reported by the participant (MD=mental demands, PD=physical demands, TD=temporal demands, PE=performance, EF=effort, FR=frustration, OW= overall perceived workload). Statistically significant values (P-value < 0.05) are labeled with asterisks

	Referral Scenario			Med Rec Scenario			Problems Rec Scenario
	R1 Mean (SD)	R2 Mean (SD)	p	M1 Mean (SD)	M2 Mean (SD)	p	P1 Mean (SD)	P2 Mean (SD)	P
Time	131.2 (89.6) *	54.1 (36.4) *	0.008	297.2 (132) *	183.9 (107) *	0.000	145.2 (68.2) *	50.7 (24.0) *	0.000
MD	39.4 (25.1) *	24.4 (13.6) *	0.041	47.8 (26.4) *	38.3 (20.3)	0.196	38.3 (26.2) *	21.1 (17.6) *	0.005
PD	16.7 (10.6) *	12.2 (7.5) *	0.026	31.1 (25.7) *	22.8 (24.4) *	0.033	18.9 (15.8) *	13.3 (7.9) *	0.048
TD	31.1 (27.0) *	16.1 (10.8) *	0.020	53.9 (23.0) *	34.4 (19.8) *	0.017	35.6 (17.4) *	25.0 (15.8) *	0.005
PE	27.2 (16.8) *	20.6 (15.5)	0.052	26.1 (14.1) *	23.3 (15.0) *	0.048	21.1 (11.9) *	15.6 (7.3) *	0.037
EF	28.9 (22.7) *	22.8 (15.2)	0.128	52.8 (26.1) *	43.9 (24.1) *	0.032	31.7 (20.6) *	23.9 (16.9) *	0.038
FR	22.2 (26.5) *	12.8 (12.3)	0.179	33.9 (22.0) *	27.2 (20.2) *	0.044	22.8 (21.7) *	17.8 (14.8)	0.156
OW	32.4 (21.7) *	20.3 (10.2) *	0.032	46.6 (20.4) *	38.0 (17.1) *	0.021	31.0 (14.1) *	21.2 (11.4) *	0.004

Efficiency

Tables 1 and 2 summarize the results of statistical analysis and time reduction percentage for each scenario. Table 1 demonstrates the difference between before and after reconciliation tasks in each scenario based on NASA-TLX measures. For instance, in the Referral Scenario column, R1 represents a referral task before reconciliation (multiple CCDs) and R2 is after reconciliation (single CCD). For each of R1 and R2, mean and standard deviation of review time and NASA-TLX measures for all participants are calculated. The values with asterisks are statistically significant.

Table 2.

Observed reduction in time and perceived workload of clinicians after using a consolidated document to perform an information reconciliation task

Measure	Referral Scenario	Med Rec Scenario	Problem Rec Scenario
Time	58.8%	38.1%	65.1%
MD	38.0%	19.8%	44.9%
PD	26.7%	26.8%	29.4%
TD	48.2%	36.1%	29.7%
PE	24.5%	10.6%	26.3%
EF	21.2%	16.8%	24.6%
FR	42.5%	19.7%	22.0%
OW	37.2%	18.4%	31.5%

In Table 2, reduction in time and perceived workload of clinicians after using a consolidated document to perform an information reconciliation task is presented.

Provider interviews

The interview session transcriptions were reviewed, and the main topics that were addressed by participants were extracted and categorized. Table 3 summarizes the responses from all participants into main topics.

Table 3.

Main topics that are discussed during open-ended discussion

Questions	Topics
1. What were the challenges you faced in finding information from the patient’s medical records in general?	Reconciliation is a huge time commitment. Too much information (I need relevant information) Uncertainty (missing, inaccurate, or not complete data) Inconsistencies (conflicts in different documents) Finding information (where to look) Complex UI (I am not sure where the information is)
2. If you had a tool that could bring together patient information from medical documents into a single view, would this be helpful to you? If yes or no, why?	This system helps in finding different types of information easily. Less time Less searching Less clicking
3. How would you use such a tool in your practice? Would you use it in advance of the encounter or during the visit?	Before visit (5 people) During visit (2 people) - Data already consolidated and can be used during visit Before and during visit (2 people)
Final discussion	All of the participants agree that consolidation system is very useful. Each section of the medical record had a hyperlink in our system that allows the users to navigate to the specific section without traversing the whole document. All participants agreed that using the hyperlink is helpful in finding information and saves a lot of time. This is not an ultimate solution. I need the data in my EMR. Categorize into type of visit note (admission, discharge, office visit), active medications, categorization of problems (heart problems) Time stamped information (active medication)

DISCUSSION

Time is precious in medicine, and perceived workload is an important determinant in the adoption and use of health information systems. According to McDonald,²¹ time spent searching for information is a barrier to high-quality medicine. Despite many positive impacts on care delivery following implementation of EHR systems and the CDA standard, time spent looking for information and other administrative tasks continue to be a challenge for clinicians.^22–24 In this study, we evaluated the ability of a novel CCD consolidation system to reduce time and effort for clinicians when performing information reconciliation tasks associated with routine care.

While prior studies have examined methods for improving reconciling medication information,^25–30 there are no previous studies that examine the time or effort required to reconcile problem lists, allergies, or other sections of electronic messages. Moreover, none of the prior systems evaluated the impact on clinicians’ perceived workflow or time spent reconciling information.^25–30 In addition, we could not identify any prior research demonstrating the consolidation of duplicative information from CCDs, although some software products have attempted to address this challenge. Thus, our system and approach are novel, providing evidence that information reconciliation is needed and impactful on both perceived workload and time on task.

Perceived Workload Reduction: Participants further perceived a reduction in workload in all scenarios where the consolidated CCD was utilized. However, some dimensions of perceived effort did not differ significantly when the consolidated CCD was presented to the user. This was especially true for the referral scenario, in which performance, effort, and frustration measures were reduced, but these changes were not statistically significant. We suspect this was due to the lower number of CCDs (N = 3) used in this scenario. Additional participants might have been necessary to confirm a statistically significant change.

Time R eduction: Overall, our CDA-consolidation system reduced the time to review medical records by approximately 50% in all the scenarios. Medication reconciliation time was not reduced as much as the time reductions in referral and problems reconciliation scenarios (38.1% vs. 58.8% and 65.1%, respectively). We suspect this was because we did not inform participants that the document used in the second task was already reconciled. As there was a high number of medications (N = 17) to reconcile, participants searched for duplications in the consolidated CCD, even though there were not any.

Perceived Frustration: Perceived frustration was not significantly changed in the problems reconciliation scenario. The first task of the problem reconciliation scenario involved reviewing 7 CCDs, which imposed higher perceived workload and time demands. We were surprised that participants did not report a higher level of frustration with this task. One possible explanation is that the participants became familiar with the process used in the study by the time they were asked to complete this final scenario. After conducting 4 tasks in the previous scenarios, they may have become unintentionally primed and therefore were faster in executing the final scenario. This explanation is further supported by the similar mean times between the referral scenario (with 3 CCDs) and the problem reconciliation scenario. Thus, “multiple-treatment interference” may have been an unintended flaw in the method design.

Mistakes in Medication Reconciliation: Three participants did not successfully reconcile some medications. In 1 case, the participant missed a medication; and in another, the participant included both “Lorazepam” and “Ativan,” which are the same drugs with different brand names. These oversights were not surprising given the mental complexity of reconciling more than a dozen medication names across multiple summary documents. Automating parts of the reconciliation process therefore has the potential for reducing errors and avoiding unintended harm beyond simply reducing time on task and perceived workload.

Interview Responses: In their open-ended responses, participants noted several challenges related to finding information in patients’ medical records. All participants believed that collecting information electronically is helpful in capturing a more comprehensive history of medical records. However, overloading providers with too much information can make it difficult to find the information needed at a particular moment. The participants suggested that providers need only relevant information in their clinical practice and prefer not to review several pages of medical records even if it is already consolidated by our system. They further complained about missing, inaccurate, incomplete, and conflicting information in the medical records, which creates uncertainty while making clinical decisions.

Missing or Inaccurate Information: Although addressing the problem of missing or inaccurate information is beyond the scope of this study, our system currently possesses several rules to detect conflicting information. For instance, if 1 CCD shows that the patient is allergic to penicillin and the other shows no known allergies, or if in 1 CCD, the patient is presented as “Smoker” and in the other one “Never Smoked Cigarette,” the systems detects these conflicts but it is limited in providing the physician a technical mechanism to fix this conflict in the information, as there is no standard method in CCD structure to expose conflicts. Therefore, exposing and resolving conflicts will be a challenge for future work.

Design of the User Interfaces: Another interview topic was the use of complex or poorly designed user interfaces as another barrier to finding information. Our small contribution in this regard was to use hyperlinks in the outline section of each CCD. This way, the participants were able to click on the section of interest for review (allergies, medications, or problems list) instead of searching for that section in the whole CCD. All of the participants found the hyperlinks very helpful for finding information and actually used it while executing the tasks during the study. It is important to note, however, that a detailed evaluation of interface design was outside the scope of this study.

Recommendations from the Participants: In our final discussion with the participants, we received some recommendations about data categorization. The participants recommended to categorize the information into types of patient visit notes (admission, discharge, or office visit), time stamp data (ie, active medications), and categorization of problems (ie, heart problems). Currently, the underlying structure of CCDs is not designed in a way to categorize the data based on these types. However, for generating a HTML version of a CCD, it is possible to improve the style sheet that transforms XML-based CCD to HTML. Through improving the style sheet, it is possible to select specific information from CCDs and present them in a categorized fashion.

The quantitative results and participants’ response to the interview questions show that our system was successful in facilitating medical information reconciliation and helped the participants to gather the required data needed for making clinical decisions faster than manual processes. However, the open-ended discussions also show that the current application is not an ultimate solution, and more work remains to bring the desired information together in a way that healthcare providers can easily understand the current and past medical history of patients. Presenting comprehensive medical history that supports accurate clinical diagnosis requires improvements in the system from different perspectives such as usability, EMR integration, user interface, and proper data representation.

Limitations

Although the results of this evaluation are promising, the study has several limitations to note. First, given the small sample size drawn from a single health system, results may not extrapolate to other regions or to clinicians with different HIE infrastructures. Second, the limited number of scenarios examined may not be representative of information reconciliation generally. Third, this study did not examine how consolidated information may be presented in commercially available EHRs, which is required to provide information reconciliation functionality. These aspects could be tested in future, larger studies involving methods for information reconciliation.

CONCLUSION

In this study, we sought to assess the impact of a CDA-consolidation system on providers’ perceived workload while executing a medical document review. Our findings demonstrated that consolidated CCDs can be reviewed in significantly less time compared to multiple CCDs (approximately 50% less for 3 scenarios). Also, participants reported they perceived less workload while reviewing consolidated CCDs. Although providers considered it challenging to review relevant information even in consolidated CCDs, participants found our CDA-consolidation system very helpful in making accurate clinical decisions and unanimously agreed that they would like to use such a system in their clinical practice.

FUNDING

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

CONTRIBUTORS

Hosseini M. and Dixon BE. conceived of the study and initiated the study design. Hosseini M. worked on system development, statistical analysis, system evaluation, and conducting interview sessions. Hosseini M. prepared the draft version of the article. Dixon E., Jones J., Faioka A., Vreeman D., and Wu H. supervised the study and helped with reviewing and editing the article. All authors contributed to refinement of the study protocol and approved the final manuscript.

Conflict of interest statement. The authors have no competing interests to declare.