Variation in Physicians’ Electronic Health Record Documentation and Potential Patient Harm from That Variation

Abstract

Background

Physician-to-physician variation in electronic health record (EHR) documentation not driven by patients’ clinical status could be harmful.

Objective

Measure variation in completion of common clinical documentation domains. Identify perceived causes and effects of variation and strategies to mitigate negative effects.

Design

Sequential, explanatory, mixed methods using log data from a commercial EHR vendor and semi-structured interviews with outpatient primary care practices.

Participants

Quantitative: 170,332 encounters led by 809 physicians in 237 practices. Qualitative: 40 interviewees in 10 practices.

Main Measures

Interquartile range (IQR) of the proportion of encounters in which a physician completed documentation, for each documentation category. Multilevel linear regression measured the proportion of variation at the physician level.

Key Results

Five clinical documentation categories had substantial and statistically significant (p < 0.001) variation at the physician level after accounting for state, organization, and practice levels: (1) discussing results (IQR = 50.8%, proportion of variation explained by physician level = 78.1%); (2) assessment and diagnosis (IQR = 60.4%, physician-level variation = 76.0%); (3) problem list (IQR = 73.1%, physician-level variation = 70.1%); (4) review of systems (IQR = 62.3%, physician-level variation = 67.7%); and (5) social history (IQR = 53.3%, physician-level variation = 62.2%). Drivers of variation from interviews included user preferences and EHR designs with multiple places to record similar information. Variation was perceived to create documentation inefficiencies and risk patient harm due to missed or misinterpreted information. Mitigation strategies included targeted user training during EHR implementation and practice meetings focused on documentation standardization.

Conclusions

Physician-to-physician variation in EHR documentation impedes effective and safe use of EHRs, but there are potential strategies to mitigate negative consequences.

INTRODUCTION

In an effort to improve health care quality and decrease costs, physicians and health systems have been encouraged to adopt electronic health records (EHRs).^1–3 Theoretically, EHRs should increase efficiency by providing rapid access to up-to-date information. However, evidence suggests that EHR implementation results in additional time spent completing documentation as well as decreased accuracy.^4–6 One possible contributor to EHR inefficiency is physician variation—that is, differences in the content, structure, or location of patient information in the EHR that are a result of how individual physicians use the EHR rather than differences in patients’ clinical status. Commercial EHR systems are commonly designed with substantial optionality to accommodate different preferences for how users record information for an identical patient in the EHR. Although there should be variation in documentation based on a patient’s clinical status, it is potentially problematic if other factors, such as user preferences, drive documentation decisions.

The small number of studies that have examined physician EHR documentation reveal substantial variation, even for basic information such as drug allergies and smoking status.^7–11 Ancker and colleagues found that the annual average proportion of encounters that updated a patient’s problem list ranged from 5 to 60% among 112 physicians in a network of federally qualified health centers.¹² However, no large-scale studies of physician variation in documentation exist. To the extent that such variation is widespread, it is critical to understand its causes (particularly, causes beyond patient mix), the ways in which it may compromise the quality of care, and the strategies that could effectively minimize any harmful effects.

Primary care providers’ documentation serves as the foundation for care coordination, population health management, referrals, and orders.^13–15 Therefore, it is critical to assess variation in this setting. We focused on isolating differences across physicians in the same practice because they should be treating a similar patient mix as well as working under the same organizational and geographic conditions. Any remaining variation is thus likely due to physician preferences, which may interfere with care delivery by making information difficult to use for subsequent providers. We used mixed methods to answer the following research questions:

1. For core categories of clinical documentation, which categories, if any, have high variation across primary care physicians in the same practice?

2. What are the perceived causes of such variation in EHR documentation and how, if at all, do primary care physicians and their staffs perceive that variation affects care delivery and outcomes?

3. What strategies could primary care practices use to prevent or mitigate the negative consequences of variation in EHR documentation?

METHODS

Summary

We used a sequential, explanatory, mixed-methods design. We first used data from a national ambulatory EHR vendor to quantify physician-to-physician variation for 15 categories of clinical documentation. Once we identified documentation categories with high variation, we conducted semi-structured interviews with physicians and staff in primary care practices to explore the causes and consequences of such variation as well as to identify strategies to prevent or mitigate negative consequences of variation. The Michigan Institutional Review Board (OHRP IRB Registration Number IRB00000246) approved this study.

Quantitative

Setting and Data

We obtained de-identified EHR log data from a commercial EHR vendor that automatically captures and stores clickstream data when users are logged in to the EHR. We worked with the vendor to aggregate the data to 15 mutually exclusive clinical documentation categories, such that a given click would represent a documentation action in the given category (see Appendix Table 4). For example, if a user entered the patient’s blood pressure, or checked a “reviewed” box after reviewing that information, the data would report action under the clinical documentation category of Collect, Update, and Review Vital Signs. Documentation actions included both structured documentation and unstructured documentation. For example, clicking an option on the pre-populated list of medications was counted under the clinical documentation category of Update Medication List, as was a free-text entry in the medication list. Viewing any part of the record without taking some type of action to add, review, or remove information was not captured.

Table 4.

Description of Clinical Documentation Categories

Clinical documentation category	Definition
Assessment and diagnosis	Provider’s clinical diagnoses, decision-making, and treatment plan
Collect clinical encounter reason	The patient’s chief complaint
Collect vitals	Standard vital signs, such as heart rate, blood pressure, and respiratory rate; includes documentation of adding, removing, or reviewing information
Conduct physical exam	Provider’s physical exam, to include abnormal, relevant normal, and pertinent negative findings
Conduct procedure	In-office bedside procedures, such as minor dermatologic or gynecologic procedures
Conduct review of systems	Standard organ-specific system review, emphasizing pertinent negatives
Confidential information	HIPAA-protected, patient-specific identifiers and patient-specified confidential information, such as HIV status; includes documentation of adding, removing, or reviewing information
Creating or sending out orders	Implementation of the treatment plan, including patient instructions
History of present illness	Narrative about the patient’s chief complaint, including pertinent positives and negatives; includes documentation of adding, removing, or reviewing information
Interpret incoming clinical data	Interpretation of the results of tests, other notes, and so on
Medication list	Prescription and nonprescription medications the patient is taking or has taken; includes documentation of adding, removing, or reviewing information
Problem list	List of diagnoses the patient has or has had; includes documentation of adding, removing, or reviewing information
Review and discuss documents	Discussion of external documents with the patient
Sign-off or close encounter	Closing the encounter
Social history	Patient’s lifestyle practices (e.g., diet, exercise) and habits (e.g., smoking, alcohol consumption); includes documentation of adding, removing, or reviewing information

Every documentation action was tied to a patient visit (“encounter”) ID as well as a user ID. Users included physicians as well as other billing providers, clinical support staff, and administrative staff. User IDs linked each documentation action to the user’s role (physician or staff) and specialty (for physicians only). Each encounter was also linked to the specific practice location with a Practice ID. For organizations that had more than one practice location, there was a Provider Organization ID. For each Practice ID, we also received information on the state where the practice was located.

The data set provided by the EHR vendor included all documentation actions in the 15 categories for each encounter that occurred in June 2012 in all ambulatory primary care practices that had implemented the vendor’s system. We restricted the data set to active primary care providers who had used the EHR for at least 6 months in a practice with 2 or more providers. The final analytic sample included 170,332 encounters by 809 primary care physicians who were located in 237 practices across 27 states.

Measures: Dependent Variables: Documentation Completion Per Physician

We assigned each encounter to the physician who completed documentation for the most clinical documentation categories. We then created 15 binary indicators for each encounter that described whether or not anyone in the practice completed each clinical documentation category (that is, one or more actions captured in clickstream data). This rolls up documentation from other billing providers, clinical support staff, and administrative staff to the physician responsible for the visit to ensure that our measure of variation is not simply picking up differences in division of labor with respect to EHR documentation. We opted to use binary indicators rather than measure the number of documentation actions within each documentation category because it was unclear whether more actions reflected more complete documentation, at least in part because each documentation category had a different number of potential actions. Finally, for each physician, we calculated the proportion of assigned encounters with completed documentation (by that physician or by someone else) for each of the 15 clinical documentation categories over the course of the month (that is, 15 outcome measures per physician). For example, if a patient’s vital signs were documented in 5 of 20 encounters assigned to a physician in the month, the physician’s proportion would be 0.25 for that measure.

Measures: Identifying Variables

We created a categorical variable for physician primary care specialty type: family medicine, internal medicine, OB/GYN, and pediatric medicine. We also created a set of identifiers to capture the nesting of physicians within practices, practices within provider organizations, and provider organizations within states.

Analytic Approach

For each of the 15 documentation categories, we calculated the median and interquartile range of documentation completion across the 809 physicians. Because there were no established benchmarks for high versus low variation, we looked at the magnitude of the interquartile ranges and identified 50% as a natural cutoff differentiating high- and low-variation documentation categories (see Appendix Fig. 1). For the high-variation documentation categories (IQR > 50%), we isolated the amount of variation across physicians in the same practice by measuring the variation accounted for across practices, provider organizations, and states. Specifically, we estimated a multilevel linear regression model with the physician proportion of documentation as the dependent variable, primary care specialty as the independent variable, and random effects variables to capture variation at practice, provider organization, and state levels. We calculated the ratio of remaining variation (that is, variation across physicians in the same practice) and the ratios of explained variation for practice, provider organization, and state levels to total variation. We tested whether these ratios were statistically different from zero by using bootstrapped standard errors. To counteract the problem of multiple comparisons, we applied the Benjamini-Hochberg false discovery rate control procedure.¹⁶ We interpreted a ratio statistically different from zero as a significant amount of variation.

Figure 1.

Differentiating high- from low-variation categories. Note: High-variation categories are colored orange.

Conceptually, this approach relied on the assumption that physicians in the same primary care specialty within the same practice would be treating a set of patients in the observed month with the same distribution of documentation needs.¹⁷ To assess the robustness of this assumption, we calculated the proportion of variation across physicians for two low-variation documentation categories to see if the ratio of remaining variation at the physician level was lower than for high-variation documentation categories (see Appendix Table 5).

Table 5.

Variation of Documentation Categories

Variation category	Documentation category	Completion (median across physicians)	Interquartile Range (25%‘ile–75%‘ile)	Percent of explained variation at each level			Remaining variation at the physician level
				State	Provider Organization	Practice
Low	Sign-off/close encounter	100%	0%
			100–100%
Low	Creating/sending out orders	100%	0.7%
			93.3–100%
Low	Collect vitals	99.1%	4.1%
			95.9–100%
Low	Collect clinical encounter reason	98.2%	7.1%
			92.9–100%
Low	Conduct physical exam	96.6%	9.0%
			90.1–99.1%
Low	History of present illness	95.2%	13.8%
			85.5–99.3%
Low	Medication list	91.5%	19.5%	7.7%	20.7%	27.8%	43.7%
			77.9–97.5%	[0.15]	[0.15]	[0.15]	[0.46]
High	Social history	76.1%	53.3%	4%	17.8%***	16%**	62.2%***
			39.5–92.8%	[0.02]	[0.04]	[0.05]	[0.03]
High	Conduct review of systems	73.5%	62.3%	3.2%	15.6%**	13.5%*	67.7%***
			32.9–95.2%	[0.03]	[0.05]	[0.05]	[0.04]
High	Problem list	33.7%	73.1%	1.3%	9.7%*	19%**	70.1%***
			3.5–76.6%	[0.01]	[0.04]	[0.06]	[0.04]
High	Review or discuss documents	18.7%	50.8%	7.1%*	5%	9.9%	78.1%***
			10.3–61.2%	[0.03]	[0.04]	[0.06]	[0.04]
High	Assessment and diagnosis	13.4%	60.4%	0%	8.3%	15.8%*	76%***
			2.6–62.9%	[0.01]	[0.04]	[0.06]	[0.05]
Low	Interpret incoming clinical documents	14.6%	17.3%	2%	15.3%**	37.7%***	45%***
			8.1–25.4%	[0.01]	[0.05]	[0.06]	[0.04]
Low	Conduct procedure	3.7%	6.7%
			1.8–8.5%
Low	Confidential information	2.6%	5.2%
			1.1–6.3%

Note: Bootstrapped standard errors appear in brackets. Because models did not converge for most low-variation tasks, we only include regression results for high-variation tasks and two low-variation tasks. With the exception of creating and sending out orders, which was at minimum completed in 14% of encounters, and interpreting incoming clinical documents, which was at maximum completed in 73% of encounters, the range of documentation completion was 0–100%

*p significant at the 0.05 false discovery rate; **p significant at the 0.01 false discovery rate; ***p significant at the 0.001 false discovery rate

All quantitative analyses were performed by using Stata version 13.¹⁸

Qualitative

Setting and Data

We identified internal or family medicine practices that used a commercial EHR in southern and central Michigan from a list of practices that had worked with the state’s Regional Extension Center to achieve Stage 1 Meaningful Use. We restricted the sample to practices with at least 2 physicians. We intentionally included practices that used EHRs from a range of vendors to maximize the generalizability of the results beyond the vendor that provided data for the quantitative analysis.¹⁹ We invited the 51 practices that met these criteria and 10 agreed to participate.

In each participating practice, we conducted face-to-face interviews with at least one physician and one other respondent who regularly used the EHR. Interviews lasted 30 to 90 min. We performed both one-on-one and group interviews, based on respondents’ preferences. All interviews were transcribed. Each respondent received a $75 gift card. Data collection occurred February through May 2016.

Our semi-structured interview guide asked respondents about perceived variation in EHR documentation, factors that caused variation, the effects of such variation, and strategies to manage variation (see Appendix 4). We piloted and refined the interview protocol in a convenience sample of 2 primary care physicians.

Analytic Approach

We developed an a priori code list for qualitative themes.^20–22 One member of the research team applied these codes to 3 transcripts. Next, 2 other members of the research team independently reviewed the final code list and the coded transcripts to ensure comprehensiveness and consistency. The original member of the research team applied the final codes to the remaining 37 interviews. We uploaded all coded interviews to Atlas.ti²³ and used the query function to group interviews by code. We synthesized this information in analytic matrices²⁴ to identify themes that emerged in interviews across multiple practices regarding the prevalence of variable documentation, resultant challenges, and strategies for addressing variation.

RESULTS

Quantitative and Qualitative Samples

For the 237 practices in the quantitative sample, the average number of physicians was 12.6. Practices had used the EHR for over 4 years on average (Table 1). The most common specialty was family medicine (69.1%), followed by internal medicine (18.1%).

Table 1.

Descriptive Characteristics of the Quantitative Sample

	Mean	Std.	Min	Max
Practice-Level Variables
Number of physicians per practice	12.6	15.9	2	65
Number of other users per practice	13.9	10.1	0	45
Number of total users per practice	26.3	22.7	2	89
Months on the EHR	51.4	44.4	6	213
Physician-Level Variables
Physician age (years)	46.3	11.5	25	81
Physician daily encounter volume	16.7	8.4	5	41
Number of users per encounter*	2.2	0.5	1	4
Physician specialty	Freq.	Percent	Cum.
Family medicine	559	69.1	69.1
Internal medicine	146	18.05	87.14
OB/GYN	50	6.18	93.33
Pediatric medicine	54	6.67	100

*Number of users completing any documentation during each encounter, including the physician

The qualitative sample included 5 independent practices and 5 practices that were part of larger health systems, collectively using five different commercial vendors. We interviewed 40 individuals in varying roles across the 10 practices, ranging from 2 to 6 interviewees per practice (Table 2).

Table 2.

Practice Characteristics in the Qualitative Sample

Org.	Practice	Size	FQHC	EHR vendor	Year EHR adopted	Respondent characteristics*
						Physicians	Clinical staff	Other staff	Total
A	1	L (10+)	Yes	NextGen	2008	2	1	1	4
B	2	M (5–9)	Yes	athenahealth	2012	2	4		6
C	3	M (5–9)	Yes	Epic	2012	1	2	1	4
	4	S (2–4)	Yes	Epic	2006	1	1		2
	5	M (5–9)	Yes	Epic	Unknown	4	1	1	6
D	6	S (2–4)	No	eClinical-Works	2015	1	1	3	5
E	7	M (5–9)	No	eClinical-Works	2013	2	1	2	5
F	8	M (5–9)	Yes	athenahealth	2014	1		1	2
	9	S (2–4)	Yes	athenahealth	2014	1	1		2
G	10	L (10+)	No	Cerner	2013	1	1	2	4
Total	NA	NA	NA	NA	NA	16	13	11	40

*Physician category includes nurse practitioners. Clinical staff category includes medical assistants and nurses. Other staff category includes medical directors, quality managers, and front office and other administrative staff

High-Variation Categories of Documentation

Ten documentation categories had low variation in the percentage of encounters for which documentation was completed (IQR < 20%), and 5 documentation categories had high variation (IQR > 50%) (See Appendix Table 5 for low-variation categories’ IQRs). The documentation category with the most physician-level variation in the percentage of encounters for which documentation was completed—Updating the Patient’s Problem List—had an IQR of 73.1% (Table 3). Results of the multilevel model showed that 70% of the variation in Updating the Patient’s Problem List was attributable to physician variation within the practice. Physicians were somewhat less variable in their rates of documentation in Reviewing and Discussing Documents (IQR = 50.8%); however, the percentage of variation at the physician level was highest in this category (78.1%) (Table 3). Results from our robustness test for two low-variation documentation categories showed lower magnitude (and variably significant) variation at the physician level (Appendix Table 5).

Table 3.

Characteristics of High-Variation Documentation Categories

Documentation category	Completion (median across physicians)	Interquartile Range (25%‘ile–75%‘ile)	Percent of explained variation at each level			Remaining variation at the physician level
			State	Provider organization	Practice
Review or discuss documents	18.7%	50.8%	7.1%*	5%	9.9%	78.1%***
		(10.3–61.2%)	[0.03]	[0.04]	[0.06]	[0.04]
Assessment and diagnosis	13.4%	60.4%	0%	8.3%	15.8%*	76%***
		(2.6–62.9%)	[0.01]	[0.04]	[0.06]	[0.05]
Problem list	33.7%	73.1%	1.3%	9.7%*	19%**	70.1%***
		(3.5–76.6%)	[0.01]	[0.04]	[0.06]	[0.04]
Conduct review of systems	73.5%	62.3%	3.2%	15.6%**	13.5%*	67.7%***
		(32.9–95.2%)	[0.03]	[0.05]	[0.05]	[0.04]
Social history	76.1%	53.3%	4%	17.8%***	16%**	62.2%***
		(39.5–92.8%)	[0.02]	[0.04]	[0.05]	[0.03]

Note: Bootstrapped standard errors appear in brackets

*p significant at the 0.05 false discovery rate; **p significant at the 0.01 false discovery rate; ***p significant at the 0.001 false discovery rate

Perceived Drivers of Variation in Documentation

All practices in the qualitative sample reported variation in documentation across physicians. Most respondents attributed variation to idiosyncratic physician choices, facilitated by the multiple options available in the EHR to document each category of information. As respondents noted, different options placed different constraints on documentation, suggesting that users selected an option that had a tolerable set of constraints. One of the more common scenarios related to physicians’ preferences for structured or unstructured documentation. For example, the medical director at one practice, who was also a practicing physician, explained that when documenting the history of present illness the EHR allowed users to choose between a structured template that would generate a note and an unstructured template with a single free-text field: “it really depends on the provider whether they check more boxes or if they type more.”

Respondents identified a number of reasons for variation. The first reason was implementation procedures. Many respondents pointed to a lack of training when they first implemented the EHR. One physician suggested that people developed different documentation behaviors in her practice because their training occurred entirely on video, instead of in person: “the videos move really fast, and people are still asking [questions when the next segment begins]…a lot of the variation really comes from that.” In contrast, respondents from a practice that perceived very little variation in EHR documentation attributed the consistency to clearly articulated documentation procedures learned during implementation. Another common explanation for variation was the differences in how physicians viewed templates. Respondents suggested that physicians who sought to mimic the experience of paper records (because they felt that paper records offered a better structure for documentation) were more likely to use free-text fields instead of structured fields, which led to different documentation styles across physicians in the same practice.

Perceived Effects of Variation

Many respondents perceived variation as having substantial negative effects on the experience of documenting care, such as undertaking redundant documentation when there were multiple places to record comparable information in order to ensure that the information could be found in all potential locations. In practices where users did not take extra time to complete documentation, the consequence was extra effort to search for information after the visit. As one physician noted, although these recurring inefficiencies were “only a few seconds, it adds up.”

A subset of respondents were concerned that variation in documentation, typically when documenting patient problems, interfered with the quality of care. As one physician noted, different preferences for maintaining the problem list created longer lists with “junk” information. “You may not know of something that’s important … if there’s a lot of irrelevant information,” the physician said. “It makes it harder to know what’s a real problem versus what’s transient.” One respondent noted a similar risk of “error via misinformation.” As another physician explained, varied documentation of diagnoses could lead to confusion, which is particularly problematic because “everything is driven by the diagnosis nowadays…. It affects the way you approach the patient…. It can affect everything.” The quality improvement director from another practice noted that the same frustrations physicians experienced when addressing variation in documentation of patient problems diminished patients’ trust in the practice because they felt like their record was out of date.

Strategies to Manage Variation

Discussion of documentation during regularly scheduled staff meetings was the most commonly identified strategy to prevent variation in documentation. Respondents said that having frequent opportunities to discuss EHR documentation was useful to identify documentation strategies that could be adopted uniformly by all users. These meetings, which typically occurred monthly or quarterly, were frequently reinforced by emails about best documentation practices. Several respondents felt these follow-up communications were essential, especially if people worked at multiple practices and used EHRs from multiple vendors.

The second most commonly identified strategy to prevent variation was thorough training during implementation. Respondents believed that a clear articulation of EHR functionalities and the ways that different documentation decisions affected where and how information was displayed could help achieve consensus regarding best documentation practices. One practice’s office manager suggested that training a practice manager ahead of all other practice staff would allow the manager to provide ongoing coaching during implementation to further minimize variation.

CONCLUSION

Discussion

A primary care practice’s ability to leverage an EHR to improve health care delivery and patient outcomes depends upon how its physicians use the EHR to document care. This study is among the first to quantitatively capture the level of variation in clinical documentation across physicians and the first to do so in a large set of practices across the nation. It is also the first to explore EHR users’ perceptions of the causes and effects of variation. After combining a large task-log data set of 170,332 encounters with in-depth qualitative interviews in 10 primary care practices, we found substantial variation in documentation for 5 categories of clinical information which was perceived to result from optionality in the EHR design and varied implementation practices. Our results revealed that such variation jeopardizes the efficient and possibly safe delivery of care.

The 5 high-variation clinical documentation categories that we identified in the EHR task-log data have substantial EHR optionality, such that physician documentation choices can result in variation.¹⁵ For example, a Review of Systems is often structured as a component of a clinical note covering the patient’s organ systems, with a focus on the subjective symptoms as perceived by the patient.²⁵ If the elicited information leads to the identification of a problem or diagnosis, that information could be documented in the Review of Systems, the Problem List, the Assessment and Diagnosis, or in all three categories. Our qualitative work reveals how preferences result in variation in this scenario. Allowing physicians to document either in free-text fields or via structured data entry gives the documenting physician more flexibility but impairs the ability of future users to search and find information.^{8, 26–28} A physician who prefers to document using more unstructured text might document a new diagnosis in the Assessment and Diagnosis section, but that new diagnosis may be overlooked by future users unless the physician also updates the Problem List. A physician who prefers templates to manage a mix of unstructured and structured text may choose to use the Review of Systems to document this same information. Our results therefore not only reveal where variation occurs but also offer a plausible mechanism to explain why we observe variation in certain documentation categories.

EHR design optionality, along with minimal organizational constraints on documentation, may relieve the strain of EHR adoption on frontline physicians by allowing them to document in the way they individually prefer.^{29, 30} However, their decisions may be guided not by a systematic determination of best practices but instead by an ad hoc and idiosyncratic process resulting in a documentation style that works well enough for a given user. Rather than optimizing documentation styles for the practice, the aggregation of varied individual choices incurs substantial costs over the long run. First, it compromises the retrieval of information at subsequent visits, which encumbers the delivery of high-quality care. For example, when different physicians in a practice use the same fields inconsistently (some documenting a new diagnosis in the Problem List and others documenting it in the Assessment and Diagnosis), it can lead to challenges with interpreting information and subsequently cause patient harm (for example, missing a diagnosis by looking in the wrong field). Second, many health care initiatives, such as precision medicine, require that EHRs contain complete and accurate patient data against which the latest evidence can be applied in order to identify opportunities to improve patient care.³¹ Similarly, the potential power of analyzing data stored in EHRs across the country undergirds the promise of the learning health system to provide ongoing feedback to both physicians and health care standards.³² Variation in EHR documentation makes it more difficult to pursue these important efforts that rely on leveraging EHR data from different institutions and settings. Together, this suggests that benefits from allowing variation in documentation may not be worth these substantial costs and that efforts to move toward more constrained, standardized documentation are therefore worth pursuing.

Our study offers insights into how practices can move toward more standardized documentation. Specifically, targeted user training during implementation to articulate the effects of documentation decisions and regular practice meetings to develop consensus around documentation are feasible and effective strategies. However, we did not find that these strategies were in widespread use because variation in EHR documentation manifests as small, frequent annoyances rather than substantial, salient problems. This makes it difficult to pursue documentation standardization as a high priority. Therefore, third-party stakeholders—in particular, payers and policymakers—may need to draw attention to the downstream costs of variation in EHR documentation and create incentives that motivate practices to pursue more standardized documentation.

Limitations

Our results should be interpreted with several limitations in mind. First, the de-identified EHR data came from a single vendor as well as lacked patient characteristics and conditions. Due to the former, it is possible that observed variation is not generalizable to other vendors. Due to the latter, it is possible that variation attributed to the physician level actually reflects differences in patients. However, while there may be some patient differences between physicians in the same practice that contribute to the variation we observe, we believe that our findings are robust to this risk because prior work describing patient panel characteristics across a sample of primary care physicians had narrow confidence intervals,¹⁷ suggesting a relatively similar mix of patients across primary care physicians in general, and differences in patient panel characteristics did not emerge as an explanation for variation in our qualitative interviews with practices using multiple vendors, which further confirmed such variation at the physician level and substantial enough to cause problems. Likewise, although data only reflect documentation during one month (June 2012), we have no reason to believe these encounters would differ meaningfully from encounters in any other month in a way that would impact variation between physicians in the same practice, and heard nothing to that effect in our qualitative interviews. Furthermore, practices using the vendor are only located in a subset of the USA; while the 27 represented states are not clustered in any particular geographic region, this nonetheless limits the generalizability of our findings. Additionally, qualitative data collection relied on respondents’ perceptions, which were not compared to data from their EHR to more conclusively determine the prevalence of certain forms of variation. Finally, the prevalence of variation by documentation category, the impacts of variation on care delivery, and the utility of identified strategies to minimize variation that we found may be different in specialty practices, which future research should examine.

Conclusion

In the first large-scale study of variation in EHR documentation, we found substantial variation in the completion of documentation for 5 clinical documentation categories. Such variation was perceived to detract from efforts to use the data subsequently and impede quality gains from the use of EHRs. Our study suggests targeted user training during EHR implementation and regular practice meetings focused on documentation could help avoid or curb variation by promoting more standardized documentation. However, this may require third-party actions to ensure practices engage in these activities in ways that result in better patient care.

APPENDIX 3: ROBUSTNESS TEST

Our robustness test compared the amount of explained variation in high-variation tasks to the amount of explained variation in two low-variation documentation categories. Because a certain amount of variation is necessary for the multilevel models to converge, we report results for the two low-variation categories for which the model converged. If we found a different amount of explained variation, it would suggest that variation is not a universal characteristic of documentation and would reinforce that there is something different about the way documentation occurs for high-variation categories. We would thus interpret more explained variation in the low-variation categories as support for the assumption that factors other than state, physician organization, and practice impact documentation in high-variation categories.

We indeed observed different patterns among the selected low-variation categories. Specifically, the proportion of explained variation across physicians was not always statistically significant and was at least 17 percentage points less than we observed among the high-variation clinical documentation categories. This supports our interpretation that documentation patterns among the high-variation categories feature more variation at the physician level that we would attribute to physician-specific documentation patterns.

APPENDIX 4: STRUCTURED INTERVIEW GUIDE

A. Interviewee and Practice Demographics

1. Please describe your role(s) in the practice and how long you have been with the practice.

2. [ADMIN ONLY] Can you tell me a bit about your practice in general?

• How many and what types of providers and staff work at the practice?

  • i.
    How many billing providers? What are their degrees? What are their specialties?
  • ii.
    How many clinical support staff? What are their degrees?
  • iii.
    How many administrative staff? What are their roles?

• What is the average tenure of providers and practice staff?

• 3.[ADMIN ONLY] Is your practice affiliated with a Physician Organization (PO)? If so, which one? How long have you been a member?
• 4.[ADMIN ONLY] Is your organization formally affiliated with any other organizations—hospital system, clinical networks, and so on? How long have you been a member?
• 5.[ADMIN ONLY] Does your organization participate in any quality improvement or pay-for-performance programs, such as Blue Cross Blue Shield’s Physician Group Incentive Program (PGIP)? Which are most notable?

B. EHR background

Now, we have some questions for you about your electronic health record use.

• 6.[ADMIN ONLY] We understand your practice uses [fill in EHR details] EHR. Is that correct?

• How long have you had this EHR?

• What prompted your practice to adopt this EHR?

• Did any external organization (for example, PO or hospital) help you acquire or implement your EHR?

• Have you met Stage 1 or Stage 2 Meaningful Use criteria?

  i. If so, when?

  ii. Were you assisted by M-CEITA or another entity?

• Is your EHR integrated with any other health information technology (for example, patient or hospital portal)?

• 7.Are you completing any documentation on paper? [Probe if necessary on patient demographics, physician or nursing notes, problem lists, medication lists, discharge summaries, laboratory reports, radiology reports and images, and diagnostic test results and images.]
• 8.Approximately how many different EHR systems have you used?

Thank you. I’d like to focus on your current EHR, but if at any point there’s a particularly noteworthy difference or example, please let me know.

• 9.I’d like to walk through how you use the EHR. Can you please tell me how you would interact with the EHR when seeing a patient for a routine health maintenance exam?

• Do you do any documentation yourself before the visit?

• Do you do any documentation yourself during the visit?

• Do you do any documentation yourself after the visit?

• Is anyone else involved in pulling or entering information?

C. Experience with EHR variation

As we mentioned at the outset of the interview, we are particularly interested in variation in EHR documentation—that is, differences in how or when you and others in your practice might complete documentation in the EHR. For much of this interview, I’m going to focus on certain places in the EHR where there is high potential for variation:

• Assessing and diagnosing the patient

• Conducting a review of systems

• Updating the patient’s problem list

• Updating the social history

• Reviewing and discussing documents

• 10.Before we dive into the specifics, does this list of areas seem intuitive to you? Is anything surprising? Is anything missing?

I’m now going to ask you a series of questions about each of the five areas we identified [as well as respondent-identified variation].

• 11.The first type of EHR documentation we want to talk to you about is the assessment and diagnosis.

• Can you walk me through how you document the patient’s assessment and diagnosis in the EHR?

  i. Does your EHR have a designated section for a patient’s assessment and diagnosis?

  ii. How do you complete this type of documentation (for example, dictation, template, or structured data entry)?

  iii. For what proportion of your patients does someone complete this type of documentation?

• Do you perceive any differences in how others in your practice document information about the patient’s assessment and diagnosis in the EHR?

  i. Please elaborate. [Probe as necessary regarding how often tasks are documented, who completes documentation, how tasks are documented, and when tasks are documented.]

  ii. Why do you think this type of variation occurs? What causes it?

  iii. How, if at all, does this type of variation affect your ability to effectively use the EHR to care for your patients? [Probe as necessary regarding clinical decision support, care coordination, and population health management.]

• Is this something that’s risen to the attention of your practice?

• Has your practice tried any strategies to address this type of variation?

  i. Strategies to minimize the occurrence?

  ii. Strategies to minimize the impact?

• Are there other strategies that you think would be useful to address this type of variation?

• Is there anything else you think we should know about variation in how the assessment and diagnosis is used in your practice?

• 12.The second type of EHR documentation we want to talk to you about is conducting a review of systems.

• Can you walk me through how you document the patient’s review of systems in the EHR?

  i. Does your EHR have a designated section for a patient’s review of systems?

  ii. How do you complete this type of documentation (for example, dictation, template, or structured data entry)?

  iii. For what proportion of your patients does someone complete this type of documentation?

• Do you perceive any differences in how others in your practice document information about the patient’s review of systems in the EHR?

  i. Please elaborate. [Probe as necessary regarding how often tasks are documented, who completes documentation, how tasks are documented, and when tasks are documented.]

  ii. Why do you think this type of variation occurs? What causes it?

  iii. How, if at all, does this type of variation affect your ability to effectively use the EHR to care for your patients? [Probe as necessary regarding clinical decision support, care coordination, and population health management.]

• Is this something that’s risen to the attention of your practice?

• Has your practice tried any strategies to address this type of variation?

  i. Strategies to minimize the occurrence?

  ii. Strategies to minimize the impact?

• Are there other strategies that you think would be useful to address this type of variation?

• Is there anything else you think we should know about variation in how the review of systems is used in your practice?

• 13.The third type of EHR documentation we want to talk to you about is updating and reviewing the patient’s problem list.

• Can you walk me through how you document the patient’s problems in the EHR?

  i. Does your EHR have a designated section for a patient’s problem list?

1. How do you distinguish between chronic and acute problems?

ii. How do you complete this type of documentation (for example, dictation, template, or structured data entry)?

iii. For what proportion of your patients does someone complete this type of documentation?

• Do you perceive any differences in how others in your practice document information about the patient’s problems in the EHR?

  i. Please elaborate. [Probe as necessary regarding how often tasks are documented, who completes documentation, how tasks are documented, and when tasks are documented.]

  ii. Why do you think this type of variation occurs? What causes it?

  iii. How, if at all, does this type of variation affect your ability to effectively use the EHR to care for your patients? [Probe as necessary regarding clinical decision support, care coordination, and population health management.]

• Is this something that’s risen to the attention of your practice?

• Has your practice tried any strategies to address this type of variation?

  i. Strategies to minimize the occurrence?

  ii. Strategies to minimize the impact?

• Are there other strategies that you think would be useful to address this type of variation?

• Is there anything else you think we should know about variation in how the problem list is used in your practice?

• 14.The fourth type of EHR documentation we want to talk to you about is the social history.

• Can you walk me through how you document the patient’s social history in the EHR?

  i. Does your EHR have a designated section for a patient’s social history?

  ii. How do you complete this type of documentation (for example, dictation, template, or structured data entry)?

  iii. For what proportion of your patients does someone complete this type of documentation?

• Do you perceive any differences in how others in your practice document information about the patient’s review of systems in the EHR?

  i. Please elaborate. [Probe as necessary regarding how often tasks are documented, who completes documentation, how tasks are documented, and when tasks are documented.]

  ii. Why do you think this type of variation occurs? What causes it?

  iii. How, if at all, does this type of variation affect your ability to effectively use the EHR to care for your patients? [Probe as necessary regarding clinical decision support, care coordination, and population health management.]

• Is this something that’s risen to the attention of your practice?

• Has your practice tried any strategies to address this type of variation?

  i. Strategies to minimize the occurrence?

  ii. Strategies to minimize the impact?

• Are there other strategies that you think would be useful to address this type of variation?

• Is there anything else you think we should know about variation in how the social history is used in your practice?

• 15.The fifth type of EHR documentation we want to talk to you about is review of documents.

• Can you walk me through how you document the review of documents in the EHR?

  i. Does your EHR have a designated section for review of documents?

  ii. How do you complete this type of documentation (for example, dictation, template, or structured data entry)?

  iii. For what proportion of your patients does someone complete this type of documentation?

• Do you perceive any differences in how others in your practice document information about review of documents in the EHR?

  i. Please elaborate. [Probe as necessary regarding how often tasks are documented, who completes documentation, how tasks are documented, and when tasks are documented.]

  ii. Why do you think this type of variation occurs? What causes it?

  iii. How, if at all, does this type of variation affect your ability to effectively use the EHR to care for your patients? [Probe as necessary regarding clinical decision support, care coordination, and population health management.]

• Is this something that’s risen to the attention of your practice?

• Has your practice tried any strategies to address this type of variation?

  i. Strategies to minimize the occurrence?

  ii. Strategies to minimize the impact?

• Are there other strategies that you think would be useful to address this type of variation?

• Is there anything else you think we should know about how review of documents is documented in your practice?

• 16.Have you encountered other types of variation…

• …in how often tasks are documented in the EHR?

• …in who documents tasks in the EHR?

• …in how tasks are documented in the EHR (for example, free-text fields versus structured data entry)?

• …in when tasks are documented in the EHR?

• 17.Regarding [respondent-identified variation]

• Can you walk me through how you document [respondent-identified variation] in the EHR?

  i. Does your EHR have a designated section for [respondent-identified variation]?

  ii. How do you complete this type of documentation (for example, dictation, template, or structured data entry)?

  iii. For what proportion of your patients does someone complete this type of documentation?

• Do you perceive any differences in how others in your practice document information about [respondent-identified variation] in the EHR?

  i. Please elaborate. [Probe as necessary regarding how often tasks are documented, who completes documentation, how tasks are documented, and when tasks are documented.]

  ii. Why do you think this type of variation occurs? What causes it?

  iii. How, if at all, does this type of variation affect your ability to effectively use the EHR to care for your patients? [Probe as necessary regarding clinical decision support, care coordination, and population health management.]

• Has your practice tried any strategies to address this type of variation?

  i. Strategies to minimize the occurrence?

  ii. Strategies to minimize the impact?

• Are there other strategies that you think would be useful to address this type of variation?

• Is there anything else you think we should know about [respondent-identified variation] in your practice?

Summarize interview thus far: So, it sounds like ______ and ______ are common types of variation in your practice, which has ______ and ______ implications for your ability to use the EHR to deliver care. Is that accurate?

It also sounds like you have done ______ and ______ to address variation in EHR documentation, is that accurate? Now, I’d like to ask you about some other types of activities at the organization-level that might address variation.

• 18.Would it help prevent variation in EHR documentation if your practice:

• Had more training from your vendor about how to use the EHR?

• Made variation in EHR documentation a topic of regular discussion (that is, at meetings)?

• Used scribes to complete documentation in a standard format?

• Used standard after-visit reports to summarize key details?

• 19.Assuming you are stuck with your current EHR, is there anything [else] you can think of that might help minimize the problems resulting from variation in documentation?
• 20.How could EHR design be improved to manage variation in documentation?

• By minimizing its occurrence?

• By minimizing negative impacts?

D. Special populations

• 21.Now, I’d like to talk to you about some specific types of patients for whom variation in EHR documentation may be different. Is there anything about documentation of health care for women; children; racial and ethnic minorities; populations with special health care needs (chronic illness, disabilities, and end of life care needs); elderly patients; low-income patients; inner-city patients; or rural patients that would make variation:

• More or less likely?

• More or less problematic?

• Otherwise different?

Thank you for taking the time to speak with me today. Is there any additional information you think we should know about variation in EHR use?

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they do not have a conflict of interest.