Abstract
Objective
The federal meaningful use (MU) program was aimed at improving adoption and use of electronic health records, but practicing physicians have criticized it. This study was aimed at quantifying the benefits (ie, usefulness) and burdens (ie, workload) of the MU program for practicing family physicians.
Materials and Methods
An interdisciplinary national panel of experts (physicians and engineers) identified the work associated with MU criteria during patient encounters. They conducted a national survey to assess each criterion’s level of patient benefit and compliance burden.
Results
In 2015, 480 US family physicians responded to the survey. Their demographics were comparable to US norms. Eighteen of 31 MU criteria were perceived as useful for more than half of patient encounters, with 13 of those being useful for more than two-thirds. Thirteen criteria were useful for less than half of patient encounters. Four useful criteria were reported as having a high compliance burden.
Discussion
There was high variability in physicians’ perceived benefits and burdens of MU criteria. MU Stage 1 criteria, which are more related to basic/routine care, were perceived as beneficial by most physicians. Stage 2 criteria, which are more related to complex and population care, were perceived as less beneficial and more burdensome to comply with.
Conclusion
MU was discontinued, but the merit-based incentive payment system within the Medicare Access and CHIP Reauthorization Act of 2015 adopted its criteria. For many physicians, MU created a significant practice burden without clear benefits to patient care. This study suggests that policymakers should not assess MU in aggregate, but as individual criteria for open discussion.
Keywords: meaningful use, national panel, policy, primary care physician, benefit, burden
INTRODUCTION
Primary care is the frontline of medicine, providing the greatest range of health care services across the widest range of patients and environments.1,2 Primary care physicians (PCPs) account for more than two-thirds of US health care encounters,3 making them a focus of regulatory programs seeking to improve health outcomes and/or reduce costs. Over time, PCPs’ work has expanded beyond caring for patients to include participating in regulatory programs, satisfying insurance requests, and adopting emerging technologies.4–10
The US Centers for Medicare and Medicaid Service’s (CMS) meaningful use (MU) program was aimed at improving the adoption and use of electronic health records (EHRs) by showing that better management of health information through technology improves health care11 through the following:
patient evaluation, diagnosis, treatment, and care coordination;
patient safety; and
population health.
Since its inception in 2011, the MU program has been criticized more than praised by practicing physicians12–14 for increasing non–value-added work during patient encounters.6,9,15 In recent years, increasing negative sentiment has led to a range of commentary comparing physicians to highly paid data entry clerks15,16 and mocking the MU program as “meaningless abuse.”13–15 In part, this reaction stems from certified EHRs that do not allow physicians to efficiently satisfy MU requirements.5,17–20 However, reports to date are primarily grounded in opinion and case studies, which do not provide a way to quantify or generalize the impact of the MU program on physicians.
The aim of this study was to quantify the benefits (ie, usefulness) and burdens (ie, workload) of the MU program criteria for US family physicians. These data can be used to guide discussions with policymakers and health IT developers about how to improve MU’s efficiency and effectiveness.
METHODS
In 2015, the American Academy of Family Physicians (AAFP) recruited a national panel of 13 physicians, industrial and systems engineers, and experts who actively work as researchers in primary care and/or maintain primary care practices. The project was to evaluate the benefits and burdens associated with the MU program from the physician’s point of view during patient face-to-face time, arguably the most valuable segment of the patient encounter. Figure 1 shows the study’s framework.
Figure 1.
Study process.
The panel developed a consensus list of tasks minimally necessary to satisfy each of the 31 MU criteria11 (see Supplementary Appendix A); for consistency purposes, the panel established definitions for commonly used terms (see Table 1). The panel developed a survey based on this list of tasks, so that physicians could more easily evaluate each criterion for usefulness to their patients (ie, benefit) and impact on their work (ie, tasks and associated actions). Through multiple in-person pilot-test interviews, 2 on-site focus groups, and online testing of physicians, the survey was refined for structure/layout, length, question intent, specificity, and wording bias (see Supplementary Appendix B). The final design allowed physicians the freedom to respond about their specific practice and patients. Time to complete the survey was estimated to be 25–30 minutes.
Table 1.
Characteristics for survey development
| 31 unique criteria | Definition |
|---|---|
| Stage 1 | 14 required core and 7 selected menu criteria |
| Stage 2 | 17 required core and 6 selected menu criteria |
| Stages 1 and 2 together | 29 criteria |
| Two additional Stage 2 criteria | Same name as Stage 1, but different requirements to satisfy |
| Common term | Definition |
|---|---|
| Patient encounter | Events or tasks dedicated to preparing for, executing, or documenting a patient visit |
| Useful | Likely to improve assessment, diagnosis, treatment, patient satisfaction, or safety |
| Compliance | The minimum action, thought, or task to complete the objective |
| Burden | Not just relative to time, it is mental or physical effort, including interruptions |
| Response vectors | Definition |
|---|---|
| Which criterion | Physicians were asked about their current MU stage |
| Task timing | Physicians identified when each MU task occurred: before, during, or after the face-to-face portion of the visit |
| Selection | Any or all responses could be selected (no forced selection when possible) |
| Benefit scoring | The percentage of the physician’s patients criterion is applicable to <20%, 40%, 60%, 80%, 100% |
| Burden scoring | The effort required to satisfy each criterion (minimal, some, a lot, extreme, N/A) |
Data collection occurred over a 4-week period from July to August 2015. Independent of the panel, AAFP marketing resources built and hosted the survey and invited participants, which allowed confidential data collection and a deidentified dataset. To maximize the response rate, physicians received a preliminary notice (ie, schedule time) and a formal invitation by e-mail with a link to the survey, plus 3 reminders to complete the survey, 2 by e-mail and 1 by mail. Potential participants were physicians who had attested to (ie, were paid for) either Stage 1 or Stage 2 MU and who had an AAFP identification number. AAFP estimates that more than two-thirds of all US PCPs have an AAFP identification number. However, since <50% of AAFP’s contact information has been updated in the last 3 years, the study was designed based on population statistics, where the number of responses, not response rate, determined significance. It was estimated that a minimum of 375 unbiased returns of the 15 000 invitations sent would be needed for a population-level dataset representative of US physicians’ normative demographics. Figure 2 shows demographics collected for analysis and comparison. The AAFP Institutional Review Board approved protocols.
Figure 2.
Demographic information collected on participants.
Analysis of data had 2 objectives relevant to this manuscript: (1) to determine the order of the criteria in terms of benefit and burden from least to most and (2) to determine if an individual criterion’s benefit and burden were significantly different from those of the other criteria. Participants evaluated the benefit and burden for each of the 31 MU criteria once, resulting in a dataset for each. Analysis started with each criterion’s benefit and burden data being descriptively evaluated and the results placed in a table for comparison of all 31 criteria to each other. After, criteria were reordered, creating 2 additional tables for each MU stage for comparison. The initial descriptive analysis provided a rank order of the criteria by examining a criterion’s median and weighted benefit and burden scores. Comparisons of demographic subgroups were performed to determine if 1 criterion was significantly more beneficial or burdensome than others based on work distribution, setting, practice type, etc.17 A Mann-Whitney test was used for 2 level demographic comparisons by criteria and Kruskal-Wallis for more than 2 for demographic variables. Next, direct analysis of the 31 criteria as a whole based on the participants’ benefit and burden scores was performed using a Kruskal-Wallis test, with results compared to the initial descriptive analyses.17 Finally, an analysis of MU stages was performed by evaluating the cumulative effect of benefit and burden based on the weight values and rank order.
Analyses of reporting biases were performed using R × C contingency tables comparing participant demographics to known national demographics. A Bonferroni correction was used to limit the potential of type I error by rerunning analyses using an adjusted alpha of 0.0016 (31 levels) based on the initial α = 0.05. To facilitate data interpretation and panel discussion, criteria were classified based on how the information or data is most likely used in caring for patients (ie, intended use):
Basic/routine care: Patient encounters with no additional treatment or evaluation required.
Complex care: Patient encounters where management or continuance of care is required and/or likely to be coordinated with other health care professionals.
Population care: Data collected regarding patient encounters where management or surveillance of a group is intended.
Findings associated with comparison and rank order of criteria for benefit and burden are reported in this manuscript, with findings associated with demographic subgroups reported in a subsequent paper.17
RESULTS
Of 480 survey responses, 439 were usable for analysis. Responses with incomplete data and duplicate responses were excluded. Individual criterion responses ranged from 166 to 428 participants, depending on whether a criterion was in one or both MU stages. Distribution by stage showed 39% of participants in Stage 1 and 61% in Stage 2. Participant demographics (Table 2) were comparable to the profession’s demographics,21,22 with the exception of fewer participants under age 40.
Table 2.
Demographic comparisons and representations
| Source | Total | Reporting period (year) | Male/female (%) | Mean age (years) | Age distribution (%) |
Regional distribution (%) |
|||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ≤39 years | 40–49 years | 50–59 years | ≥60 years | Southwest | Mountain West Coast | Midwest | Southeast | Northeast | |||||
| American Medical Association US Physicians Report | 221 367 | 2013 | 56/44 | 49.1 | 26.6 | 25.8 | 25.6 | 22.0 | 11.5 | 20.7 | 25.2 | 26.7 | 13.9 |
| AAFP Annual Census | 68 529 | 2013–2015 | 58/42 | 47.4 | 27.7 | 29.1 | 27.0 | 16.2 | 11.0 | 20.8 | 26.8 | 25.1 | 16.2 |
| MU Study Participants | 428 | 2015 | 62/38 | 53.3 | *7.7 | 30.6 | 31.1 | 30.6 | 7.8 | 17.0 | 31.0 | 24.1 | 20.1 |
|
*Significant underreporting for MU study participants under the age of 40 years, P = 0.01 | |||||||||||||
| Physician experience with EHRs, median | 9.0 | Years | (95% responded with >4 years) |
| Practice size (# physicians) | 14.2% | Solo practice | |
| 26.4% | 2–5 physicians | ||
| 20.4% | 6+ physicians | ||
| 39.0% | Undefined | ||
| Urban vs rural practice | 50.9% | Urban | |
| Practices serving underserved patients | 22.3% | ||
| Practice organization type | 31.5% | Self-owned | |
| 31.5% | Physician group | ||
| 37.7% | Hospital/government/other | ||
A Bonferroni correction was used to adjust the alpha to 0.0016 based on the 31 criteria evaluated. This provided a conservative significance level that confirmed 40 of the 41 initial significant findings from the original alpha of 0.05 (Supplementary Appendix C). On average, a MU criterion was applicable to 58% of patients (range 20%–88%); the average burden level required to comply was “some” to “a lot,” which corresponds to 43% effort (range 16%–64%), where extreme burden is 100%. Analysis showed that criteria were perceived as being beneficial to a physician’s practice when applicable to >66% of patient encounters; those applicable to <52% were viewed negatively. Further analysis showed that 12 of 31 criteria were significantly burdensome, though 4 of these burdensome criteria were perceived as beneficial to patients (3 basic care and 1 complex care) (Table 3). Figure 3 provides a graphic representation of all 31 criteria by plotting the Kruskal-Wallis Z-scores for benefit (vertical axis) and burden (horizontal axis). To show the MU program effect, the prevalence of each stage’s general and required (ie, core) criteria were plotted by zone (Figure 4). Distributions show that Stage 1 required criteria skew positively (ie, zone 1) but progression to Stage 2 skews negatively (ie, zone 4). For a complete criteria listing with classification, benefit, burden, and Figure 3’s zone designation, see Supplementary Appendix C. Subsequent Supplementary Appendix C tables show data stratified by MU stage as a core/menu item listed by rank-order benefit for comparison.
Table 3.
MU criteria with high benefit and burden
| High benefit MU criteria: applicable to >66% of patients (P < 0.0013) | Information use | |
|---|---|---|
| Record and chart vitals | Basic | |
| Maintain active medication list | Basic | |
| a | Record electronic notes in patient records | Basic |
| Generate and transmit scripts electronically (Stage 1) | Basic | |
| Protect electronic health information | Basic | |
| a | Maintain current problem list with active diagnosis | Basic |
| Generate and transmit scripts electronically w/formulary check (Stage 2) | Basic | |
| Maintain active allergy list | Basic | |
| a | Perform medication reconciliation when receiving patient from another setting or care provider or believes encounter relevant | Complex |
| a | Use computerized physician order entry (CPOE) for new or renewal medication orders | Basic |
| Record smoking status for patients 13 and older | Basic | |
| Incorporate clinical lab test results into EHR as structured data | Basic | |
| Perform drug allergy checks | Basic | |
| High burden MU criteria (P < 0.0014) | Information Use | |
| a | Record electronic notes in patient records | Basic |
| a | Maintain current problem list with active diagnosis | Basic |
| a | Perform medication reconciliation when receiving patient from another setting or care provider or believes encounter relevant | Complex |
| a | Use CPOE for new or renewal medication orders | Basic |
| Perform drug formulary checks | Complex | |
| Provide summary care record for each transition of care or referral to the next transition of care | Complex | |
| Use secure electronic messaging to communicate with patients | Complex | |
| Provide clinical summary for patient for each office visit | Complex | |
| Use EHR technology to identify patient-specific education resources and provide to patient | Complex | |
| Implement one trackable clinical decision support rule w/drug-drug and drug-allergy (Stage 2) | Complex | |
| Implement one trackable clinical decision support rule (Stage 1) | Complex | |
| Provide patients with electronic copy of health information within 4 business days of being available | Complex | |
aIndicates criteria both beneficial and burdensome.
Figure 3.
Thirty-one MU criteria plotted for benefit and burden based on physician face-to-face time.
Figure 4.
Comparison of MU stages.
DISCUSSION
This study assessed family physicians’ perceptions of MU criteria benefit and burden levels during face-to-face patient care. Physicians perceived that more than half (18) of the criteria are useful for at least 50% of patient encounters, with 13 of these being useful in at least 66% of patient encounters. These numbers represent actual meaningful use as the program intended. However, physicians perceived 13 criteria to be of low benefit for patient care, and of those, 8 were perceived as highly burdensome. Correspondingly, it should be noted that physicians would likely take a negative view of any burden (ie, increased mental or physical effort) to comply with a task that is minimally beneficial to patient care. Hence, for these criteria to remain in a program such as MU (or its successor, the Merit Based Incentive Payment System), a strong case for clinical benefit would need to be made to justify mandatory inclusion.
A common burden was significant data entry, which was associated with many criteria’s tasks and the physician’s specific EHR. Further, 4 beneficial criteria posed a significant burden (see Table 3). However, performing medication reconciliation from another practice setting has long been a difficult task, even before EHRs or MU. Additionally, previous research found that problem lists23 were frequently not recorded,24 maintained properly,25 and/or standardized26 in patient records27 prior to MU. Therefore, some of this burden is MU addressing clinical issues that should be addressed. However, MU and the associated EHRs were supposed to provide solutions and make systems better, not simply replace the old systems. Hence, for these 4 beneficial criteria, EHRs should be designed and implemented in ways that improve patient care without negatively affecting physician workflow (ie, minimal burden, making MU criteria compliance easy).
Approximately 70% of physicians perceive Stage 1 as beneficial to patients. Stage 2 criteria were perceived as less beneficial and more burdensome to comply with. Thus, most physicians perceive a negative shift in the value of MU moving from Stage 1 to 2. This shift may be due to Stage 1 being more focused on basic care and Stage 2 more focused on complex and population care. This provides a possible explanation as to why negative sentiment regarding MU seems to have increased.
The most basic requirement of any job is having the right tools to complete each task. For MU, the physician’s tool is a certified EHR. These findings call into question the level of functionality and support EHRs provide to physicians, given the volume of significantly burdensome criteria reported. Further, our findings support literature addressing frustration with the inefficiency of EHRs8,12,13,14 and provide a quantitative estimate of the benefit and burden associated with each MU criterion, estimates that can inform policy and EHR design priorities.
Recommendations
Primary care is a complex, adaptive work system. Only by understanding and supporting physicians’ daily work will patient care improve.28 MU criteria with high benefit and low burden should continue to be supported. Additional tasks (ie, burdens) that are mandated but not clinically beneficial to the majority of patients can divert attention and effort away from direct patient care. Burdens associated with beneficial routine tasks (ie, basic care) should be minimized via better EHR design to allow for the addition of more complex patient-specific tasks. At a policy level, compliance metrics should not be 100%, so adaptive patient care is encouraged. This is supported by the finding that beneficial criteria were applicable to approximately 75% of patients, with no criteria being applicable to 100% of patients. Burdens associated with some low-benefit tasks, such as public health initiatives, should be minimized via automated EHR reporting. Thus, as a general point, components of tasks that do not contribute to the Quadruple Aim29 or fail to improve clinician and staff efficiency should be minimized or eliminated. However, a warning: not all meaningful activities in care processes can or should be measured. Further, attempting to measure some activities or outcomes, especially via documentation, may actually drive out what is important in the care process.30
Limitations
This study has several limitations. Fewer physicians under the age of 40 years completed the survey as compared to national demographic prevalence. However, the response rate for physicians under the age of 40 years is consistent with surveys performed by the AAFP marketing research group over the past 10 years, and additional testing confirmed no significant influence of reported findings. This study also focused on US family physicians, so findings may not be generalizable to specialists. The study design promoted a best-case scenario, as the lowest benefit score a physician could give a criterion was 20%. This could make criteria seem more useful than they actually are and would affect lower weighted criteria more. In addition, benefit was measured as the percentage of patients benefitting from a given criterion; the importance of the criterion regardless of the number of patients benefitting was not considered. The study asked physicians to focus on benefits and burdens related to face-to-face patient encounters. Thus, population health–focused criteria may have been viewed as less beneficial. This study did not elicit patients’ perception of benefits. Therefore, criteria that physicians may not have found very beneficial from their viewpoint might be beneficial to patients (eg, receiving notices about preventive care or having health information sent to them within a given time period). The study also did not capture nonphysician staff perceptions. Last, this study produced a small population dataset representative of US family physician demographic norms. Small population datasets are subject to self-selection bias, and the potential for self-selection bias only increases as topics, such as MU, become more polarized. Therefore, we suggest using these results to further discussions about MU criteria’s present role in performance-based payment systems, and how best to support primary physicians’ care.
CONCLUSION
These findings represent practicing US family physicians’ perceptions of the benefits and burdens associated with MU criteria. Their perceptions were that multiple MU criteria had high levels of burden, including some associated with basic/routine patient care, which constitute the foundation of patient encounters. Further, roughly one-third of MU criteria were perceived as useful in <50% of patient encounters, which means that time is taken away from typical patients for these non–value-added tasks (ie, system waste) that must be performed for compliance.6,9,27,31 Hence, the policy becomes a burden on the quality of care that physicians can provide to their patients. In total, these findings provide insight into CMS’s comment that MU has “lost the hearts and minds of the physicians.”32
At a national level, demand is growing for value-/quality-based health care, and policymakers are pushing physicians to be responsible for managing groups of patients and providing population-level care. Recently, CMS incorporated the MU criteria into the new federal merit-based incentive payment system program. To be effective, future policy (or revisions of current policy) must use sound scientific data from methods that measure the real impact on physicians and their patients before a program’s implementation.8,20,30,33 While federal programs like MU have the potential to motivate change, designers and stakeholders must be careful to avoid negatively impacting individual patient care by understanding and supporting their work system.34–36
Supplementary Material
ACKNOWLEDGMENTS
Study funding was provided by AAFP. Special thanks to Heidy Roberson-Cooper for serving on the MU panel and the AAFP marketing research department for managing the study’s data collection.
Author Contributions
All authors had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: all authors. Acquisition of data: GTH, SEW, AAFP Marketing Research Division. Statistical analysis of data: GTH, SEW. Interpretation of data: all authors. Drafting of the manuscript: all authors. Critical revision of the manuscript for important intellectual content: all authors. Obtained funding: GTH, SEW. Administrative, technical, and material support: GTH, SEW. Study supervision: GTH, SEW.
Conflict of interest disclosures
None to report.
Funding/support
This study was supported by the American Academy of Family Physicians board.
Role of the Sponsors: The content is solely the responsibility of the authors and does not necessarily represent the official views of the AAFP. The AAFP had no role in the design, conduct, and reporting of this research.
Previous Presentations: Presented in part at the 2017 I-PrACTISE Conference, Madison, WI, USA.
SUPPLEMENTARY MATERIAL
Supplementary material is available at Journal of the American Medical Informatics Association online.
REFERENCES
- 1. Dalaney BC, Fitzmaurice DA, Riaz AR et al. , Can computerized decision support systems deliver improved quality in primary care? Brit Med J. 1999;31911:1–3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Rich EC, Lipson D, Libersky J et al. , Organizing care for complex patients in the patient-centered medical home. Ann Fam Med. 2012;10(1):60–62. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Agency for Healthcare Research and Quality. Medical Expenditure Panel Survey Household Component Data – Table 1: Usual Source of Health Care and Selected Population Characteristics, United States 2008; Generated interactively. (November 29, 2011). [Google Scholar]
- 4. Alexopoulos C, Goldsman D, Fontanesi J et al. , A discrete-event simulation application for clinics serving the poor. Proceedings of the Institute of Industrial Engineering 2001 Winter Simulation Conference. 2001;2:1386–91. [Google Scholar]
- 5. Beasley JW, Wetterneck TB, Temte J et al. , Information chaos in primary care: implications for physician performance and patient safety. J Am Board Fam Med. 2011;24:745–51. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Bailey JE. Does health information technology dehumanize healthcare? AMA J of Ethics. 2011;133:181–85. [DOI] [PubMed] [Google Scholar]
- 7. Baron RJ. What’s keeping us so busy in primary care? a snapshot from one practice. N Engl J Med. 2010;36217:1632–36. [DOI] [PubMed] [Google Scholar]
- 8. Friedberg MW, Chen PG, van Busum KR et al. , Factors Affecting Physician Professional Satisfaction and their Implications for Patient Care, Health Systems, and Health Policy. Rand Health Q. 2014;34:1. [PMC free article] [PubMed] [Google Scholar]
- 9. McCrory K. Meaningful use doesn’t benefit my patients. Phys Pract. 2015. https://www.healthit.gov/providers-professionals/how-attain-meaningful-use. Accessed April 21, 2016. [Google Scholar]
- 10. Rosenbaum L. Transitional chaos or enduring harm? the EHR and the disruption of medicine. N Engl J Med. 2015;37317:1585–88. [DOI] [PubMed] [Google Scholar]
- 11. US Centers for Medicare and Medicaid (CMS). How to Attain Meaningful Use. 2015. November 4, 2015 www.healthit.gov/providers-professionals/how-attain-meaningful-use. [Google Scholar]
- 12. American EHR Partners. Physicians Use of EHR Systems. 2014. www.americanehr.com/research/reports/Physicians-Use-of-EHR-Systems-2014.aspx. Accessed October 20, 2015. [Google Scholar]
- 13. Mitra R, Mirro M. Keeping meaningful use from becoming meaningless abuse: a cardiology/ electrophysiology perspective. EP Lab Digest. 2013;137. [Google Scholar]
- 14. Watcher B. Meaningful Use. Born, 2009, Died, 2014? 2014. http://community.the-hospitalist.org/2014/11/13/meaningful-use-born-2009-died-2014/. Accessed April 21, 2016. [Google Scholar]
- 15. Green M. 25 Quotes that Show Just How Fed Up Physicians Are with EHRs. 2016. www.dolbey.com/25-quotes-that-show-just-how-fed-up-physicians-are-with-ehrs/. Accessed April 21, 2016. [Google Scholar]
- 16. Davis J. American Medical Association CEO: beware of poor-quality digital “advancements.” Healthcare IT News. 2016. http://m.healthcareitnews.com/news/american-medical-association-ceo-beware-poor-quality-digital-%E2%80%98advancements%E2%80%99. Accessed June 29, 2016. [Google Scholar]
- 17. Holman GT, Waldren S, Rivera AJ, Dardick LD. Using challenges associated meaningful-use criteria to prioritize needed changes in electronic health records. Proceedings of Human Factors and Ergonomics Society International Health Care Symposium. San Diego, CA; April 13–16, 2016. [Google Scholar]
- 18. Krist AH. Electronic health record innovations for healthier patients and happier doctors. J Am Board Fam Med. 2015;28(3):299–302. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Krist AH, Beasley JW, Crosson JC et al. , Electronic health record functionality needed to better support primary care. J Am Med Inform Assoc. 2014;215:764–71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Sinsky CA, Beasley JW, Simmons GE, Baron RJ. Electronic health records: design, implementation and policy for higher value primary care. Ann Int Med. 2014;160(10):727–30. [DOI] [PubMed] [Google Scholar]
- 21. American Medical Association. Physician Characteristics and Distribution in the US. Chicago: American Medical Association; 2015. [Google Scholar]
- 22. American Academy of Family Physicians. Annual Census Report. AAFP; 2016. Accessed May 3, 2016. [Google Scholar]
- 23. Holmes C. The problem list beyond meaningful use: part I: the problems of the problem list. J AHIMA. 2011;822:30–33. [PubMed] [Google Scholar]
- 24. Bensten BG. The accuracy of recording patient problems in family practice. Acad Med. 1976;51(4):311–16. [DOI] [PubMed] [Google Scholar]
- 25. Campbell JR. Strategies for problem list implementation in a complex clinical enterprise. In Proceedings of the AMIA Symposium. American Informatics Association; 1998: 285. [PMC free article] [PubMed] [Google Scholar]
- 26. Hummel J, Evans P. Standardizing the Problem List in the Ambulatory Electronic Record to Improve Patient Care. Seattle, WA: Washington and Idaho Regional Extension Center; 2012. [Google Scholar]
- 27. Endsley S, Magill MK, Godfrey MM. Creating a lean practice. Fam Pract Manag. 2006;134:34. [PubMed] [Google Scholar]
- 28. Holman GT, Beasley J, Karsh BT et al. , The myth of workflow in the primary care. J Am Med Inform Assoc. 2015;107:1–19. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Bodenheimer T, Sinsky C. From the triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med. 2014;12(6):573–76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Shi L, Lock DC, Lee DC et al. , Patient-centered medical home capability and clinical performance in HRSA-supported health centers. Med Care. 2015;53:389–95. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Grove AL, Meredith JO, Macintyre M et al. , Lean implementation in primary care health visiting services in National Health Service UK. Qual Safety Health Care. 2010;195:e43. [DOI] [PubMed] [Google Scholar]
- 32. Waldren S. The uncertain future of meaningful use. AAFP News 2015. www.aafp.org/news/opinion/20160128guestedmu.html. Accessed April 21, 2016. [Google Scholar]
- 33. Shanafelt TD, Hasan O, Dyrbye LN et al. , Changes in burnout and satisfaction with work-life balance in physicians and the General US Working Population Between 2011 and 2014. Mayo Clinic Proc. 2015;9012:1600–13. [DOI] [PubMed] [Google Scholar]
- 34. Carayon P, Hundt AS, Karsh BT et al. , Work system design for patient safety: the SEIPS model. Qual Saf Health Care. 2006;15(Suppl 1):i50–58. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Karsh BT, Holden RJ, Alper SJ, Or CK. A human factors engineering paradigm for patient safety: designing to support the performance of the healthcare professional. Qual Saf Health Care. 2006;15(Suppl 1):i59–65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. Stig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare system. Qual Saf Health Care. 2010;19(Suppl 3):i68–74. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.