Recommendations for a Mixed Methods Approach to Evaluating the Patient-Centered Medical Home

Abstract

PURPOSE

There is a strong push in the United States to evaluate whether the patient-centered medical home (PCMH) model produces desired results. The explanatory and contextually based questions of how and why PCMH succeeds in different practice settings are often neglected. We report the development of a comprehensive, mixed qualitative-quantitative evaluation set for researchers, policy makers, and clinician groups.

METHODS

To develop an evaluation set, the Brown Primary Care Transformation Initiative convened a multidisciplinary group of PCMH experts, reviewed the PCMH literature and evaluation strategies, developed key domains for evaluation, and selected or created methods and measures for inclusion.

RESULTS

The measures and methods in the evaluation set (survey instruments, PCMH meta-measures, patient outcomes, quality measures, qualitative interviews, participant observation, and process evaluation) are meant to be used together. PCMH evaluation must be sufficiently comprehensive to assess and explain both the context of transformation in different primary care practices and the experiences of diverse stakeholders. In addition to commonly assessed patient outcomes, quality, and cost, it is critical to include PCMH components integral to practice culture transformation: patient and family centeredness, authentic patient activation, mutual trust among practice employees and patients, and transparency, joy, and collaboration in delivering and receiving care in a changing environment.

CONCLUSIONS

This evaluation set offers a comprehensive methodology to enable understanding of how PCMH transformation occurs in different practice settings. This approach can foster insights about how transformation affects critical outcomes to achieve meaningful, patient-centered, high-quality, and cost-effective sustainable change among diverse primary care practices.

INTRODUCTION

The patient-centered medical home (PCMH) movement on the American health care scene is relatively new, though it has roots in both pediatrics and general practice.^1–3 The conceptualization of PCMH began with a broad notion of transforming primary care practices by combining the best primary care attributes with new means of structuring patient care, enhancing patient engagement and care coordination, achieving improved health outcomes, providing a better patient experience of care, improving efficiency and use of health information technology, and, ultimately, reducing costs.^4–9 Stakeholders disagree about exactly what a medical home is, to what transformation should aspire, and how “medical home-ness” should be assessed.^10–13 Evaluations of such new organizational structures were intended to consider quality, cost, and experience of patients, clinicians, and medical staff,^5,14,15 though the elements most pertinent to fostering payers’ support of these initiatives were increased efficiency and lowered costs.¹⁶

The informative richness of evaluation efforts can be compromised if their strategies have too narrow a focus. Practices’ participation in payer- or government-driven PCMH involves high stakes, with payment dependent upon required performance reporting. In this environment, evaluators tend to focus data collection on a constrained set of concrete quality measures to limit the reporting burden on primary care sites and to allow for an objective, comparative assessment of progress.^17–19 Moreover, there has been a reluctance to collect data that are not directly tied to incentives, such as the patient, family,^12,17 and clinician’s²⁰ experience of care or the transformation process itself.²¹ Often neglected in evaluations is a qualitative exploration of the mediating mechanisms and modifying conditions²² of transformation. Qualitative inquiry into these conditions helps explain and provide context for how PCMH implementation can be variable and affected by the political, organizational, philosophical, cultural, community, financial, practice, clinician, staff, and patient factors within the highly stressed atmosphere of primary care delivery.^19,21,23

Efforts to evaluate the PCMH model in the United States have occurred within the rapidly evolving environment of PCMH transformation. Diverse stakeholders are enthusiastic about the potential of the PCMH to improve health care delivery and reduce disparities.^11,24–28 Recent reports point to the success of PCMH initiatives in achieving various subsets of desired aims.^{18,21,23,29–32} Yet, as Hoff et al note,²³ there is “conceptual sponginess” to the PCMH for which concepts and labels are variously defined and differentially applied. As a result, the field—possibly prematurely—is seeking a specified set of clinical benchmarks^19,21,33,34 on which to focus to determine whether PCMH works. Such a focus leaves behind the important questions of how and why PCMH may work in particular contexts, and in what ways a practice has or has not undergone fundamental and enduring transformation in the way it functions.^21,23,35 Given the variable manifestations inherent in the politics, economics, and organizational contexts of real-world primary care practice,^21,35,36 it is important to consider whether ideal transformation processes can be universally implemented and whether at this stage they can be evaluated with universally standardized measures.^{23,33,36–42} Furthermore, it is important to consider what constitutes a comprehensive set of ingredients for successful transformation and how to evaluate the processes contributing to its success.^4,43

To address these gaps, we set out to elucidate the domains and evaluation strategies that constitute a comprehensive methodology to determine how transformation occurs within the context of practice and to explain why such transformation happens. We wanted to establish a set of metrics and methods that, when used together, address the many contextual issues involved in PCMH transformation. The set would be designed for use by researchers, policy makers, and primary care clinician groups when conducting and evaluating PCMH initiatives. Their findings could then be used to monitor change within practices, compare practices’ abilities to promote and sustain change, and to understand what conditions lead to stronger practice transformation.

METHODS

In 2011, the Brown Primary Care Transformation Initiative at Brown University’s Department of Family Medicine convened a PCMH evaluation think tank to begin development of a comprehensive PCMH evaluation data collection set that could be implemented feasibly. The think tank included 28 national and regional authorities in PCMH evaluation from family medicine, internal medicine, pediatrics, nursing, epidemiology, anthropology, sociology, psychology, health insurance, e-health, health systems transformation, the Rhode Island Department of Health, and other public health entities.

After the think tank convened, our Brown Primary Care Transformation Initiative team engaged in an immersion-crystallization process of qualitative data analysis,⁴⁴ meeting repeatedly to review the think tank notes, recordings, and transcripts. Before and after the think tank convened, 1 coauthor (D.R.P.) used the MEDLINE database to identify and review for quality and relevancy the English language articles addressing PCMH evaluation. Based on the literature findings and the published and online evaluation tools and methods, we selected tools and adapted and created others to compile a contextually comprehensive, qualitative and quantitative mixed methods PCMH evaluation set for use at baseline and follow-up. Our objective was to assemble an evaluation set that could be implemented according to PCMH project needs to uncover and explain how and why transformation occurs (or is impeded) within the cultural and philosophical construct of primary care practice and care seeking.

RESULTS

Compiling a Feasible, Contextually Comprehensive Evaluation Set

The organizing framework outlined by think tank participants called for a comprehensive evaluation set that is explanatory, attends to the context of transformation, and elicits the experiences of the diverse stakeholders’ (patients, parents or caregivers, clinicians, staff) broad range of issues involved in PCMH transformation. Mixed methods were promoted to capture the requisite baseline and follow-up data: qualitative interviews, participant-observation, focus groups, surveys, other quantitative measures, and patient outcomes. The strategies included in the set assess practice workflow and interpersonal communication, practice baseline culture, and culture transformation; patient and family centeredness and authentic patient activation; enhanced communication and trust among practice employees and among employees and patients; transparency; meaning, joy, and collaboration in delivering and receiving health care; and community integration. The evaluation set is displayed in Tables 1 through 8. All patient instruments are available in English and Spanish because 12.9% of the US population spoke Spanish at home in 2011, and of those speakers, 5.9% reported they speak English less than very well.⁴⁸

Table 1.

Quantitative PCMH Evaluation Measures: Practice, Clinician, and Staff Survey Instruments

Measurement Tool	Domains	Source, Version, Purpose, Availability
Baseline Practice Survey45	Demographics and practice information	Original NDP questionnaire (96 items). Adapted questionnaire by BPCTI (27 items). Provides information for NCQA and meaningful use. No cost Supplemental Appendix 1, http://www.annfammed.org/content/13/2/168/suppl/DC1
PCMH Implementation Survey45,46	Change capacity: teamwork, work environment, culture, trust, communication	Modified by BPCTI from 2 longer NDP scales: NDP Practice Adaptive Reserve (23 items), Modified Scale = 23 items (reflection item deleted and confidence item added), and Practice Environmental Checklist (123 items). Combined modified scale = 32 items. No cost Supplemental Appendix 2, http://www.annfammed.org/content/13/2/168/suppl/DC1
Practice Demographic Questionnaire for clinicians	Demographics of individual practice clinicians	BPCTI (9 items). For physicians, NPs, and PAs. No cost Supplemental Appendix 3, http://www.annfammed.org/content/13/2/168/suppl/DC1
Practice Demographic Questionnaire for staff	Demographics of individual practice staff	BPCTI (5 items). For nurses, medical assistants, receptionists, and other staff. No cost Supplemental Appendix 4, http://www.annfammed.org/content/13/2/168/suppl/DC1
The Clinician Activation Measure assessment (CS-PAM)	Clinician support for and beliefs about patient activation and patient self-management	Validated tool from Insignia Health (13 items). Requires purchase of a license http://www.insigniahealth.com/solutions/clinician-activation-measure
Maslach Burnout Inventory47	Measure of burnout: emotional exhaustion, depersonalization, personal accomplishment	MBI-HSS (22 items). For clinicians and staff. Available in 25 languages, free with purchase of license for English version. Website disclaimer gives no warranty for translation quality http://www.mindgarden.com/products/mbi.htm

BPCTI = Brown Primary Care Transformation Initiative; MBI = Maslach Burnout Inventory; MBI-HSS = MBI-Human Services Survey; NCQA = National Center for Quality Assurance; NDP = National Demonstration Project; NP = nurse practitioner; PA = physician assistant; PCMH = patient-centered medical home.

Table 8.

PCMH Implementation Process Evaluation Methods

Method	Domains	Source, Version, Purpose, Availability
Written reflections and progress notes	Facilitation staff document the changing contextual circumstances in the practices, in the broader environment, and in their own facilitation roles	BPCTI (5 trigger questions). Facilitation staff keep an ongoing typed log of reflections Supplemental Appendix 13, http://www.annfammed.org/content/13/2/168/suppl/DC1
Focus groups	Moderated group discussions about enabling factors and barriers to achieving evaluation or facilitation goals, how staff roles and relationships with practices evolved and the impact of this evolution, and notions about how and why the practices are or are not transforming in specific domains	BPCTI (4 core questions about evaluation data collection; 22 core questions about transformation facilitation) Supplemental Appendix 14, http://www.annfammed.org/content/13/2/168/suppl/DC1

BPCTI = Brown Primary Care Transformation Initiative; PCMH = patient-centered medical home.

Quantitative Measures

We selected or adapted publicly available written survey instruments that include the critical PCMH components we identified (see Supplemental Appendixes 1 and 2, http://www.annfammed.org/content/13/2/168/suppl/DC1 for adapted instruments). Our goal was to include instruments that we believe, when used together, are comprehensive yet manageable. Instruments in the evaluation set address practice, clinician, and staff demographics; job satisfaction; burnout; and clinician support for patient activation. Our inclusion of 3 patient instruments results from a careful selection of those we consider to be the most suitable for PCMH evaluation; they are relatively quick to complete (15 to 30 minutes to complete all 3), and they address different measures that together provide a contextually based view of the patient’s perspective.

Practices seeking PCMH accreditation are required to use the National Committee for Quality Assurance (NCQA) accreditation measure.⁴⁹ We suggest practices also use the practice self-evaluation Patient-Centered Medical Home Assessment (PCMH-A), an excellent tool for assessing baseline needs and monitoring transformation.⁵⁰ Hospital utilization can be measured by emergency department and hospital admissions and include ambulatory care–sensitive conditions (medical problems that might be prevented or can be treated outside a hospital).⁵¹ Quality is measured with specific clinical benchmarks within broad areas of care (eg, diabetes would have performance goals for hemoglobin A_1c levels, documented eye examinations, etc). These benchmarks are variously defined among PCMH projects according to local requirements and data availability, so only suggested areas are listed in Table 5 (Supplemental Appendixes 1–5, http://www.annfammed.org/content/13/2/168/suppl/DC1).⁵²

Table 5.

Examples of Clinical Benchmark Categories

Adult measurement category examples

Comprehensive diabetes care

Tobacco use assessment and counseling

Hypertension control

Breast cancer screening

Cervical cancer screening

Colorectal cancer screening

Depression screening and treatment

Weight, BMI screening

Asthma treatment

Cholesterol management

Pediatric measurement category examples

Well-child checks

Immunizations

Developmental screening

BMI measurement and classification

Patients with persistent asthma on controller medication

Screening for chlamydia in sexually active adolescent girls

Oral health risk assessment

Hearing and vision checks

Lead screening

BMI = body mass index.

Table 2.

Quantitative PCMH Evaluation Measures: Patient Survey Instruments

Measurement Tool	Domains	Source, Version, Purpose, Availability
Patient Activation Measure (PAM)	Patient activation regarding patients’ knowledge, skills, and confidence for self-management	Validated tool from Insignia Health (13 items) to inform patient activation efforts English and Spanish. Requires purchase of a license http://www.insigniahealth.com/solutions/patient-activation-measure
HRSA Patient Satisfaction Survey	Adult experiences of care at the practice	HRSA (32 items). English and Spanish. No cost. HRSA version 12/25/2012 Supplemental Appendix 5, http://www.annfammed.org/content/13/2/168/suppl/DC1
Interpersonal Process of Care Survey: Short Form (IPC-18)	Communication, patient-centered decision making, and interpersonal style	University of California, San Francisco Department of Medicine, Center for Aging in Diverse Communities (18-item short form). For patients from diverse racial/ethnic groups to describe disparities in interpersonal care, predict patient outcomes, and examine outcomes disparity reduction efforts. English and Spanish. No cost http://dgim.ucsf.edu/cadc/mm/ipcare.html

HRSA = Health Resources and Services Administration; PCMH = patient-centered medical home.

Table 3.

Quantitative PCMH Evaluation Measures: Examples of Meta-Measures

Measurement Tool	Domains	Source, Version, Purpose, Availability
Physician Practice Connections – Patient-Centered Medical Home (PPC-PCMH)	9 Standards: access and communication, patient tracking and registry functions, care management, self-management support, electronic prescribing, test tracking, referral tracking, performance reporting and improvement, advanced electronic communication	NCQA revised standards for January 1, 2014. Most commonly used measure of PCMH accreditation http://www.ncqa.org/Programs/Recognition/Practices/PatientCenteredMedicalHomePCMH.aspx
Meaningful Use	3 Core areas: data capture and sharing, advancing clinical processes, achieving improved patient outcomes	Standards defined by the CMS Incentive Programs to regulate use of electronic health records. Eligible providers and hospitals earn incentive payments by meeting criteria http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Meaningful_Use.html
Medical Home Implementation Quotient (MHIQ)	9 Modules: patient-centered medical home, practice management, health information technology, quality and safety, practice-based team care, care coordination, practice-based services, access to care and information, care management	TransforMED. Self-assessment tool to help a practice learn more about the medical home model and gauge status within the medical home continuum http://www.transformed.com/mhiq/welcome.cfm
Patient-Centered Medical Home Assessment (PCMH-A)	8 Change concept areas: engaged leadership, quality improvement strategy, empanelment, continuous team-based healing relationships, organized, evidence-based care, patient-centered interactions, enhanced access, care coordination	MacColl Center for Healthcare Innovation. Helps practices gauge progress implementing PCMH change concepts. Tested by 65 sites participating in the Safety Net Medical Home Initiative http://bsmod.dom.wustl.edu/documents/PCMH-A_SNMHI_080410.pdf

CMS = Centers for Medicare & Medicaid Services; NCQA = National Committee for Quality Assurance; PCMH = patient-centered medical home.

Table 4.

Hospital Utilization Measures

Hospital admissions per 1,000 members

Ambulatory care–sensitive conditions admissions per 1,000 members

Emergency department visits per 1,000 members

Avoidable emergency visits (ambulatory care–sensitive admissions per 1,000 members)

Hospital readmissions within 30 days

Qualitative Data Collection

Qualitative data complement quantitative data to obtain a contextualized understanding of what occurs in practices that may influence transformation and to help explain quantitative findings. Several types of participant observation allow evaluators to compare interview and survey findings with direct observation of how the practice functions. Observation templates guide trained participant-observers in writing field notes regarding the office environment. Patient pathways are conducted where an evaluator accompanies patients from registration through checkout, and for staff pathways the evaluator shadows individual staff during typical workdays (Table 6).

Table 6.

PCMH Qualitative Observational Evaluation Methods

Method	Domains	Source, Version, Purpose, Availability
Direct observation within the practice environment (example: medical practice waiting room observation guide)	Examples: use of space, people flow, and interpersonal interactions in waiting rooms, nurses’ stations and other back areas; décor and tone of each area; communication among clinicians, staff, and patients	BPCTI. Participant observation sessions conducted throughout the practice at varying times and days of the week Flexibly structured field note template Supplemental Appendix 6, http://www.annfammed.org/content/13/2/168/suppl/DC1
Pathway observations with staff	Staff experience of work. Observe work tasks and workflow for specific staff person and during interactions with coworkers: type of tasks, redundancy, efficiency, demeanor, behaviors, interactions	BPCTI. Observation template guide Supplemental Appendix 7, http://www.annfammed.org/content/13/2/168/suppl/DC1
Pathway observations with adult and pediatric patients	Patient (or parent or guardian) experience of visit. Observe workflow, patient data collection, confidentiality procedures, observation of clinician and staff communication with patient, parent, or guardian, time duration for visit segments	BPCTI. Observation template guide. Researcher takes notes while accompanying patients from check-in through checkout. Informal interviewing during wait times Supplemental Appendix 8, http://www.annfammed.org/content/13/2/168/suppl/DC1

BPCTI = Brown Primary Care Transformation Initiative; PCMH = patient-centered medical home.

In-person individual interviews with clinicians, staff, and patients should be conducted by trained interviewers using semistructured question guides. Core questions are supplemented with spontaneous probes and follow-up questions to elicit information in participants’ own words about how they think about primary care and PCMH transformation (Table 7) (Supplemental Appendixes 6–12, http://www.annfammed.org/content/13/2/168/suppl/DC1).

Table 7.

PCMH Qualitative Interview Evaluation Methods

Method	Domains	Source, Version, Purpose, Availability
Used with clinicians and staff
Individual baseline interviews with clinicians and staff	Role description, perspective of and experience working in practice, teamwork, conflict resolution, change processes, goals for change	BPCTI (20 core open-ended questions). In-person or telephone interview conducted with physicians, NPs, PAs, nurses, medical assistants, receptionists, and other staff. Supplemental Appendix 9, http://www.annfammed.org/content/13/2/168/suppl/DC1
Follow-up interviews with clinicians, PCMH practice champions, and staff	Perceptions of transformation progress, vision of practice as a PCMH, roles in transformation efforts, communication, patient engagement, changes in interpersonal interactions	BPCTI (19 core open-ended questions). In-person or telephone interview conducted with physicians, NPs and PAs, nurses, medical assistants, receptionists, and other staff. Supplemental Appendix 10, http://www.annfammed.org/content/13/2/168/suppl/DC1
Used with patients
Individual adult patient interviews	Patient’s experience and opinions about the practice, cognizance of practice transformation, understanding of PCMH	BPCTI (12 core open-ended questions). In-person interview conducted with patients aged >18 y. Supplemental Appendix 11, http://www.annfammed.org/content/13/2/168/suppl/DC1
Individual or pair: parent or guardian and pediatric patient interviews	Parent and child’s experience and opinions about the practice, cognizance of practice transformation, understanding of PCMH	BPCTI (14 core open-ended questions). In-person interview conducted with parent or guardian alone, parent-child pair if child is capable of participating, or child alone if child is capable of speaking completely for him/herself. Supplemental Appendix 12, http://www.annfammed.org/content/13/2/168/suppl/DC1

BPCTI = Brown Primary Care Transformation Initiative; NP = nurse practitioner; PA = physician assistant; PCMH = patient-centered medical home.

Implementation Process Evaluation Methods

The qualitative methods described above elicit information on transformation from clinicians, staff, and patients, who are internal to the practice. To understand practice evolution as part of an externally facilitated initiative, it is helpful to document the perceptions of the external team. Such process evaluation documents additional contextual and explanatory factors that may influence the quality and outcomes of a PCMH initiative. Our evaluation set includes guides for periodic reflection and progress notes for evaluators and facilitators and twice-yearly focus groups with these project staff. Written reflections allow staff to delve deeply into specific issues that feel important to their work and the transformation process.³⁶ Focus groups, led by a trained focus group moderator, promote group interaction around matters concerning the overall transformation initiative beyond those that commonly dominate staff meetings (Table 8, and Supplemental Appendixes 13 and 14, http://www.annfammed.org/content/13/2/168/suppl/DC1).

Application of Evaluation Data to Promote Practice Transformation

Individualized practice reports can be created by analyzing baseline and follow-up data from all methods. Reports summarize strengths and challenges of the PCMH transformation domains. We recommend that facilitation staff meet in person with practice representatives to discuss findings and devise transformation goals and strategies. Care must be taken in the reports to maintain respondent confidentiality by not linking comments with specific employees or job roles. Process evaluation data can be used to modify facilitation strategies during the transformation and, at the completion of the initiative, to better understand the impact of practice transformation on staff interactions.

DISCUSSION

PCMH researchers have noted that a comprehensive evaluation must be developed to understand and explain not only how and why practices transform but also how stakeholders experience the transformation.^19,23 Enhanced insights are needed so the field can develop recommendations for practice facilitation that best achieves the goals of the PCMH model. Mixed methods approaches that involve diverse stakeholders may be best suited to revealing these complex insights. Taken together, findings from each of these “different ways of knowing”⁵³ help explain the findings from the others and lead to conclusions that would not be reached by any single approach.^54–57

Often missing in PCMH evaluation is the analysis of the context and process of practice transformation efforts—data that help explain why anticipated changes were or were not achieved.^35,36 Comprehensive use of mixed methods evaluations can harness the power of different types of inquiry and resulting data. Surveys capture only part of the story, and individuals in practices may be too embedded within their cultural milieu to describe accurately the transformation processes, facilitators, and barriers. Similarly, evaluating performance through the electronic health record data alone may misrepresent the actual quality of care provided by the practice.⁵⁸ Despite our attempt to design an evaluation set that can be feasibly implemented, however, limitations may include lack of staff trained to implement the multiple methods, the burden of data collection, and the unpredictable consequences of change. The contextually comprehensive approach we describe may provide insights into some of the more fundamental changes that are needed to drive transformation toward the joy and enhanced quality outcomes and satisfaction that many are hoping to foster in the health care experienced by patients, clinicians, and staff.⁵⁹