Context Matters: The Experience of 14 Research Teams in Systematically Reporting Contextual Factors Important for Practice Change

Abstract

PURPOSE

We aimed to advance the internal and external validity of research by sharing our empirical experience and recommendations for systematically reporting contextual factors.

METHODS

Fourteen teams conducting research on primary care practice transformation retrospectively considered contextual factors important to interpreting their findings (internal validity) and transporting or reinventing their findings in other settings/situations (external validity). Each team provided a table or list of important contextual factors and interpretive text included as appendices to the articles in this supplement. Team members identified the most important contextual factors for their studies. We grouped the findings thematically and developed recommendations for reporting context.

RESULTS

The most important contextual factors sorted into 5 domains: (1) the practice setting, (2) the larger organization, (3) the external environment, (4) implementation pathway, and (5) the motivation for implementation. To understand context, investigators recommend (1) engaging diverse perspectives and data sources, (2) considering multiple levels, (3) evaluating history and evolution over time, (4) looking at formal and informal systems and culture, and (5) assessing the (often nonlinear) interactions between contextual factors and both the process and outcome of studies. We include a template with tabular and interpretive elements to help study teams engage research participants in reporting relevant context.

CONCLUSIONS

These findings demonstrate the feasibility and potential utility of identifying and reporting contextual factors. Involving diverse stakeholders in assessing context at multiple stages of the research process, examining their association with outcomes, and consistently reporting critical contextual factors are important challenges for a field interested in improving the internal and external validity and impact of health care research.

INTRODUCTION

Context is important for most phenomena of health care and health.^1–9 But contextual factors rarely are recorded, analyzed, or included in research reports.⁷ As a result, attempts to replicate research often fail, because context important to interpreting and applying findings remains unknown. Efforts to translate research into practice often fail because contextual factors important for understanding and knowledgably synthesizing findings across studies in meta-analyses and evidence-based guidelines are not known.¹⁰ As a recent analysis noted, “Investigation of how intervention effects are modified by context is a new methodological frontier in community intervention trial research.”¹¹

Although prior research has attempted to identify contextual factors based on theory,^8,11,12 technical expert panels,^6,12 reviews of (often decontextualized) published research,^{2,7,9,10,12,13} or surveys of individuals participating in quality improvement projects,¹⁴ we had a unique opportunity to develop and empirically apply an approach to identifying and reporting contextual factors across 14 studies of primary care practice change funded by the Agency for Healthcare Research and Quality (AHRQ). We attempted to consider and prioritize the most important contextual elements from a wide range of factors that could affect the interpretation and transportability of findings.

Specifics of contextual factors identified in these studies are reported in appendixes to each article in this supplement.¹⁵ Here, we summarize major contextual factors identified across these studies and make recommendations for reporting context affecting the interpretation, implementation, and dissemination of research.

METHODS

In the summer of 2010, AHRQ awarded 14 grants to evaluate successful “natural experiments” with potential to inform the process by which primary care practices transform into patient-centered medical homes (PCMHs). McNellis and colleagues¹⁶ present an overview of each project. The projects included a wide range of community, health care, and payer systems; practice types; and patient populations; they also had considerable diversity in the natural experiments they evaluated.

Grantees agreed to apply a template for context reporting, recently proposed by Stange and Glasgow,¹⁷ as their projects were nearing completion. They involved as many stakeholders as feasible in identifying contextual factors most important to understanding what happened and why in their studies; what others would want to know when transporting study findings to different settings, times, and situations; domains of potentially important contextual factors; and up to 500 words of explanatory text.

For this article, a subset of authors (A.T-C., D.L.S., N.W.) reviewed the 14 articles and completed templates, and identified themes by iteratively grouping similar reported factors. Subsequently, each research team determined 3 or 4 groups of contextual factors they judged to be most important for understanding or transporting study findings. The same subset of authors reviewed these contextual factors and refined emergent themes. Themes were verified and further refined through member checks with at least 1 representative from participating teams (11 of 14 teams participated in the verification). The subset of authors identified quotations to illustrate each theme and depicted themes in a contextual framework. In addition, each team submitted a list of tips for reporting contextual factors that were thematically organized by 2 authors (P.F.C., J.R.H.). This process informed the development of a refined template for reporting context.

RESULTS

Framework of Contextual Factors

The framework that emerged from our analysis classified contextual factors into 5 domains: 3 pertaining to specific levels and 2 themes that cut across levels, as depicted in Table 1 and Figure 1.

Table 1.

Themes and Examples of Contextual Factors

Domain	Description	Examples
Level specific
Practice	Characteristics that describe the clinic or practice setting and that are directly related to experiences of patients and staff	Employee mix (eg, number of physicians and midlevel professionals of certain specialties; part time/full time; residents; staffing mix) Clinician demographics, attitude, and training (eg, family medicine) Patient panel size and characteristics (eg, SES, ethnic minority composition, payer mix: uninsured, Medicaid, private, etc) Recognition type (eg, NCQA, other, none) Ownership of practices (eg, physician owned, independent, group practice) Structural capabilities (eg, EHRs, financial management system, QI system with ability to provide feedback, linkage to hospital) Leadership style (eg, visionary, strong personality, facilitative)
Larger organization	Factors related to the larger organization (if any) with which an individual practice is associated	Ownership, structural capabilities, and leadership style also may relate to this level Competing priorities: what else was going on in the organization, and when? (eg, EHR implementation midway, financial crisis, financial health of practices, applying for recognition) Degree of integration (eg, insurance, hospital system, specialty care, primary care, and AMC) Contractual arrangements (eg, public payers, private payers, payment model such as extent of capitation)
External environment	The health care system, policy, and community milieu relevant to the project	Market environment (eg, competition) Community characteristics (eg, SES, level of urbanization, availability of transport) Political authority Grant or other external financial support Level of coordination/involvement with community (eg, statewide project, multipayer project, collaborative, advocacy) Payment model(s) available (eg, fee for service, capitated, pay for performance, care management payment)
Cross-cutting theme
Implementation pathway	The specific elements and processes of an intervention, including operational changes and feedback loops	What elements of the PCMH were implemented; when, and over what period of time? What were the specific operational changes? (eg, expanded medical assistant role, building team communication, new employees hired; use of specific QI method, involvement of staff and patients in QI, feedback on performance to clinicians) Formal program identity (eg, demonstration project; pilot project; organizational transformation) History (eg, experience with transformation, burnout, adaptive reserve) Provision of a safe place to experiment and even fail Patient involvement in development Intervention group (eg, specific disease; specific demographic subpopulation) Assistance received (eg, external consultants, internal practice coaches; employee site visits; collaborative participation) Main intervention objectives and outcomes (eg, health status, patient satisfaction, financial stability)
Motivation for implementation	Key personal, organizational, and cultural drivers of change at multiple levels	Patient experience, quality, cost of care Incentives, NCQA/other recognition, marketing

AMC = academic medical center; EHR = electronic health record; NCQA = National Committee for Quality Assurance; PCMH = patient-centered medical home; QI=quality improvement; SES=socioeconomic status.

Figure 1.

Understanding an intervention’s context: emergent contextual framework.

The 3 levels of context highlight the interrelated and sometimes hierarchical nature of contextual factors. Level 1, the practice level, includes practice characteristics and factors directly related to experiences of patients and clinicians. Facets such as clinician and employee mix, staff demographics and training, patient panel size and characteristics, and structural capabilities define this level. Level 2, the larger organizational level, captures factors such as leadership structure and character, degree and nature of formal integration of care processes, and contractual arrangements, including payment models, that help shape the practice environment. Level 3, the external environment, denotes factors outside of the practice/organization, such as political authority, market structure, external sources of financing, and coordination with the community.

The 2 cross-cutting themes highlight important contextual processes not readily discernible in these levels. One theme, implementation pathways, captures locally relevant elements of an intervention, including operational changes (eg, addition of new employees, redefined roles, team communication strategies, feedback loops) as well as objectives of the intervention and outcomes (eg, health status of targeted populations or populations, patient satisfaction, and financial stability). The other theme, motivation for implementation, allows researchers to identify key drivers of an intervention including an advocacy-driven mission or other goals such as enhanced patient experience, quality and cost of care, and incentives that motivate implementation of the intervention.

To illustrate the framework, we provide examples and representative quotations below.

Levels of Context

At the practice level, training of practice members, characteristics affecting how they work together, and leadership are often relevant contextual factors. One team noted that their “practice included primary and mental health care providers…the colocation of services made it easier to integrate care.” Another team highlighted the importance of strong “visionary leaders among their practice leads (usually a lead provider)” and “the presence of boundary spanners at all levels within the practice helped sustain the work of transformation and diffuse information.”

At the organizational level, membership in a larger network often has implications for initiatives to bring about practice change. For example, the fact that one study was conducted in a network of 17 Federally Qualified Health Centers (FQHCs) was important because the “FQHC care model incorporates many elements of PCMH, so the concepts were not new to the staff.”

At the external environment level, factors pertaining to the political milieu, market forces, and community resources all may come into play. For example, for some projects, the availability of external financial support was viewed as critical. As one team noted: “The PCMH transformation initiative was supported by the 3 largest commercial insurers and the 3 Medicaid managed care organizations in the region. Practices received quarterly lump-sum payments based on the size of their practices.” For other projects, the built environment was important. According to another team: “The different geographic settings presented different needs (eg, transportation is an issue for patients in rural areas).”

Cross-Cutting Themes

Operational changes, and their sequence of implementation during the project, were important. In one study:

New members of the care team were hired (eg, diabetes educators, expansion of social work staff) and tasks were assigned to others (eg, depression screening and monofilament testing conducted by nurses, outreach team contacting patients about concerning clinical measures).

Another team observed: “A stepwise implementation process (over time), built on an existing [electronic health record]; started with appropriate access, then added the care team, and then the planned care components.”

The motivation for implementing the intervention also was seen as critical. “Many of the practices wanted to ‘do what’s best for their patients’; improving clinical care was a main driver of the initiative and a source of motivation for all levels of stakeholders.”

Brief Examples

The examples below, and the appendixes of each article in this supplement,¹⁵ illustrate the importance of contextual factors identified by our method. Typically, these factors are only hinted at in research reporting.

Example 1

A study of 120 certified medical homes in Minnesota found extensive variation among them in performance measures (SD = 30%) and presence of PCMH practice systems (SD=20%).¹⁸ To understand the external and internal validity of this finding, it is important to know that 75% of these clinics were part of large medical groups (having ≥21 clinics). The practices would therefore have more resources for change and be under pressure to be similar in function and performance within each large group. In addition, to become certified, clinics had to meet standards for access, registry use, care coordination, care plans, and continuous improvement, which were included in the systems. Finally, the standards and certification process were rigorous enough that only 20% of primary care practices in the state applied for certification. These contextual factors should have produced much smaller variation, so the study findings suggest it would be a mistake to expect that all PCMHs are meaningfully similar or will have similar results.

Example 2

The North Carolina Improving Performance in Practice initiative¹⁹ described an external quality improvement environment that included key stakeholders from Medicaid case management, an Area Health Education Center–led practice support program with medical society backing, and a major private payer reimbursing for higher-quality care. As part of the larger organization driving change, coaches helped practices implement key improvement elements. At the practice level, top leaders facilitated changes by providing vision and then delegating operational authority to a middle manager to carry out activities. Cross-cutting motivating factors included an economic recession period that motivated practice members to look for alternative revenue streams. When trying to transport this intervention, it would be important to align practice change with available external incentives, such as Maintenance of Certification for medical boards, meaningful use, and insurer support. It also would be important to identify practices that value population health and consider the availability of practice coaching to keep practices on track when making office system changes.

Example 3

The Group Health Cooperative project,²⁰ set in a system that integrates insurance and care delivery for a defined population, has many baseline organizational features that may not be present in other settings. At Group Health, physicians have historically been paid by salary rather than according to fee for service or capitation. Patients are empaneled to primary care physicians who work in teams of physician assistants, registered nurses, clinical pharmacists, licensed practical nurses, and medical assistants. Primary care teams are supported by a “medical neighborhood” that includes a 24-hour consulting nurse service, case management programs, and 10 specialty and behavioral health units, 6 urgent care centers, and 7 acute care hospitals (6 contracted and 1 maternity hospital). Although clinics are part of an integrated system clinic, systemwide and local leadership still had considerable impact on the way key elements were rolled out and overall willingness to change in each clinic.

During the study period, several important changes affected the transformation process, including substantial (19%) growth of the enrolled patient population between 2007 and 2011, and concomitant growth in the number of employees, creating unanticipated and uneven challenges for training, team development, and patient paneling. Further, in 2007, Group Health adopted the Lean Production System as its central method for quality improvement, change management, and reduction of waste. Unlike other PCMH demonstration projects that rely on practice facilitation or coaching, Lean methods emphasize use of front-line team members in process improvement and problem solving. The study period also featured other care innovations and changes in Group Health’s integrated system that may have influenced study results, including initiatives on standardized opioid prescribing, high-end imaging, hospital transitions, and urgent care expansion.

Several of these changes intersect with contextual factors categories, such as the growth of enrollment, which started with changes in the environment (local markets) but had subsequent implications at both the organizational and practice levels. The use of Lean was an implementation pathway, but also had important organization- and practice-level impacts.

Example 4

In a state-led medical home initiative involving 152 primary care practices in 7 regions across Pennsylvania, an analysis of 25 diverse adult medicine practices focused initially on improving diabetes care.²¹ Important contextual factors included (1) the study’s PCMH transformation collaborative developed from an existing statewide collaborative exploring care of patients with chronic conditions (adults with diabetes and children with asthma) and overlapped with other health care reform initiatives; (2) research network practices participated in a learning collaborative with practice facilitator support; and (3) the initiative had multipayer support, which was unique to this region. Building on an existing initiative (the Chronic Care Initiative) provided a sense of partnership and common purpose, and a common condition (diabetes) around which to focus PCMH development; however, sites had difficulty separating the 2 initiatives, with the least engaged stakeholders (medical assistants and front-office staff) often conflating them.

The overlapping, but at times conflicting, goals and objectives of ongoing system initiatives complicated program implementation. The use of a learning collaborative to support regional PCMH transformation provided a common structure and expectations for the PCMH initiative. Each practice site received financial support for a care manager and had a process focus of identifying high-risk patients for outreach and better service engagement. Early on, the collaborative provided on-site practice coaches with in-person meetings for participating sites. Later, practice coaches were replaced with telephonic practice support. Practices provided monthly performance reports to the collaborative, which created a common set of expectations. Finally, the PCMH transformation initiative was supported by the 3 largest commercial insurers and 3 Medicaid managed care organizations in the region. Practices received quarterly lump-sum payments that were based on the size of their practices (clinician full-time equivalents) and largely on attainment of 2008 NCQA PCMH recognition. Without a comprehensive incentive model, it is unlikely that a breadth of practices would have participated.

Tips and a Template for Assessing and Reporting Context

Table 2 shows tips for reporting contextual factors in research based on our collective experience. Figure 2 is the refined worksheet or template for assessing contextual factors. The instructions provide information on whom to engage in the process and when and how to collect and analyze the information. The template output consists of a table of bullet points and accompanying explanatory text that can be used as a source document for reporting contextual factors in a variety of ways appropriate to the audience.

Table 2.

Tips for Assessing and Reporting Contextual Factors

Engage diverse perspectives

Research participants (organizations, patients and clinicians, investigative team)

Relevant theoretical models

Prior research

Potential end users of study findings

Consider multiple levels

From the macro to the micro

Interactions between levels

Evaluate the evolution of contextual factors over time

Initial conditions and history

Changes over the course of the study

Look at both formal and informal systems and culture

Peer across the boundaries

Look for (mis)alignments

Be sensitive to the locus of power

Appraise internal and external motivations

Evaluate resources, support, and financial and other incentives

Assess (often nonlinear) interactions between contextual factors and both the process and outcome of studies

Report within the body of scientific articles key contextual factors that others would need to know (1) to understand what happened in the study and why, and (2) to be able to transport and knowledgeably reinvent the project in another situation

Figure 2.

Context Matters worksheet.

DISCUSSION

Divorcing a phenomenon from its context often is seen as a way of being objective and scientific; however, the scientific method is about systematic observation and hypothesis testing. For complex phenomena in health care and health, this systematic scientific way of generating new knowledge is enriched and made whole by considering context.

The contextual factors reported in the articles in this supplement, and the overarching lessons drawn here, show that it is feasible to identify and convey relevant information that goes well beyond the usual decontextualized way of reporting research. This is a systematic way to consider the facets of a study’s context that are essential to understanding what happened during implementation and to making reasonable judgments in transporting the findings to a new context. Reporting rich contextual information is a way of understanding contextual factors not as threats to external validity, but as opportunities to maximize learning that enrich the usefulness of systematic reviews and provide information needed to foster learning organizations^22–25 and health care systems.²⁶

This robust work was grounded in the individual and shared experiences of research teams working on a common area of investigation in diverse settings. The studies used varied methods, and many of them used qualitative methods specifically designed to incorporate relevant context from multiple perspectives.

A limitation of our analysis is that we may not have captured all domains of relevant context. The largely retrospective nature of our reporting, which undoubtedly led to underreporting of some factors, and the commonality in the research question funded by AHRQ through a single request for applications mean that other kinds of research may find different domains of context to be important. Grantees’ use of qualitative methods may underrepresent the challenges of paying attention to and reporting context in other quantitative study designs that are intended to filter out contextual factors. Importantly, this work did not prospectively evaluate the association between identified contextual factors and important process and outcome measures. Formally linking context with process and outcomes remains an important area for further methods development. These limitations, as well as the study strengths, helped to inform our recommendations for the field.

Other empirical research and theoretical models inform the approach reported here. Kaplan and colleagues¹² developed a Model for Understanding Success in Quality (MUSIQ) that identifies 25 contextual factors likely to influence quality improvement success. In an exploratory survey of individuals participating in 74 quality improvement projects, they found an association of these contextual factors with at least 1 quality improvement outcome.¹⁴ Others have focused on the importance of contextual factors in implementing evidence-based guidelines,^27,28 understanding complex interventions,^11,29,30 and considering health and health care as complex systems.^31–36

In addition, a number of health and health care models provide important clues to domains of contextual factors. For example, the chronic care model^37–39 and the expanded chronic care model⁴⁰ show specific community and health care system domains that may be important influences on productive clinician-patient interactions. The consolidated framework for advancing implementation science⁴¹ identifies intervention characteristics, outer and inner setting factors, characteristics of stakeholders, and processes that may be important for intervention implementation. The Conceptual Model for Considering the Determinants of Diffusion, Dissemination, and Implementation of Innovations in Health Service Delivery and Organizations provides a framework for considering the interaction of micro and macro contextual factors.⁴²

Glasgow et al,^43,44 Green et al,^45,46 and others^13,47 have suggested that the research community develop standards and expand the emphasis on external validity, to balance the current strong focus on internal validity. This process includes identifying and reporting relevant contextual factors needed to generalize, or at least knowledgeably reinvent research findings in other settings, situations, and times.

The richness of contextual information is not readily amenable to a checklist approach to researching and reporting, and contextual factors are likely to be interconnected.⁴⁸ The approach reported here,¹⁶ however, shows that context reporting can begin with considering domains of potentially relevant contextual factors prompted by a multilevel grid/list, and then engaging multiple perspectives in deciding which specific factors are likely to be most important. Using both qualitative and quantitative methods to systematically assess these factors and how they change during the course of a study, linking them to outcomes, and then reporting them has great potential to more rapidly advance knowledge generation and inform implementation and health improvement compared with current decontextualized approaches to research and dissemination. We encourage investigators to consider potentially important contextual factors when first focusing a research question and designing a study, and to continue to assess these factors as they evolve throughout a study. We believe that funders should give extra consideration to research applications that pay attention to context over those that focus solely on internal validity, and that journals should encourage reporting of relevant context and make needed publication space available.

Individual research reports could be made more useful by reporting key contextual factors in structured abstracts and describing them in sufficient detail in the body of the article so that the reader can interpret their effect on internal and external validity of findings. Different contextual factors could be included in different sections of an article. For example, factors known at the study outset may best fit in the background or in an enhanced methods section, in the description of the setting or intervention. Contextual factors identified during the course of a study or that change over time may best fit in the results section. The most important contextual factors deserve interpretation in the discussion section.

Reporting relevant context would make meta-analyses and other forms of systematic reviews more useful by helping to make sense out of the heterogeneity of effects. This practice would lead to more useful interpretive syntheses that go beyond the common conclusion in reports that “the evidence is insufficient” and “more research is needed.”

For now, we encourage others to use, adapt, and expand on the context reporting template presented in this article, and to enhance the value, validity, and transportability of research by placing it into context.