The role of organizational learning and resilience for change in building quality improvement capacity in primary care

Abstract

Background:

The extent that organizational learning and resilience for the change process, i.e. adaptive reserve, is a component of building practice capacity for continuous quality improvement is unknown.

Purpose:

To examine the association of adaptive reserve and development of quality improvement capacity.

Methodology:

142 primary care practices were evaluated at baseline and 12 months in a randomized trial to improve care quality. Practice adaptive reserve (AR) was measured by staff survey along with a validated quality improvement capacity assessment (QICA). We assessed the association of baseline QICA with baseline AR, and both baseline and change in AR with change in QICA from 0-12 months. Effect modification by presence of QI infrastructure in parent organizations and trial arm were examined.

Results:

Mean QICA increased from 6.5 to 8.1 (P <0.001) and mean AR increased from 71.8 to 73.9 points (P <0.001). At baseline, there was a significant association between AR and QICA scores: the QICA averaged 0.34 points higher (95% CI 0.04-0.64, P =0.03) per 10-point difference in AR. There was a significant association between baseline AR and 12-month QICA – which averaged 0.30 points higher (95% CI 0.02-0.57, P =0.04) per 10-points in baseline AR. There was no association between changes in AR and the QICA from 0-12 months, and no effect modification by trial arm or external QI infrastructure.

Conclusions:

Baseline AR was positively associated with both baseline and follow-up QI capacity, but there was no association between change in AR and change in the QICA, suggesting AR may be a precondition to growth in QI capacity.

Practice Implications:

Findings suggest that developing AR may be a valuable step prior to undertaking QI-oriented growth, with implications for sequencing of development strategies, including added gain in quality improvement capacity development from building AR prior to engaging in transformation efforts.

Introduction:

Expectations for primary care have surpassed the traditional practice model in past decades (Bodenheimer et al., 2014; Wagner, 2000). Aligning primary care with the Triple Aim requires an ability to build and sustain continuous improvement in quality and safety of care (Berwick et al., 2008; James & Savitz, 2011; Luxford et al., 2011; Wang et al., 2018), also known as quality improvement (QI) capacity (Taylor et al., 2013). QI capacity is a property of a practice and its staff encompassing the abilities and knowledge to improve quality, including having sufficient staff numbers and roles, experience, and skills, and infrastructure such as data systems to enable practices to conduct effective QI (Mery et al., 2017; Parchman, Anderson, et al., 2019). While efforts to improve care delivery have transformed small- to medium-sized practices (Crabtree et al., 2011; Nutting et al., 2011), there remains variety in the type and implementation of externally-supported and internal QI strategies (Balasubramanian et al., 2018; Shea et al., 2018). Thus, for organizations and leadership seeking to actively scale or standardize these strategies, little is known about how these practices, which deliver over half of primary care (Rui & Okeyode, 2017), can actively build QI capacity or which components are required for effective change.

We sought to further understand these gaps in knowledge by examining how organizational learning could be associated with practice QI capacity, within a randomized trial of external practice support strategies to improve quality of cardiovascular care within small and medium-sized primary care practices in the northwest U.S.

Theory:

Organizational learning theory provides insight into how practices might develop and learn the capability for continuous improvement (Berta et al., 2015). Capacity for change contains two components – external and internal capacity (Miller et al., 2010). External capacity refers to practice alignment and interaction with a larger health system. Internal capacity includes a practice’s resources, structure, and functional processes. Growth in internal capacity depends on resilience for the change process (that is, a practice’s ability to weather and withstand the stresses of transformation) and the ability for deeper understanding about the learning process itself – so called higher-order learning – within an uncertain environment like healthcare (Berta et al., 2015). These qualities are known as a practice’s adaptive reserve (AR), which broadly reflects the practice’s organizational learning and development abilities as a function of culture, trust, staff relationships and facilitative leadership (Jaén et al., 2010; Miller et al., 2010). Low AR can lead to “change fatigue,” (Nutting et al., 2009) and AR has been inversely related to burnout and emotional exhaustion (Blechter et al., 2018; Willard-Grace et al., 2014). AR can also improve during transformation efforts (Nutting et al., 2010).

Continuous practice improvement requires the structural and functional components of internal capacity, attention to the local environment, and AR (Miller et al., 2010; Nutting et al., 2010). Practices with strong AR may be more able to engage and respond to the challenge of transformation and actively reform the tools and structures required for improved care delivery. Implementation of QI strategies interact with domains of the AR, such as staff relationships, reflection, and interdependent learning (Arar et al., 2011; Lanham et al., 2016). Furthermore, successful growth in prior practice transformation efforts have been restricted within practices potentially lacking in AR – such as those with a physician-centric or authoritarian leadership structure (Nutting et al., 2011). In the National Demonstration Project (NDP) for development of patient-centered medical homes, practices with higher AR implemented more components of the change intervention (Crabtree et al., 2010). Despite the potential importance of AR, a practice’s QI capacity growth could also be assisted by outside support strategies, such as practice facilitation, or by the environmental context, such as the presence of a parent organization with QI infrastructure. For example in the NDP, AR developed in practices who received facilitation, but did not in those that only received shared visits (Crabtree et al., 2010). Establishing if AR is a precondition, and if so to what extent, to engaging in QI capacity building is operationally relevant for healthcare leadership or organizations wanting to improve care delivery.

In this analysis, we examined the extent that AR is associated with QI capacity. We also examined for modification either by a) intervention arm of the randomized trial of external support strategies or b) presence of a parent organization with QI infrastructure.

Method:

Study design

Healthy Hearts Northwest (H2N) was one of seven EvidenceNOW collaboratives funded by the Agency for Healthcare Research and Quality to build QI capacity in primary care focused on cardiovascular risk factors. The study protocol and primary outcomes have been previously described (Parchman, Dorr, et al., 2019; Parchman et al., 2016). Briefly, H2N was a randomized trial of external practice support strategies in small- and medium-sized primary care practices in Washington, Oregon, and Idaho. Four study arms compared strategies for developing QI capacity; practice facilitation (PF) alone, PF with shared learning between practices, PF with educational outreach by experts in decision-support tool implementation, or PF with both strategies. External facilitators worked with all practices. Study measures were completed at baseline between January and August of 2016 and approximately 12 months later (median 11.0 months; range 4.8 – 23.1). H2N was approved by Kaiser Permanente Washington Health Research Institute’s Institutional Review Board.

Measures: adaptive reserve

Adaptive reserve was measured through staff surveys completed at baseline and 12 months. The 19-item survey used in H2N was based on a previously validated 14-item survey (Jaén et al., 2010) with four questions added from the original 23-item survey and one question added for H2N addressing psychologic safety (Nembhard & Edmondson, 2006) (full survey available in Supplemental Digital Content Table e1). Responses were on a 1-5 Likert scale, with higher scores indicating better AR, with a possible total score from 19-95 points. Consistent with prior analyses of the AR (Nutting et al., 2010), responses were summed, then averaged at the practice level as AR is conceptually and functionally thought to represent a single latent construct within a practice (Jaén et al., 2010; Miller et al., 2010; Nutting et al., 2010). Survey responses were confidential. All practice staff were invited to participate via email link, with responses aggregated and disclosed to clinics only if participation reached 50% of clinic staff. Practices received a $50 gift card for response rates of 50% or greater.

Measures: quality improvement capacity assessment

A practice’s state of QI-related structures and abilities was evaluated using a practice evaluation tool, the QI capacity assessment (QICA). Previous work has described the tool and validated the correlation between practice scores on the QICA and scores on clinical quality metrics (Parchman, Anderson, et al., 2019). The QICA has also been shown to moderately correlate with a practice’s score on the Change Process Capacity Questionnaire, a measure of change prioritization not specific to quality improvement processes (Spearman correlation 0.351) (Parchman, Anderson, et al., 2019; Solberg et al., 2008). One QICA was completed per practice at baseline and 12-month follow-up during a visit with a practice facilitator through a discussion of each item and staff consensus on a response. The QICA is an adaptation of the previously validated Patient-Centered Medical Home Assessment (PCMH-A) (Daniel et al., 2013) as a shorter, more pragmatic tool to guide practices and facilitators during QI capacity-building activities (Parchman et al., 2017). To create the 20-item QICA, expert reviewers with substantial experience in practice transformation mapped 19 items from the PCMH-A, plus an additional item, onto 7 domains perceived as related to QI capacity: embedded clinical evidence; data utilization to improve performance; establishing a regular QI process; identifying at-risk patients; defined roles and responsibilities; improving patient self-management support; and linking patients to outside resources (Parchman, Anderson, et al., 2019). Each domain was assessed by 1 to 4 items measured on a 12-point scale with written descriptions demarking a level ranging from D (point values of 1-3) to A (10-12), with higher scores indicating better QI capacity. As an example, a low-scoring practice in the domain of data utilization might have scored itself as a 2 (Level D) if performance measures “are not available” while a high-scoring practice might score a 10 (Level A) if measures “are comprehensive, including clinical, operational, and patient experience measures – and fed back to individual providers”. Item scores were averaged to give a total score ranging from 1-12 points.

Aims

Our analysis had two aims. The first aim was examining the association of the baseline QICA with baseline total AR: We hypothesized these measures would be positivity associated. (hypothesis 1). The second aim was to examine if AR was a prerequisite for change in the QICA. We hypothesized that baseline AR would be positively associated with change in the QICA (hypothesis 2), and that change in AR would predict change in the QICA between 0-12 months (hypothesis 3).

Covariates

Practice characteristics were considered a priori as potential sources of variation between the relationship of AR and the QICA, including rural versus urban setting, ownership (independent, health/hospital system, Federally Qualified Health Center, or Indian Health Service/Tribal), and size (1, 2-5, or 6-10 providers). Location was included as Washington/Idaho versus Oregon (as corresponding to coverage of the two facilitator organizations) (Parchman, Dorr, et al., 2019). AR survey respondent role was captured as proportion of practice survey respondents as physicians for the baseline survey, and as clinicians at the follow-up survey given changes in role reporting options at follow-up. Time between QICA surveys was also examined as a potential covariate, but ultimately excluded from final models given absence of difference in effect size or statistical significance.

Statistical analysis

We began with descriptive analyses of practices and staff respondents. Time between QICA surveys was explored for association with both predictor and outcome of interest using linear regression. A priori, all other practice characteristics were included in adjusted models. Incomplete surveys were excluded. Multivariate linear regression was used for three primary models. Our first model examined the association between practice AR and the QICA score at baseline. Second, we examined the association between practice AR at baseline and the difference in the QICA – measured as the QICA at 12 months after adjusting for the baseline QICA score (consistent with ANCOVA approach for increased power, acceptable in randomized trials) (Van Breukelen, 2006). The third model examined the association between the delta of the AR (12-month minus baseline score) and the difference in QICA, measured as the QICA at 12 months adjusting for the baseline QICA. This model also adjusted for baseline AR scores, so the results could be interpreted conditional upon similar AR practices. We used robust standard errors for coefficients. T-tests with unequal variance compared the changes in descriptive measures as needed. All hypothesis testing was two-sided with an alpha of 0.05. Analyses were performed on R 3.5.0 (www.r-project.org).

We performed effect modification analyses by examining the significance of an interaction term between the outcome and exposure of interest. First, we examined the effect across trial arm by adding an interaction term between trial arm and either the baseline or change in AR. As participation in the trial differed from randomization (Parchman, Dorr, et al., 2019), the number of visits with practice facilitators was also examined as a separate interaction. For our second modification analysis applicable to the change in AR and change in QICA score, we examined for interaction by practice external capacity (i.e. if the practice was part of a larger organization with central QI capacity).

As secondary analyses, we examined if the change in the QICA depended on a threshold baseline AR score. For this, we divided practices into quantiles of baseline AR, then repeated the regression for the association of the delta in AR (0 to 12 months) on the change in the QICA as the 12-month QICA score adjusting for baseline QICA, for the top- and bottom-scoring 25% of practices by baseline AR. We also assessed if respondent role impacted findings, but excluded this term due to lack of statistical significance.

Results:

A total of 1,891 staff responses from 187 practices comprised the baseline cohort (Supplemental Digital Content Figure e1). There was a 67.5% response rate with a mean of 10.1 (range 0-66) baseline surveys returned per practice. Participating practices were mostly small practices of 2-5 providers (54%) that were either independently owned (44%) or part of a health or hospital system (40%). Almost half (46%) reported some presence of a centralized QI support (Table 1). Among practices that completed both baseline and follow-up assessments (n = 142), the average baseline AR was 71.8 (SD = 7.1, range 51.4-95.0) with a QICA of 6.5 (SD = 1.4, range 3.3-10.2). At 12 months, the average AR increased to 73.9 (SD = 7.0, range 49.5-95.0, change from baseline P<0.001) with a QICA of 8.1 (SD = 1.3, range 4.2-11.2, change from baseline P<0.001). Of practices completing both assessments, on average, only 78 (55%) participated as randomized in the facilitation strategies. Characteristics of excluded practices are shown in Supplemental Digital Content Table e2.

Table 1.

Practice and staff characteristics at baseline.

	All Practices n (%), except as noted
	N = 187
Adaptive reserve, clinic mean (SD)	71.3 (7.7)
QICA clinic mean (SD)a	6.6 (1.5)
Survey respondent (n = 1,891)
Physician (MD/ND/DO)	252 (13.3)
NP or PA	129 (6.8)
Clinical staff	753 (39.8)
Non-clinical staff	572 (30.2)
Other or unknown	185 (9.8)
Practice ownership
Independent	83 (44.4)
Health system	75 (40.1)
FQHC	20 (10.7)
Tribal/Indian Health	9 (4.8)
Practice in Oregon	85 (45.5)
Rural designation	85 (45.5)
Group size
Solo (1 provider)	33 (17.6)
Small (2-5 providers)	101 (54.0)
Medium (6+ providers)	53 (28.3)
Presence of centralized QI team	86 (46.0)
Intervention arm randomization, (%)
Facilitation only	43 (23.0)
+ Shared learning (SL)	48 (25.7)
+ Education outreach (EOV)	48 (25.7)
+ SL and EOV	48 (25.7)

^aQICA = Quality improvement capacity assessment.

After adjustment for covariates, there was a significant association between baseline AR and the baseline QICA: The QICA was on average 0.34 points higher (95% CI 0.04-0.64, P=0.03) for every 10-point change in practice level AR (Table 2). There was no difference by trial arm (P=0.56).

Table 2.

Hypotheses (H) for association of clinic-level QI capacity assessment (QICA) and adaptive reserve (AR) in participating practices, between baseline (0 months) and follow-up (12 months).

	Practices	Change in QICA per 10-point change in AR	95% CI	P-value
H1: Baseline QICA is associated with baseline AR	187
Unadjusted		0.37	0.08 – 0.65	0.011
Adjusteda		0.34	0.04 – 0.64	0.026
H2: Change in QICA is associated with baseline AR	154
Unadjusted		0.33	0.07 – 0.58	0.012
Adjustedb		0.30	0.02 – 0.57	0.038
H3: Change in QICA is associated with change in AR	142
Unadjusted		0.32	0.02 – 0.62	0.039
Adjustedc		0.29	−0.03 – 0.61	0.080

^aAdjusted for state of practice, rural locality, practice ownership, and group size.

^bAdjusted for baseline QICA score, state of practice, rural locality, practice ownership, and group size.

^cAdjusted for baseline QICA score, baseline AR score, state of practice, rural locality, practice ownership, and group size.

Examination of change over time in the QICA (0-12 months) showed a significant association with baseline AR, but not with change in AR (Table 2). After adjustment for covariates, the 12-month QICA was on average 0.30 points higher for every 10 points greater (95% CI 0.02-0.57, P=0.04) baseline AR for practices with the same initial baseline QICA score. Finally, we found no significant association between 12-month QICA score and change in AR from 0 to 12 months (P=0.08, Figure 1). There was no difference with inclusion of study arm or number of facilitator visits (both P>0.05) in either model.

Figure 1.

Hypothesized relationships between adaptive reserve (AR) and quality improvement capacity assessment (QICA).

Secondary analyses of practices scoring in the top quartile of baseline AR (average 76.4-95.0 points) versus bottom quartile (51.4-67.3) showed no association between change in QICA and difference in AR from 0 to 12 months (all P>0.05). There was no evidence of effect modification by a centralized QI component for the practice (P=0.34).

Discussion:

To our knowledge, this is the first examination of the association of a measure of practice organizational learning, the adaptive reserve, with capacity for continuous quality improvement. Within 187 practices enrolled in a randomized trial to improve the quality of cardiovascular care within the northwest U.S., practices began with a medium- to low-level of QI capacity (6.5 out of 12 points on the QICA) and a wide range of AR scores, consistent with varying abilities to develop and learn as organizations (Nutting et al., 2010). During the study, both AR and QICA scores increased. We found these two measures were related but did not consistently change together. Practice baseline AR was positively associated with a higher baseline QICA and with a higher QICA at 12 months, concordant with our first two hypotheses. However, there was no association between change in AR and change in QICA from 0-12 months (hypothesis three).

Practice transformation is a complex, dynamic process that requires aligned leadership, functional structures, and the opportunity and motivation to change (Crabtree et al., 2011). This process does not take place in a vacuum; interaction with dynamic local environments also matters. In our study, this included diversity of patient population served and exposure to QI efforts aside from H2N (U.S. Centers for Medicare & Medicaid Services, n.d.). This heterogenous backdrop emphasizes the importance of a resilient practice that can respond and adapt – precisely where AR is valuable. Similar to successful changes seen in the NDP among practices with high AR, we found the baseline AR was positively associated with change over time in the QICA.

Our findings suggest the mechanisms behind growth in AR may be independent of those behind growth in practice QI capacity. This is not entirely surprising. The QICA relates to activities and processes, and does not capture concepts such as the influence of leadership on AR. A practice’s QI capacity encompasses infrastructure, strategies, and abilities which can be directly impacted by organization, prioritization, and investment – illustrated by the relationship of the QICA to change process capability and quality outcomes (Parchman, Dorr, et al., 2019; Solberg et al., 2008). AR, on the other hand, may be related to continuous, higher-order learning that encompass the ability to reflect on and incorporate new knowledge through growth in trust, reflection, and teamwork (Berta et al., 2015; Miller et al., 2010). While this ability does not necessarily mean an organization will accomplish a certain level of QI capability – it might be a precondition to do so (Noël et al., 2013). Growth in QI capacity may require additional elements – though not, as we thought, the presence of an external QI infrastructure. In other evaluations within EvidenceNOW, achievement of high quality process outcomes was not directly related to a practice’s AR – underscoring that the relationship may be contingent on other mechanisms (such as QI capacity) (Henderson et al., 2018).

As the H2N trial focused on developing QI capacity, rather than team-based organizational learning, some practices would be expected to have changes in the QICA without an increase in AR, which occurred in around 1 in 4 practices. Inversely, those practices that increased their AR only were of even greater interest to us. We wondered if practices with low AR required an investment in team-based learning to be ready to develop their QI capacity – perhaps being smaller practices that were more physician-centric at onset, given previously demonstrated challenges in these types of practices to achieve successful transformation due to barriers such as restricted communication or lack of facilitative leadership (Nutting et al., 2012). This implies a minimum threshold of AR to develop QI capacity, though not perhaps as clearly a linear relationship as we had explored in the secondary analysis of practices in the top and bottom quartile of AR. Indeed, of the 19 practices in the top 50% for improvement of AR but without change in their QICA score (<1.0 point difference), most were independently owned (58%) small or solo practices (68%) with a baseline AR of 67.1 – significantly lower than the average for all 142 practices (71.8; P<0.03).

One mechanism of growth common to both measures could be practice facilitation itself, aligned with findings from the NDP where AR improved in facilitated practices (Crabtree et al., 2010) As facilitation was present in all our practices, we could not determine its specific effect, but there were no differences between intervention arms in any of our analyses. Facilitation could improve practice AR through the process of guided problem-solving. Under a supportive leadership structure, this could lead to staff engagement, relationship building, and empowerment – characteristics captured in the AR. While facilitation might also lead to growth in the QICA, this could be related the accomplishment of specific functional capabilities (e.g. data use processes).

Our study has a number of limitations. First, the QICA was developed specifically for use in H2N to guide facilitators and clinics in their work to build QI capacity. While it has been previously validated (Parchman, Anderson, et al., 2019), further studies on its performance and characteristics (such as examinations of the domains) in other settings would be beneficial. Secondly, our AR survey was adapted slightly from that used in previous work, though drawn from validated sources (Nembhard & Edmondson, 2006; Nutting et al., 2010). Our AR had a Cronbach’s alpha of 0.95, consistent with retained allegiance to an underlying construct. While outside the scope of this study, further research would be beneficial to expand the understanding of meaningful differences for both the AR and QICA. Third, we recognize outside efforts like the national Million Hearts study (U.S. Centers for Medicare & Medicaid Services, n.d.) could have influenced the QICA (though less likely AR). However, as our aims differed from those in the larger H2N trial and were related instead to the interaction of the two scales, we would not expect this to affect the interpretation of our results. Finally, our study may have limited generalizability beyond small- to medium-sized primary care practices similar to those included in our study. There were differences in practices retained in the H2N trial from those that were not included (Supplemental Table e2). However, at the practice level, we found no differences when assessing participation engagement by number of encounters with the facilitator. In the staff survey, while our pattern of respondents was similar to previous work (Parchman et al., 2013), we acknowledge the possibility of nonresponse bias, although this is less concerning as responses are believed to capture an underlying unified construct (Daniel et al., 2013).

Practice Implications:

In summary, we found an association between a measure of organizational learning and resilience for change and quality improvement capacity in small- to medium-sized primary care practices. While baseline AR was positively associated with both initial and follow-up QI capacity, we did not find dependent change in both measures – suggesting AR as a precondition to efforts to change QI capacity. While more research is needed, this interpretation implies greater gains could occur from developing AR as a preliminary activity to QI efforts, for example by team-building, communication training, or developing knowledge management processes (Berta et al., 2015; Jaén et al., 2010; Lanham et al., 2009). This has practice implications for healthcare management and organizational leadership seeking to engage in practice transformation around quality improvement capacity growth. While more research is necessary, this suggests interventions targeting AR first would produce greater returns in subsequent practice transformation efforts.