Positive and Negative Spillovers of the Health Disparities Collaboratives in Federally Qualified Health Centers

Abstract

Introduction

Quality improvement (QI) interventions are usually evaluated for their intended effect; little is known about whether they generate significant positive or negative spillovers.

Methods

We mailed a 39-item self-administered survey to the 1256 staff at 135 federally qualified health centers (FQHC) implementing the Health Disparities Collaboratives (HDC), a large-scale QI collaborative intervention. We asked about the extent to which the HDC yielded improvements or detriments beyond its condition(s) of focus, particularly for non-HDC aspects of patient care and FQHC function.

Results

Response rate was 68.7%. The HDC was perceived to improve non-HDC patient care and general FQHC functioning more often than it was regarded as diminishing them. In all, 45% of respondents indicated that the HDC improved the quality of care for chronic conditions not being emphasized by the HDC; 5% responded that the HDC diminished that quality. Seventy-five percent stated that the HDC improved care provided to patients with multiple chronic conditions; 4% signified that the HDC diminished it. Fifty-five percent of respondents indicated that the HDC improved their FQHC’s ability to move patients through their center, and 80% indicated that the HDC improved their FQHC’s QI plan as a whole; 8% and 2% indicated that the HDC diminished these, respectively.

Discussion

On balance, the HDC was perceived to yield more positive spillovers than negative ones. This QI intervention appears to have generated effects beyond its condition of focus; QI’s unintended effects should be included in evaluations to develop a better understanding of QI’s net impact.

Quality improvement (QI) innovations tend to be evaluated for their intended effects and are appraised less frequently for whether they generate positive spillovers or undesired negative effects.¹ Because some QI programs purport to transform how practices approach and solve quality problems, some hypothesize that QI techniques can spread, thereby yielding benefits beyond their initial clinical targets.² QI interventions, however, can require substantial investments from physician and nonphysician staff, and may lead to negative spillovers by diverting attention and resources away from non-QI tasks or activities that are also important to patient outcomes.^3–7

Only a handful of studies specifically evaluate QI interventions for positive or negative spillovers. These studies find support for certain positive spillovers. Physicians who are engaged in QI activities tend to report less professional isolation, stress, and dissatisfaction, and QI programs can also ameliorate staff burnout and promote positive morale.^8,9 This literature finds neutral or mixed support for whether QI generates unintended negative effects. One evaluation of a QI program that improved aspects of geriatric care (eg, treatment for falls, incontinence, and cognitive impairment) found no change in processes of care not being emphasized with clinicians.¹⁰ Another QI program improved evidence-based processes for treating alcohol withdrawal, but also ascertained that related hospital length of stay and mortality increased.¹¹

Because both positive and negative spillovers probably occur as a result of QI interventions, one must assess the degree to which these occur, assess whether certain types of organizations are more likely to experience one over the other, and better understand the mechanisms by which benefits and detriments occur so that positive effects can be promoted and negative ones mitigated.

Since 1998, federally qualified health centers (FQHC) have participated in the Health Disparities Collaboratives (HDC), a large-scale collaborative QI intervention sponsored by the Health Resources and Services Administration’s Bureau of Primary Health Care (HRSA’s BPHC).¹² At the time, the HDC was the largest, longest, and most substantial national commitment to the collaborative QI model. Evaluations of the HDC have found this intervention to be effective at improving processes of care for asthma and diabetes.^13,14 Studies of the HDC show that the HDC brings new costs to health centers⁴ at the same time it is cost-effective from a societal perspective.⁵

It is unknown whether the HDC has had a positive or negative impact on clinical care or organizational function beyond the conditions of focus. Our aims are to understand the degree to which FQHC personnel involved in the HDC perceived positive or negative changes in their non-HDC clinical care or organizational function, may have applied the techniques they learned from the HDC to non-HDC conditions, or found that the HDC diverted attention and resources away from non-HDC FQHC programs and efforts. Additionally, we assess whether FQHCs’ experience with the HDC differed depending on FQHC size or patient case-mix because these may promote positive spillovers or buffer against negative unintended consequences.

METHODS

Study Setting

Approximately 1200 FQHCs care for more than 20 million patients annually.^15,16 As previously detailed, the HDC was a QI intervention that brought FQHCs together in collaborative sessions to learn rapid-cycle QI techniques using the Plan-Do-Study-Act (PDSA) paradigm and the Chronic Care Model.¹³ When the HDC began in 1998, all participating FQHCs focused on improving the quality of diabetes care. By 2005, FQHCs chose from several different HDCs whose clinical targets included diabetes, asthma, depression, cardiovascular disease, and others.^15,16

The HDC required FQHCs to form multidisciplinary teams with team leaders and members drawn from all positions within the FQHC including senior leaders (eg, chief executive officers), frontline providers (eg, doctors and nurses), and support staff (eg, receptionists). Although all team members were responsible for implementing the HDC, leaders championed and coordinated the team’s efforts.

In 2005, virtually all 161 FQHCs in the Midwest and West Central regions of HRSA’s BPHC (states spanning Montana in the northwest, Arizona in the southwest, Louisiana in the southeast, and Ohio in the northeast) had participated in at least 1 HDC for 2 or more years. We included 135 FQHCs in our study; we excluded 10 that declined participation and 16 that could not provide full lists of their HDC teams.

Data Sources

Survey

We hypothesized that FQHC personnel could view the impact of the HDC on non-HDC patient care and organizational function as positive or negative, depending on the degree to which the HDC introduced new skills or demands to the FQHC. For example, if the HDC taught FQHCs to use patient registries and quality of care measures, and they applied these techniques to the care of patients with non-HDC conditions (eg, developed a hypertension registry based on experience with a diabetes registry), then FQHC personnel could view the HDC as improving the quality of non-HDC care. Conversely, if the emphasis on HDC-related quality measures was such that staff felt it distracted from their ability to focus on non-HDC care, then FQHC personnel could view the HDC as having a negative impact on non-HDC care.

We developed our self-administered questionnaire using the above conceptual model and a series of semi-structured interviews with 50 HDC team leaders and members at 8 different FQHCs. During these interviews, we asked informants to describe how they adapted the HDC to non-HDC situations, trade-offs they faced, and sequelae they experienced or observed. We coded responses, identified major themes, developed survey items, pretested questions among personnel outside of the target population, and finalized the survey. During this process, we noted that FQHC personnel had to have direct knowledge of the HDC to make observations, positive or negative, about the HDC. Being a team leader or member, therefore became an inclusion criterion for the study.

Our survey comprised following 4 main domains: (1) Spread (ie, use of HDC techniques and strategies for non-HDC patients, conditions, and situations); (2) Diversion (ie, extent to which the HDC effort drew time, energy, or resources away from other FQHC activities); (3) Non-HDC patient care (ie, quality of clinical care for patients with non-HDC conditions or clinical situations); and (4) Non-HDC organizational function, (ie, ability to perform non-HDC health center operations). Respondents were given the option of rating the HDC’s impact as being positive or negative using a 5-point Likert-like scale ranging from “Greatly decreased” to “Greatly increased”; midpoint was neutral “Has not changed.” The remainder of the questions pertained to informants’ positions within their FQHC, their role within the HDC effort, and demographic information (see Appendix, online only, Supplemental Digital Content available at: http://links.lww.com/MLR/A120).

We targeted all 1256 personnel identified as team leaders or members at the 135 eligible FQHCs irrespective of the number or type of HDCs in which they were engaged. In all, 75% had participated in a single HDC, whereas the remainder had engaged in 2 or more HDCs; 75% were using the HDC to improve diabetes care, whereas the remainder was split evenly between those targeting depression and cardiovascular disease care. We sampled all team leaders and members irrespective of the position within the FQHC they occupied; each was asked to complete every question within the survey.

We collected surveys between October 2005 and April 2006, a period when FQHCs were actively engaged in the HDC, following the standards of Dillman Total Design Method.¹⁷ This included mailing the survey to all eligible respondents in 3 rounds via regular and express mail, and supplementing with reminder telephone calls and letters of support by relevant HRSA’s BPHC officials.

We linked our survey data to HRSA’s 2005 Uniform Data System to obtain information on FQHC (eg, number of delivery sites) and patient population (eg, poverty level) characteristics.

Analysis

We used survey and Uniform Data System data to describe the characteristics of respondents and FQHCs that participated in our survey, and to compare respondent FQHCs to nonrespondent ones within the Midwest and West Central regions of HRSA’s BPHC and nonrespondent FQHCs nationally.

For our main analysis, we used the individual respondent as the unit-of-analysis and described the proportion of respondents identifying a particular response option for each of our individual survey questions in the 4 domains using simple descriptive statistics. Prior to doing so, we explored whether responses differed significantly depending on informant type and unit-of-analysis, recognizing that our respondents were nested within their FQHC.¹⁸ In these analyses, we found that responses to our questions did not differ significantly when respondents had different roles in the HDC (ie, team leaders vs. members) or held different positions within the FQHC (eg, senior leaders vs. frontline providers). We also found that respondent-level responses did not differ from mean response levels at each FHQC. Thus, we did not stratify responses by informant type or adjust responses for clustering, and annotated significant differences where appropriate.

For our second aim, we examined whether FQHC size or case-mix influenced the reporting of positive or negative spillovers. We transformed our Likert-like response variable into a dichotomous outcome variable that separated those who had noted that a particular aspect of patient care or organizational function had “greatly increased” from those who had not. Dichotomizing each question at this point was appropriate because most of our findings were in favor of identifying positive spillovers. We then used logistic regression to examine whether our main predictors—the number of delivery sites and the ratio of number of patients to number of delivery sites, the percent of the patient population that was Hispanic, Black, non-English speaking, living below the federal poverty line, uninsured, or insured by Medicaid—was associated with these outcomes, adjusting for clustering by FQHCs.

We assessed validity in a subset of 15 FQHCs that responded to this survey and participated in a prior chart-based assessment of the HDC, which examined the quality of diabetes care in 1998 and 2004. Using the following available measures that study examined whether in a single year FQHCs performed: HbA1c tests at least once, HbA1c twice or more 3 months apart, lipid tests once, foot and eye examinations once, aspirin and angiotensin-converting enzyme (ACE) inhibitor prescriptions. It also examined whether patients with diabetes had low-density lipoprotein levels less than 130 mg/dL, blood pressure less than 130/80 mm Hg, or HbA1c less that 9.5%. We compared the change over time in these measures to staff’s survey item responses that asked about change in “the quality of care provided by my FQHC for patients with chronic condition(s) emphasized by the HDC effort.” Because quality changes at these 15 FQHCs were nearly exclusively rated as increased (only 1 FQHC indicated “no change”), we dichotomized this perceived quality increase as “greatly” versus “somewhat” increased. We then fit separate multilevel logistic regression models for each chart audit quality measure. In each model, the patient level chart audit outcome was the dependent variable and year, perceived quality increase, and the year by perceived quality increase interaction were treated as fixed effects. A statistically significant, positive coefficient for the interaction term was indicative of a positive correlation between respondent perceptions of increased quality- and chart-based measures of quality. These multilevel models included FQHC and the year by FQHC interaction as random effects to incorporate the nesting of patients within FQHCs in 2 different years in the chart audit design. We report the chart-audit measures for which the association with the survey rating is significant.¹³

We used STATA version 10 (College Station, TX) for all analyses. This study was approved by the Institutional Review Boards of the University of Chicago and National Opinion Research Center.

RESULTS

The informant response rate was 68.7% (863 of 1256); at least 1 person responded from 132 of the 135 FQHCs targeted such that the FQHC response rate was 98%. Three FQHCs closed partway through the study and are not included; 2 of these were due to Hurricane Katrina. Item nonresponse ranged from 1.6% to 4.9% for all items except for 2 questions related to the PDSA cycle, which were 6.6% and 6.7%, respectively.

Respondent and FQHC Characteristics

The majority of survey respondents were female (74%), and of White racial/ethnic background (73%) (Table 1). On average, they were about 40 years of age, had worked at their health center for 8 years in a full-time capacity, and in their current position for 6 years. Respondents reported being in their current HDC role for 3 years and spending about 9 hours per week exclusively engaged in HDC work. Of total, 24% of respondents had been or were HDC team leaders, while 76% were team members. Fifty-seven percent of respondents were front line providers, 14% occupied the role of a Senior Leader while 24% comprised support staff.

TABLE 1.

Respondent Characteristics Obtained From Survey

Respondent
Female gender (%)	74
Race/ethnicity (%)
White (non-Hispanic)	73
African American (non-Hispanic)	11
Hispanic	12
Asian/Pacific Islander	3
American Indian/Alaska native	0.3
Mixed/other	2
Age in years (mean ± SD)	40 ± 10
Experience in years (mean ± SD)
At health center	8 ± 7
In current position in health center	6 ± 6
In collaborative role	3 ± 2
Current position in health center (%)
Senior leaders
Chief executive, financial, or operating officers	8
Medical directors	6
Frontline providers
Physicians	23
Nurse practitioners or physician assistants	13
Medical assistants or certified nursing assistants	21
Support staff
Nursing coordinators or administrators	10
Care managers, health educators, promotors	6
Receptionists, medical records, or data entry clerks	6
Dieticians, therapists, pharmacists, or dentists	4
Collaborative role (%)
Team leader	24
Team member	76
Hours per week at working at health center	40 ± 10
Hours per week exclusively engaged in collaborative work	9 ± 12

In all, 75% of the FQHCs had HDCs focused on diabetes, 21% on cardiovascular care, 14% on depression care, 11% on asthma care, and 6% on cancer screening (Table 2). Forty-three percent of the patients served by these FQHCs were White, 30% were Hispanic, 20% were Black, and 2% were either Asian or American Indian. Patients tended to be female (59%), relatively young (64% less than 40 years of age), living at or below the federal poverty level (54%), and uninsured (56%). Twenty percent were noted as “best served in a language other than English.”

TABLE 2.

Health Center Characteristics Obtained From the Uniform Data System

Health Center
No. delivery health centers (mean, range)	6, 1–29
Patients (%)	43*
Race/ethnicity	30†
White	20†
Hispanic	1
Black	1
Asian	5
American Indian	20
Unreported/refused to report	59
Best served in language other than English	41
Female gender
Adults (20 yr and older)
Patient income as the percent of poverty level (%)	54†
100% and below	12
101%–150% of the poverty level	6
151%–200% or more of the poverty level	7
201% and more of the poverty level	22
Unknown
Principal third party insurance source (%)	15
Private	12
Medicare	16
Medicaid (regular or CHIP)	1
Other public insurance	56
Uninsured

^*Respondent federally qualified health centers (FQHCs) cared for a significantly higher proportion of patients with Hispanic backgrounds (30% vs. 22% in the rest of the Midwest and West Central regions).

^†Respondent FQHCs cared for a significantly lower proportion of patients with Asian backgrounds, best served in language other than English, unknown income levels, and Medicaid (1% vs. 4%, 25% vs. 20%, 22% vs. 29%, 16% vs. 23% outside the Midwest and West Central regions, respectively).

Respondent FQHCs were similar to all FQHCs nationally in all above respects except that they cared for a significantly higher proportion of Hispanic patients; they also cared for a significantly smaller percent of patients who were Asian, best served in a language other than English, who had unknown income levels, and Medicaid insurance.

Impact of the HDC on HDC and Non-HDC Patient Care

Most of respondents indicated that the HDC somewhat or greatly increased the quality of care provided to patients with the chronic condition emphasized by their FQHC’s HDC effort (86%) (Table 3). Nearly half of respondents indicated that the HDC increased the quality of care delivered to patients with chronic conditions not emphasized by the HDC (45%), particularly for patients with routine screening needs (65%). Respondents also indicated that the HDC increased the quality of care delivered for potentially life-threatening conditions (40%). A majority of respondents signified that the HDC increased the ability of FQHCs to manage patients with multiple chronic conditions during a routine office visit (76%), to manage those with chronic conditions during an acute care visit (67%), or who only use the clinic on a walk-in basis (40%). Most respondents reported that the HDC had not changed the quality of care provided to patients with acute, likely benign, conditions (71%).

TABLE 3.

Impact of the Health Disparities Collaboratives (HDC) on HDC and Non-HDC Patient Care (%)

	Greatly Decreased	Somewhat Decreased	Has Not Changed	Somewhat Increased	Greatly Increased
As a Result of the HDC, The quality of care provided by my Federally Qualified Health Center (FQHC) for patients with
1. Chronic condition(s) emphasized by the HDC effort	1	4	10	48	38
2. Chronic conditions not emphasized by the HDC effort	1	4	50	41	4
3. Routine screening needs (eg pap smears, colonoscopy, mammogram)	1	1	33	48	17
4. Acute, potentially life-threatening conditions (eg suspected heart attack, severe bacterial infections)	1	4	55	30	10
5. Acute, likely benign conditions (eg sore throat)	0	2	71	22	5
Their FQHC’s ability to manage
6. Patients with multiple chronic conditions during a routine office visit	1	3	21	51	24
7. Patients with chronic conditions during an acute care visit	0	2	30	52	15
8. Patients who only use the clinic on a walk-in basis	0	5	54	31	9

Impact of the HDC on Non-HDC Organizational Function

Overall, our respondents perceived the HDC to benefit FQHC operations (Table 4). A clear majority indicated that the HDC somewhat or greatly increased their FQHC’s ability to improve its QI plan as a whole (80%), maintain ongoing relationships with their patient populations (76%), and work as a team (76%). They also rated the HDC’s impact on the ability to move patients through the FQHC or improve “no show” rates as being moderately positive. Over half of respondents (55%) noted that the HDC increased their ability to move patients through their FQHC, and over one-third (38%) reported that the “no show” rates had improved. The majority of respondents indicated that the HDC increased their personal job satisfaction (63%), but did not change their ability to attract (69%) or retain (69%) high-caliber employees; approximately a quarter indicated that these abilities were increased.

TABLE 4.

Impact of the Health Disparities Collaboratives (HDC) on Non-HDC Organizational Function (%)

	Greatly Decreased	Somewhat Decreased	Has Not Changed	Somewhat Increased	Greatly Increased
As a result of the HDC, my Federally Qualified Health Center (FQHC)’s ability to
1. Improve its quality improvement plan as a whole	1	1	19	61	19
2. Work as a team	0	1	24	61	15
3. Maintain ongoing relationships with our patient population	1	2	22	57	19
4. Move patients through the health center	1	7	38	45	10
5. Improve “no show” rates for appointments	1	3	58	33	5
6. Improve personal job satisfaction	1	6	30	43	20
7. Retain high-caliber employees	1	6	69	21	4
8. Attract new high-caliber employees	1	4	69	22	4

Spread of HDC Techniques to Non-HDC Patients, Conditions and Situations

Overall, about half of respondents reported that the HDC had not changed their use of QI techniques for non-HDC conditions and activities whereas the other half indicated that they had increased their use of these tactics (Table 5). A majority signified that the HDC somewhat or greatly increased their use of process or outcome measures to gauge performance for non-HDC conditions (56%). Approximately half indicated that their FQHC increased its use of rapid PDSA cycles for non-HDC conditions (48%), and nearly half reported increased use of rapid PDSA cycles for nonclinical FQHC activities (43%), for patient registries to track clinical care for non-HDC conditions (43%), and for systems or personnel to link patients with non-HDC conditions to outside resources (45%). Two percent of respondents reported that these activities diminished.

TABLE 5.

Spread of Health Disparities Collaboratives (HDC) Techniques to Non-HDC Patients, Conditions and Situations (%)

	Greatly Decreased	Somewhat Decreased	Has Not Changed	Somewhat Increased	Greatly Increased
As a result of the HDC, our Federally Qualified Health Center (FQHC)’s use of
1. Process or outcome measures to gauge performance for non-HDC conditions	0	2	42	47	9
2. Use of rapid Plan-Do-Study-Act cycles for non-HDC conditions	0	2	50	40	8
3. Use of rapid Plan-Do-Study-Act cycles for non-clinical health center activities	0	2	55	35	8
4. Patient registries to track clinical care for non-HDC conditions	0	2	52	38	5
5. Systems or personnel to link patients with non-HDC conditions to outside resources	0	2	56	37	8

Diversion of Time, Energy, and Resources From Non-HDC Patients Activities

When asked to indicate the degree to which the HDC effort drew time, energy, and resources away from other FQHC activities, 40% of respondents indicated that the HDC drew away “none at all” or “a little,” 35% indicated that the HDC drew away “a moderate amount” of resources, and the remaining 25% indicated that it drew away “quite a bit.” None indicated that it drew away “a great deal” (Table 6). In terms of the amount of time that providers spent with patients, respondents indicated that the HDC generally did not change the amount of time providers spent on non-HDC issues with HDC patients (61%) or non-HDC patients in general (72%), although a small proportion (3%–10%) noted that the providers decreased the amount of time they spent on non-HDC patients or issues. Although most respondents signified that the HDC did not change their FQHC’s ability to manage its non-HDC programs (63%) or to take on new non-Collaborative programs (53%), a significant minority reported these abilities as being decreased (13% and 20%, respectively).

TABLE 6.

Diversion of Time, Energy and Resources From Non-Health Disparities Collaboratives Activities (%)

	Not at All	A Little	A Moderate Amount	Quite a Bit	A Great Deal
As a result of the HDC
1. The degree to which the HDC effort has drawn time, energy, and resources away from other Federally Qualified Health Center (FQHC) activities increased	11	29	35	9	7
	Greatly Increased	Somewhat Increased	Has Not Changed	Somewhat Decreased	Greatly Decreased
As a result of the HDC*
The amount of time providers spend on	4	26	61	9	1
2. Non-HDC issues with HDC patients	3	22	72	3	0
3. Non- HDC patients in general
Our FQHC’s ability to
4. Manage its non-Collaborative programs	2	23	63	12	1
5. Take on new non-collaborative programs	4	25	53	17	3

^*Responses oriented from lesser to greater degrees of diversion.

Impact of FQHC Size and Patient Case-Mix on Positive Spillovers

In general, neither FQHC size nor case-mix had a significant impact on whether respondents rated the quality of non-HDC patient care and non-HDC organizational function as “greatly increased.” There was 1 main exception to this finding. For each 10% increase in proportion of uninsured patients, there was a 30% to 70% (P = 0.008 –0.04) reduction in the odds of rating as “greatly increased” the quality of care delivered to patients with acute potentially life-threatening conditions, the ability to manage patient with multiple chronic conditions, the ability to maintain ongoing relationships with patients, and personal job satisfaction (Data not shown).

Validation of Respondent Perceptions with Chart Abstraction. We found a substantial correlation between survey-reported and chart-based improvements in the quality of diabetes care based on the subset of 15 FQHCs for which we had both survey and chart audit information. Overall, 87% of the 129 informants of this 15 FQHC subset marked that quality increased. This corresponded to the chart abstraction findings that patients with diabetes at these FQHCs had a greater odds of receiving HbA1c and lipid testing, foot and eye examinations, and aspirin and ACE inhibitors prescriptions in 2004 than in 1998 (odds ratios ranged between 2.2 and 6.7, all P < 0.001). Additionally, the 4 FQHCs that rated their quality of care as “greatly improved” versus those that did not were more likely than other centers to perform lipid testing, prescribe ACE inhibitors, and have patients with blood pressures <130/80 mm Hg in 2004 when compared with 1998 (P < 0.05).

DISCUSSION

Little is known about whether QI interventions yield significant benefits or harms beyond the diseases and processes on which they explicitly focus, but there is evidence supporting the existence of both positive and negative spill-overs. Three key challenges face this line of inquiry. The first is identifying the assortment of changes that can result from these interventions; the second is ascertaining whether positive and negative spillovers occur; the third is then determining the degree of clinical or organizational impact. Thus, the field would benefit from an assessment that simultaneously addresses the variety, direction, and degree of potentialities.

This study shows that personnel involved in deploying the HDC in a large number of FQHCs viewed the HDC as generating both positive and negative spillovers. Respondents perceived more spillover benefits than detriments and viewed these benefits as accruing to patients and their FQHCs. Staff in FQHCs that served greater proportions of uninsured patients were less likely to perceive some of these positive spillovers.

The response rate to our study was excellent. Our study is limited, however, because it relies on whether informants perceived positive or negative unintended consequences, not whether they really occurred. Although direct assessment of these consequences would be preferable, there is a wide array of possible effects resulting from QI interventions. It is understandable if investigators limit themselves to certain kinds of spillovers (eg, cost consequences from the FQHC perspective) or measures (eg, those that represent the care that is not being emphasized). However, as an alternative tactic, we used FQHC personnel as informants to characterize the existence and nature of positive and negative spillovers. Staff perceptions are important both as possible indicators of spillovers, and in their own right—negative perceptions of QI interventions may compromise the program itself. Additionally, we supplemented survey-based assessments with chart information on quality of diabetes care. Although we were not able to validate every outcome in the survey, the diabetes chart review provides objective support for the validity of staff perceptions. This study was, therefore, an important step in identifying consequences that could be associated with a multidimensional QI intervention.

Our study is limited by whom we surveyed and social desirability bias because our survey sample had to be limited to those with enough experience with the HDC to answer our questions. On the basis of qualitative interviews, we conducted to develop the survey and observed that those who were not sufficiently involved with the HDC effort at their FQHC could not understand questions and concepts of spread and diversion. Our study is probably also limited by memory errors whereby informants may not recognize or recall if spillovers occurred.¹⁹ All together, this potentially overestimates reports of positive consequences and underestimates negative ones, because it may be difficult for those so deeply involved with the HDC to see or acknowledge negative aspects of the intervention. As a result of this potential bias, it may be important to temper reports of positive spillovers and to take any reports of negative consequences very seriously.

In summary, this study is an important step in evaluating the nature and significance of positive and negative spillovers associated with large-scale collaborative QI efforts. Future evaluations of QI interventions should also aim to estimate the net benefit or harm of QI to patients and the organizations that host them.