Community Health Center Quality Improvement: A Systematic Review and Future Directions for Research

Abstract

Background

Leaders and policymakers need an accurate appraisal of the federally qualified community health center (CHC) quality improvement (QI) literature to make informed decisions for the CHC program.

Objectives

This paper aims to (1) summarize the content and findings of CHC QI studies to date, (2) systematically rate the quality of those studies, and (3) outline 10 important areas for future CHC QI research.

Methods

We searched medical and nonmedical databases to identify QI studies in CHC settings. We systematically reviewed identified studies for the features of their QI interventions and for the methodological quality of their evaluations. We combined results from the review with input from the CHC community to generate an agenda for future CHC QI research.

Results

Eighteen studies were identified and reviewed. Interventions mainly targeted chronic conditions and screening practices and used 1 to 11 of 14 different QI tactics; evaluations comprised 14 observational and 4 randomized study designs. CHC QI interventions have been effective in improving processes of care for diabetes and cancer screening in the short term; their effectiveness in the long term and regarding outcomes of care have not been demonstrated.

Conclusions

QI interventions in CHC setting are promising, but future interventions and evaluations should answer critical basic questions about QI, including the following: What are the best models of QI? How can QI improvements be effectively implemented and sustained? What are the global effects of QI (positive and negative)? How can QI be made financially viable and sensible from both the CHC and societal perspectives?

The federally qualified community CHC program is a growing federal program that provides primary health care to more than 15 million vulnerable people nationwide and has an annual budget of approximately $1.5 billion dollars.¹ Currently, the CHC program faces the challenge of continually improving its quality of care despite rising numbers of uninsured and underinsured people and a presidential mandate to increase CHC capacity and number. The program’s ability to meet this challenge is considered critically important to our nation’s ability to eliminate racial and ethnic disparities in health care.^2–9

Since the last major review, the literature on CHCs has shifted away from comparing CHCs to non−CHC settings and toward describing and evaluating their QI efforts.^10–24 Since 1998, numerous QI efforts have been described.^25–56 Among these is the Health Disparities Collaborative (HDC), a state−of−the−art QI effort sponsored by the Health Resources and Services Administration’s (HRSA) Bureau of Primary Health Care; this QI effort has been implemented in 647 CHC sites across the nation.²⁶ Moreover, study designs evaluating CHC QI interventions have increased in sophistication over the past 10 years; control groups and randomized methods are now being used.^{29,41,43,44,46,50}

Leaders and policymakers need to make decisions about CHC QI to ensure that the CHC program meets the ambitious goals laid before it. Their decisions should be informed by an accurate appraisal of the CHC QI improvement literature. This paper aims to provide that by (1)summarizing the content and findings of CHC QI studies to date, (2) systematically rating the quality of those studies, and (3) outlining 10 important areas related to future CHC QI research.

METHODS

We identified written descriptions of QI efforts attempted in CHC settings and then excluded studies that did not meet our specific inclusion criteria (Figure 1). To identify QI efforts published in the medical literature, we searched the electronic databases (Medline, CINAHL, CCTR, PsycInfo, Cochrane DSR, ACP Journal Club, DARE, and EBM Reviews; see Appendix A). To find efforts published in the nonmedical literature, we asked leaders within the National Association of Community Health Centers (NACHC) and the HRSA’s Bureau of Primary Health Care for references and/or internal documents, and searched the websites of private organizations that promote and/or fund QI efforts.

Figure 1. Quality improvement study identification process.

*QI = quality improvement, i.e. any effort directed at changing elements within an organization to increase the appropriate provision of health care; CHC setting = community health center setting, i.e. at least half of the total sites implementing the QI intervention were federally qualified health centers.

We included articles that (1) described QI interventions, (2) formally evaluated the stated intervention, and (3) adequately employed the CHC setting.^{28,30,34,36–38,40,42,47,49,51,56} To be considered QI, an intervention had to be directed at changing elements within an organization to increase the appropriate provision of health care. This definition is based on Donabedian’s classic “structure, process, outcome” approach to defining and improving quality.^57,58 To be considered a formal evaluation, the article had to present data to support or refute the effectiveness of the described intervention. To be considered a CHC setting, at least half of the total sites implementing the intervention had to be designated as federally qualified CHCs. We contacted authors for clarification if the nature of the intervention, evaluation, or setting were unclear. We excluded editorials, commentaries, non−English or non−United States articles, and duplicates.

After identifying CHC QI studies between January 1966 and December 2005, we systematically reviewed each included study using two readers (AC and AW). We identified the clinical purpose of the QI intervention and the QI tactics that were used. We then judged the methodological quality of the studies based on the validated Downs and Black tool and resolved disagreements by consensus⁵⁹ (see Appendix B). We generated a “top 10” list for CHC QI research by combining the results of the systematic review with our experience and the input of our community partners.

Community Partners

An initial manuscript and “top 10” list was refined with input from (1) key personnel at ACCESS Community Health Network, a network of 45 federally qualified CHCs in Chicago, Illinois, (2) staff at NACHC, HRSA, and the AHRQ, and (3) an invited reviewer from each of the following: Puget Sound Neighborhood Health Centers, the Institute for Healthcare Improvement, the National Center for Primary Care at Morehouse School of Medicine, the Department of Family Medicine at Case Western Reserve University, and The Commonwealth Fund.

A revised manuscript and “top 10” list were then presented to all participants of the “Health Centers and the Medically Underserved: Building a Research Agenda” conference during a plenary session.⁶⁰ The participants of this meeting comprised over more than clinical and administrative staff from federally qualified CHCs across the nation, as well as researchers and policymakers. During a breakout session, the “top 10” list was discussed in detail by a voluntary subset of approximately 35 participants. We edited the “top 10” list based on participants’ reactions and comments during the general and breakout sessions of this conference. Although this process included substantial input from the CHC community, the final “top 10” list are the views of the authors and are not necessarily the opinions of the NACHC, HRSA, AHRQ, or conference participants.

RESULTS

We identified 438 descriptions of QI in CHC settings; 31 were formal evaluations of QI interventions; 18 fully met our inclusion criteria for being conducted in a federally qualified CHC setting (see Figure 1).

QI Intervention Features

The QI interventions in these 18 studies targeted a total of 13 different clinical conditions; 4 interventions simultaneously addressed more than one clinical condition (Table 1). Five interventions targeted common chronic conditions (4 diabetes, 1 asthma), 11 interventions were aimed at screening practices for 7 different conditions or populations (5 cancer, 1 tuberculosis, 1 domestic violence, and 1 adolescent issues), 4 focused on immunization delivery (2 adult, 2 pediatric), and 3 addressed smoking.^{25,27,29,31–33,35,39,41, 43–46,48,50,52,54,55}

Table 1.

Quality Improvement Intervention Features

^•Consistently applied;

^*occasionally applied.

IT, information technology; QI, quality improvement.

The QI interventions used a total of 14 different QI tactics, which included gaining leadership buy−in, targeting provider and nonprovider staff with condition−specific education, generating provider reminders, implementing flow sheets/checklists for staff, creating patient registries or providing information technology support for CHCs, and giving verbal or written feedback to staff informally or as part of a formal collaborative process. QI tactics were aimed at a subset of CHC employees (providers, staff, patients) and/or the CHC as a whole. No QI intervention used all 14 of the described tactics, 5 employed 6 or more tactics, 13 used 5 or fewer tactics, and only 1 intervention used a single tactic. One intervention trained CHC staff on a specific QI methodology, Plan−Do−Study−Act (PDSA cycles); 2 allowed CHCs to choose their own tactics.^27,33,55 No intervention tied financial incentives to QI activities or measures, although 2 QI interventions provided lump−sum payments to offset the administrative costs of the intervention and 4 provided materials, supplies, and/or technical support, and 1 intervention lengthened the office visit duration.^{29,31,35,46,55}

Study designs and methodological Quality

All 18 studies used process measures or perception−based endpoints to assess the effectiveness of the QI intervention; 4 also assessed patients’ clinical outcomes (e.g., HgA1c values, smoking quit rates).^27,32,33,41 In terms of study designs, 14 studies used observational methods (10 cross−sectional and 4 with case−control or cohort features) and 4 used group randomization (Table 2). The methodological quality of these studies ranged from 7 to 16 on the Downs and Black scale (which ranges from 0 to 26 from lowest to highest quality); 3 studies scored between 15 and 19 points, 2 scored between 11 and 14 points, and 13 scored less than 10 points. The kappa agreement for these scores was 0.63. Most studies did not randomize subjects, and many did not account for nested and/or clustered designs in analysis. All but one study chose CHCs through a competitive process, existing relationships, or characteristics that were thought to predispose them to success.²⁹

Table 2.

Quality improvement Study Features, Quality, and Findings

	Study Design	Number of CHC Sites vs. Total Sites	Study Duration (mo)	Downs and Black Score (max = 26)	Significant Improvements	No Change
Diabetes care50	CC	3/3	26	8	—	Diabetes-related screening and referral rates
Diabetes care46	CC	2/2	36	9	—	Diabetes-related screening and referral rates
Diabetes care33	XSPP+L	4/5	30	9	Diabetes-related screening and referral rates	HgA1c and LDL-C values
Diabetes care27	XSPP+L	19/19	12	13	Diabetes-related screening and referral rates	HgA1c values
Cancer screening54	XSPP	1/1	36	10	Mammography rates	Clinical breast exam rates
Cancer screening43	R	5/8	12	16	Mammograms and fecal occult blood testing rates	Pap testing rate
Cancer screening44	R	5/8	24	16	—	Mammograms, fecal occult blood testing, and Pap testing rates
Cancer screening/smoking cessation52	XSPP	1/1	24	8	Screening and counseling rates at 5 months	Screening and counseling rates at 14 months
Cancer screening/ immunizations30,31	XSPP	3/5	39	10	Cholesterol screening, Pap smears, mammogram, smoking assessment rates	Influenza immunization rate
Immunization (hepatitis B)39	XS	1/1	30	7	—	Hepatitis B immunization status
Immunization (influenza)55	XSPP	2/2	12	9	Influenza immunization rate for 1 center	Influenza immunization rate for other center
Immunizations (childhood)45	XSPP	1/1	24	9	Opportunities for catch- up vaccination	—
Tuberculosis screening48	XSPP	2/2	7	8	Latent tuberculosis screening rate	—
Adolescent screening35	XSPP	5/5	26	9	Patient report of services received	Provider report of services given
Oral-pharyngeal cancer awareness25	XSPP	NS	3	10	Provider knowledge and perceived competence	—
Domestic violence screening32	XSPP	12/12	12	10	Perceived knowledge and screening rates	Violence documentation rates
Smoking cessation41	R	5/5	42	14	Pregnancy smoking rates	Postpartum smoking rates
Asthma care29	R	16/16	NS	16	Interval history and peak flow documentation	Controller prescription and action plan completion rates

CC, case-control features; L, longitudinal cohort features; NS, not specified; R, randomized trial; XS, cross-sectional; XSPP, cross-sectional pre- and postintervention design.

Findings of the QI Studies

The four QI interventions aimed at improving diabetes care significantly increased diabetes−related screening and referral rates, but not HgA1c levels despite variations in intervention intensity and study duration.^27,33,46,50 The five interventions targeting cancer screening significantly increased some screening rates, but improvements were not significant beyond 2 years in three of the studies.^{31,43,44,52,54} The three interventions involving smoking varied too widely to draw summary conclusions; two provided counseling but in very different patient populations; the other only assessed smoking status without requiring cessation counseling.^31,41,52 The four interventions related to immunizations were also too disparate to consider as a cohesive set; one targeted hepatitis B status in a Vietnamese pediatric population, another focused on missed vaccine opportunities in general pediatrics, and the remaining two worked toward improving adult influenza vaccine status.^39,45,55

QI interventions employing six or more tactics tended to be more successful that those that did not; all 5 of the QI interventions employing six or more tactics had statistically significant improvements whereas 9 of the 13 interventions employing five or fewer tactics did. Interventions that provided additional support (financial or technical) were more likely to succeed; 5 of the 7 interventions that provided additional support had statistically significant changes compared to the 5 of 11 interventions that did not.^{27,29,31,46,55} The 5 studies that scored 11 or better on the Downs and Black scale showed that process of care measures could significantly change in 1 year, although improvements in processes were not necessarily sustained or associated with changes in outcome measures.

SUMMARY

Conclusions about QI interventions in the CHC setting are difficult to make because QI interventions target numerous clinical conditions, employ a heterogeneous group of QI tactics, are supported financially and technically in different ways, and are mainly evaluated using observational study designs for various lengths of time. Still, published studies of CHC QI suggest that interventions for diabetes care and cancer screening have been effective; too few studies have been published regarding improving asthma care, immunization delivery, and smoking cessation counseling to make summary assessments.

Interventions that used six or more strategies or were supported financially or technically might have a better chance of succeeding. Research that helps to improve the science behind the QI may help us to better understand how to increase the effectiveness of these efforts.

The future of QI in CHCs looks promising, but several lines of questioning need to be pursued to maximize the effectiveness of these efforts. With the literature review as a starting point, we present a “top 10” research agenda that has been developed with the input of several community partners across the nation.

TOP 10 RESEARCH QUESTIONS FOR COMMUNITY HEALTH CENTER QUALITY IMPROVEMENT

1. What is the Best model (or Are the Best models) for QI in CHCs?

One of the most fundamental questions is what model or models are best for QI in CHCs. For example, CHCs employ a variety of community approaches that are not stressed by current models of QI (e.g., enabling services, lay health workers, and seasonal clinics). Although the HDC is the dominant QI model and includes community outreach as one of its components, its major emphasis is still on improving office−based ambulatory care and there may be ways to modify or develop the HDC into an even better model. Future research should include whether future QI models can better address the broader mechanisms by which CHCs improve the health and health care of their catchment population.

2. Can the Elements of Successful Multicomponent QI Interventions Be Prioritized?

Given evidence that more comprehensive QI programs may be more effective, it is important to determine whether the individual tactics of QI interventions work synergistically to comprise more than the sum of their parts. This question is important to answer because, if synergies are critical, then deconstructed or scaled−down versions of multicomponent QI interventions may not be as effective as full−scaled interventions. If synergies do not exist, then perhaps elements of multicomponent interventions can be prioritized and implemented in stages, thus making QI interventions more affordable.

3. How Should QI Interventions Be Tailored to Different CHCs?

CHCs differ tremendously in terms of size, budget, staffing, number of satellite clinics, services, and special populations served.¹ Even centers thought to be predisposed toward success drop out or do not improve, and centers that receive the same intervention do not perform the same.^{27,31,35,41,55} A greater understanding of CHC organizational stage of development or readiness for change could also aid in the design of tailored interventions, and perhaps enhance CHC ability to initiate and sustain QI programs. For example, certain QI methods may be more appropriate for mature CHCs because they are more financially stable, whereas others may be better in younger CHCs that may adopt changes more easily.³⁵ Research should identify which CHCs and QI interventions work together best.

4. How Can QI Interventions Improve the Overall Quality of Care?

To date, QI efforts in CHCs have focused on improving condition−specific care for certain subsets of patients. Providing support and incentives for identifying and servicing a small subset of a person’s needs may put that person’s other needs at risk for neglect. QI research should investigate methods for improving an individual’s overall health care, not just that within isolated dimensions. Broad metrics of quality need to supplement disease−specific items. For example, an office visit may be evaluated not only on whether or not immunizations were provided, but also by what proportion of issues raised by the patient were resolved by the provider in that office visit or a telephone call, or on the quality of shared decision making between the patient and provider, or overall quality−of−life or functional status.^61–65

5. How Can We Increase the Chances that QI Interventions Will Be Successfully Implemented?

No one knows the complete constellation of factors that are necessary for QI interventions to be successfully implemented. Tremendous variety exists among CHCs as well as QI interventions themselves, and interventions may need to be accompanied by additional funds and/or technical assistance. Future QI research should help to identify the true predictors of success (or combinations thereof) so that QI interventions can be reliably and successfully implemented.

6. What Is the Best Way to Sustain QI Activities?

Many of the studies in this review described interventions that performed well for a period of time and then waned.^{27,31,32,41,44,48} Potential reasons include personnel turnover, inability to institutionalize QI culture and processes, traditional concepts of productivity, and lack of reimbursement for improvement. We need to identify and test solutions for these and other roadblocks.

7. What Are the Unintended Consequences of QI?

Unintended consequences of QI need to be studied because an improved understanding may increase our ability to mitigate potential harms, and implement and sustain QI efforts. On the one hand, positive unintended consequences may occur when QI tools are adopted for situations or conditions that are not of the initial QI focus. Studies thus far have not reflected whether additional gains have been wrought off existing QI efforts. On the other hand, negative unintended consequences may occur if QI draws resources or attention away other aspects of patient care. The QI field should stop treating unintended consequences as merely a hypothetical concern, and begin testing their existence empirically.^66,67

8. What Incentives Should Be Used to Promote QI Activities in CHCs?

None of the identified studies described using performance incentives as part of their QI interventions, although some did provide grants or personnel to offset the costs of the intervention.^{27,29,31,46,55} Proponents of pay for performance argue that financial rewards for meeting quality benchmarks will support QI. This hypothesis has not been tested in CHCs, and it is unclear which financial and non−financial incentives may be most effective. It is critical to determine ways that performance incentives can be structured to improve care and outcomes while minimizing gaming and avoiding harming subsets of patients such as through dumping.

9. How Should Case-Mix Adjustments Be Used to Improve Assessments of QI Interventions?

Risk adjustment is a technique that facilitates fair comparisons between groups of patients when equivalence of study populations cannot be achieved through randomization.⁶⁸ This case−mix adjustment can offer more meaningful analysis to individual CHCs when they reflect on the care they provide to patient populations over time, and may help to compare CHCs more fairly to one another as well as non−CHC providers. It may help HRSA’s Bureau of Primary Health Care to differentiate CHCs struggling with especially risky patients from fundamentally poor performers, and avoid penalizing CHCs for caring for more vulnerable patients when they are compared with providers treating more advantaged patients.

10. What Are the Cost Implications of QI Interventions From Both the Business Case and Societal Vantage Points?

Most likely, QI interventions cost CHCs money in the short term. The question is whether these costs constitute worthwhile investments that pay health dividends in the long run. This analysis must be done from the perspective both of the CHC (the business case vantage point),⁶⁹ and society at large (the societal vantage point such as through societal cost−effectiveness analysis).⁷⁰ This is necessary because costs generated for QI make it more expensive for CHCs to provide care, but cost savings (e.g., prevented hospitalization from better ambulatory care) may accrue to another entity such as an insurance plan. QI interventions must be viable from the CHC’s perspective to be practically sustainable. The QI intervention is only likely to be worthwhile if it has societal value.

CONCLUSIONS

This paper summarizes QI studies in federally qualified CHCs. Future research should answer critical basic questions including what are the best models of QI, how to most effectively implement and sustain improvements, what are the global positive and negative effects of QI, and how to make such interventions financially viable and sensible from both the CHC and societal perspectives. We hope that leaders and policymakers will use this appraisal to make needed decisions about QI investments for the CHC program. Continued collaboration among CHCs, policymakers, and researchers ensures that QI efforts continue to offer the most innovative and effective strategies for delivering high−quality cost−effective health care for vulnerable populations.

APPENDIX A. Search Strategy for community health center Quality improvement efforts

I. Medical Literature Search.

We used the following terms and combinations in Medline^®, All EBM Reviews – Cochrane DSR, ACP Journal Club, DARE, CCTR, CINAHL (January 1966 to December 2005) to identify QI efforts in the medical literature.

Search terms	citations Reviewed
1. “Community Health Center” (MeSH) OR “community health center” (non-MeSH) or “health center”
Combined With
2. “Quality Assurance” (MeSH) OR “Total Quality Management” (MeSH) OR “Outcome Assessment” (MeSH) OR “continuous quality improvement” (non-MeSH) OR “cqi” (non-MeSH) OR “PDSA” (non-MeSH) OR “quality improvement”	153
3. “Health Planning Guidelines” (MeSH) OR “Guidelines” (MeSH) OR “Practice Guidelines” (MeSH) OR “Guidelines” (MeSH)	14
4. “Guideline adherence” (MeSH) OR “guideline adherence” (non-MeSH)	5
5. “outreach visits” (non-MeSH)	0
6. “multiple interventions” (non-MeSH)	1
7. “Intervention Studies” (MeSH) OR “interventions” (non-MeSH)	17
8. “opinion leaders” (non-MeSH)	0
9. “Reminder Systems” (MeSH) OR “reminders” (non-MeSH)	8
10. “audit” (non-MeSH)	43
11. “Computers” (MeSH) OR “computer” (non-MeSH)	5
12. “Health Education” (MeSH) OR “Education” (MeSH) OR “Patient Education” (MeSH) OR “education” (non-MeSH)	45
13. “Telephone” (MeSH) OR “telephone” (non-MeSH)	44
14. “Evaluation Studies” (MeSH) OR “evaluation” (non-MeSH)	38

II. Institutional Documents.

We asked for references from leaders within the National Association of Community Health Centers, Health Resources and Services Administration, and the Bureau of Primary Health Care, and searched the websites of private organizations that promote and/or fund quality improvement efforts to identify institutional documents.

Governmental and Private organizations
1. Bureau of Primary Health Care. Models That Work. Document # PC00004, PC00045, PC531, PC00008, PC00004. Ordered from U.S. Department of Health and Human Services website: http://ask.hrsa.gov/Search.cfm
2. National Association of Community Health Centers, Inc. Studies of Health Center Quality of Care. Available at: http://www.nachc.com/research/Files/HCQualityStudies8.04.pdf
3. The Commonwealth Fund: Quality Improvement Publications: http://www.cmwf.org/topics/topics.htm?attrib_id=9100&portal=yes
4. Kaiser Network Search: “community health center” http://www.kff.org/content/reports.cfm
5. Robert Wood Johnson Foundation. Turning Point Program. http://www.turningpointprogram.org
6. W.K. Kellogg Foundation. Community Voices Program. http://www.communityvoices.org/About.aspx

APPENDIX B. Downs & Black¹ Study Quality Criteria

In order to systematically judge the methodological quality of the QI studies identified for this review, we needed an evaluation tool that was capable of assessing the quality of a broad range of study designs.¹–³ We chose the Downs & Black tool because it was designed to assess both randomized and non-randomized study designs. In addition, it had been tested for internal consistency, reliability, and validity and was relatively easy to use. Still we found it necessary to make six clarifications to the scoring instructions in order to consistently and meaningfully apply the Downs & Black criteria to studies of quality improvement interventions.

These clarifications follow below:

1. In the “Reporting” and the “External Validity” sections of the Downs & Black tool (Items #1–13), “patients” was interpreted to mean the individual patients involved in the study while “subjects” was interpreted to mean the main target of the intervention as articulated by Introduction and Methods portions of the study. For example, if the intervention was mainly focused on providers, then “subjects” was interpreted to mean providers only. If the intervention was focused upon the center as a whole, then “subjects” was interpreted to mean the center as a whole.

2. In the “Internal Validity” sections of the Downs & Black tool (Items #14–26), “patients” and “subjects” were both interpreted to mean the main target of the intervention as articulated by Introduction and Methods portions of the study.

3. Downs & Black items could be satisfied by information obtained from relevant work cited by a primary study (e.g. a few studies referenced previous work that provided information on study population and intervention design.⁴–⁸).

4. Regarding Downs & Black Item #3 (Patient Information), a study received full credit (one point) if at least three individual−level patient characteristics were described (e.g. age, sex, race). A study did not receive full credit if it only provided encounter characteristics (e.g. “visit type”) or only provided center−level or provider−level characteristics (e.g. urban versus rural center, medical sub−specialty).

5. Regarding Downs & Black Item #5 (Principal Confounders), a study received full credit (two points) if the confounding was controlled for in every aspect of the analysis. A study received “partial” credit (one point) if confounding was controlled for in part of the analysis. A study received “no” credit (zero points) if confounders were not controlled for in any part of the analysis.

6. Regarding Downs & Black Item #18 (Statistical Tests), a study could only receive full credit (one point) if the statistical analysis deliberately addressed the issue of clustering when centers were identified as the unit−of−analysis.

7. Downs & Black Item #27 (Power) was not used as part of the evaluation because none of the studies provided power calculations in the manner described by the authors.