Evaluation of a Practice-Based Intervention to Improve the Management of Pediatric Asthma

Abstract

Pediatric asthma remains a significant burden upon patients, families, and the healthcare system. Despite the availability of evidence-based best practice asthma management guidelines for over a decade, published studies suggest that many primary care physicians do not follow them. This article describes the Provider Quality Improvement (PQI) intervention with six diverse community-based practices. A pediatrician and a nurse practitioner conducted the yearlong intervention, which was part of a larger CDC-funded project, using problem-based learning within an academic detailing model. Process and outcome assessments included (1) pre- and post-intervention chart reviews to assess eight indicators of quality care, (2) post-intervention staff questionnaires to assess contact with the intervention team and awareness of practice changes, and (3) individual semi-structured interviews with physician and nurse champions in five of the six practices. The chart review indicated that all six practices met predefined performance improvement criteria for at least four of eight indicators of quality care, with two practices meeting improvement criteria for all eight indicators. The response rate for the staff questionnaires was high (72%) and generally consistent across practices, demonstrating high staff awareness of the intervention team, the practice “asthma champions,” and changes in practice patterns. In the semi-structured interviews, several respondents attributed the intervention’s acceptability and success to the expertise of the PQI team and expressed the belief that sustaining changes would be critically dependent on continued contact with the team. Despite significant limitations, this study demonstrated that interventions that are responsive to individual practice cultures can successfully change practice patterns.

Electronic supplementary material

The online version of this article (doi:10.1007/s11524-010-9471-3) contains supplementary material, which is available to authorized users.

Introduction

Adherence to the National Asthma Education and Prevention Program (NAEPP) guidelines for asthma management effectively reduces morbidity and mortality.^1–4 Although the guidelines have been available for over a decade, published evidence indicates that many primary care practitioners do not follow them.^5,6 Studies of approaches to improving practitioners’ adherence to the guidelines have found that a variety of multidimensional approaches to clinician education are somewhat effective in changing physician behavior and improving health outcomes.^4,7–11 However, a randomized, controlled trial of a continuous quality improvement intervention did not significantly affect processes or outcomes of care for children with asthma.¹² What generally emerges from the literature is that to be effective, interventions must be tailored to the environment and culture in which they are implemented.

Academic detailing is one promising approach for effecting changes in individual practitioners’ clinical decision making to improve quality of care.^13–15 Academic detailing interventions center around visits by a trained health professional who provides evidence-based information and recommendations to practitioners in their offices. Key principles of academic detailing include assessment of practice-specific barriers to appropriate care; use of simple, tailored messages to address those barriers; and delivery by a respected person.^13,14 This article describes an intervention that used problem-based learning coupled with improved systems support within an academic detailing model and presents early qualitative and quantitative findings on the intervention’s effectiveness in changing practice patterns. Evaluation questions addressed the process of implementation, the primary changes associated with the intervention, and the barriers to implementing and sustaining those changes.

This intervention was conducted in Richmond, Virginia where, as in other parts of the country, pediatric asthma represents a significant healthcare burden. The average pediatric asthma hospitalization rate in the Richmond metropolitan area (RMA) in 2005 was 29.3 per 10,000, more than twice the overall Virginia average of 14 per 10,000.¹⁶ In 2003, the Central Virginia Asthma Coalition received a multiyear implementation grant from the Centers for Disease Control and Prevention’s Controlling Asthma in American Cities Project and used it to develop Controlling Asthma in the Richmond Metropolitan Area (CARMA). CARMA was a community collaborative whose mission was to work in partnership with families and the community to decrease morbidity and improve the quality of life for children with asthma in central Virginia. The CARMA catchment area included the city of Richmond and two surrounding counties, Henrico and Chesterfield.

Methods

Setting and Recruitment

In 2004, CARMA initiated a multilevel Provider Quality Improvement (PQI) intervention to improve healthcare providers’ adherence to the NAEPP Expert Panel Report-2 (EPR-2) guidelines.² This community-based intervention targeted a diverse group of pediatric practices in the RMA that were not exclusively part of a health plan or an academic setting. The Bon Secours Health System Institutional Review Board reviewed the PQI intervention and concluded that it met the qualifications for exemption. To conform to Health Insurance Portability and Accountability Act regulations, the physician member of the PQI team signed a business associate confidentiality agreement with each practice before conducting chart reviews.

From the 44 pediatric practices in the catchment area, a convenience sample of six practices were offered the opportunity to participate and all agreed. Some were selected based on provider interest as evidenced by one of their physicians attending a pediatric asthma care seminar. Others were selected to represent a range of socio-demographic and practitioner conditions in the CARMA catchment area (see Table 1).

Table 1.

Characteristics of practices in PQI intervention

Practice	Location	Number of physicians	Number of sites	Percent Medicaid	Affiliation
P1	Inner city residential	1	1	>90	Independent
P2	Inner city residential	1	1	>90	Independent
P3	Inner city residential	3	1	>90	Community health center
P4	Urban residential	4	1	∼45	Independent
P5	Urban business district	8–10	2	∼50	University
P6	Suburban	9	2	∼10	Independent

The PQI intervention was implemented at each of two locations of the private multisite practice and only at the campus office of the university practice. The PQI team loaned a spirometer to participating practices for the duration of the 1-year intervention as an incentive for participation.

Intervention

The CARMA PQI intervention team consisted of a primary care pediatrician (0.4 full time equivalent), who was well known in the community, and a nurse practitioner (0.8 FTE), who was a certified asthma educator. The team’s purpose was to help practices implement and follow the key clinical activities recommended by NAEPP as essential for the provision of evidence-based asthma care.¹⁷ Two elements of the chronic care model helped guide the process to produce changes in asthma care: delivery system redesign and decision support.¹⁸ Hence, the main intervention components were (1) physician and staff education in the NAEPP EPR-2 guidelines² and spirometry administration and interpretation, (2) practice systems redesign to support optimal asthma care, and (3) decision support to ensure appropriate implementation of the key clinical activities. The team encouraged the practices to select the elements of care and tools on which to focus and guided the practices through repeated cycles of trial and feedback.

Each practice designated a physician champion and a nurse champion who served as key points of contact for the PQI team and as peer leaders and advocates for change within their practice. The intervention began with a planning session between the practice champions and the PQI team that included assessment of practice characteristics (e.g., practice functions and roles; the provision of scheduled follow-up visits and patient education and, if offered, by whom; and tools used for education and diagnosis) and practice-specific areas for improvement. A baseline review by the PQI team of a non-random sample of charts helped determine the emphasis for each practice. Practices were asked to pull charts of their choice for children aged two years and older who had had an asthma-related visit at least once in the previous 12 months. The chart review documented the following eight indicators related to NAEPP key clinical activities (Table 2): asthma classification, patient and family education, spirometry results, use of asthma action plans, scheduling of follow-up visits, trigger identification, documentation of current symptoms, and prescription of spacers for use with asthma inhalers.

Table 2.

Chart review indicators and criteria by NAEPP² key clinical activity areas

Key clinical activity area	Definition of chart review indicators
Assessment and monitoring	1. Documentation of symptom frequency (daytime, nighttime, or both) for at least one asthma-related visit over the preceding 12 months
	2. Documentation of NAEPP asthma classification at least once over the preceding 12 months
	3. Documentation of at least one routine (non-urgent) asthma-related visit in the preceding 12 months
	4. Tracing or report of spirometry testing performed within the preceding 12 months
Identifying and controlling factors contributing to asthma severity	5. Listing of child’s asthma triggers or results of allergy testing in chart
Pharmacotherapy	6. Documentation that a spacer was prescribed or given for use with an asthma inhaler
Education for partnership in care	7. Either an updated (within one year) copy of a written asthma action plan in the chart, or documentation that an updated plan was provided
	8. Documentation of asthma education topics covered in the preceding 12 months, or the presence of an updated asthma action plan in the chart

In six to eight one-hour sessions during the first four months of the yearlong intervention, the champions worked intensively in the practice setting with the PQI team to master practice-specific objectives based on the key clinical activities. Physician training covered clinical care consistent with the EPR-2 guidelines, using proven provider education methods, such as problem-based learning and training materials based on the Physician Asthma Care Education model.^19,20 The guidelines were also embedded in office visit templates developed with the practices to use for charting and all communications between the PQI team and the practices, whether face-to-face, e-mail, or phone. When difficult cases were encountered, the physician PQI team member consulted with local experts and communicated recommendations back to practitioners, either orally or via e-mail.

The PQI pediatrician trained the physicians and nurse practitioners and the PQI nurse educator trained the nursing staff in parallel sessions. Although all clinical staff members were encouraged to attend, attendance, which always included the champions, ranged from two to eight people, representing anywhere from 10% to 100% of a site’s practice staff. This variable attendance obliged the champions to disseminate necessary content to non-attending staff.

The PQI team promoted delivery system redesign by helping the practices select and integrate tools such as templates for charting asthma office visits, patient education handouts and directions for medications, and written asthma action plans. Printed materials, teaching devices, and spacers to use with asthma inhalers were assembled in toolkits that contained everything providers might need during asthma visits in one convenient, portable location. The cost of tools and supplies provided to the practices (e.g., spirometers, toolkits) varied from $1,000 to $1,500 per practice.

The PQI team redistributed the division of labor in the practices by empowering non-physician office staff to increase their involvement in asthma education and support. The PQI pediatrician worked with the physicians to promote their buy-in and acceptance of delegating asthma education to staff. The PQI nurse practitioner taught staff how to provide patient and family education about medications and the appropriate use of inhalers, peak flow meters, and spacers. Nursing staff were encouraged to use patient management tools such as office visit templates, telephone triage forms that included asthma symptom assessment, and tickler files to track follow-up visits. Strategies for promoting planned follow-up visits included using medication refill phone calls as opportunities for reassessment and for scheduling appointments.

The PQI team adapted practice-specific teaching and review methods and timelines to match the intervention’s one-year time frame. After both the physician and nurse champions had reached their educational objectives in the four-month training, the intervention’s emphasis shifted to establishing and integrating system changes. At regular meetings, the PQI team and practice staff analyzed progress and addressed practice-specific obstacles to change, such as time constraints, staff resistance, inadequate knowledge of guidelines or tools, staff turnover, or difficulty locating needed toolkits. The PQI team constantly reinforced the establishment of systems to ensure follow-up, appropriate device use, office visit template use (as a reminder of what to focus on and for efficient documentation), and asthma action plans to promote self-management. Although practices instituted various systems to support planned care (e.g., lists of patients with asthma, a “tickler system” for scheduling follow-up visits, a policy of no prescription refill without a follow-up visit), no practice used a registry. During brief unscheduled visits, the PQI team replenished practice materials and resources, including office visit templates, asthma action plans, medication devices, patient handouts, and other educational resources. Between visits, the PQI team was available for technical assistance via telephone, fax, and e-mail. The PQI team conducted chart reviews at three- to six-month intervals and shared the results with the practice to highlight positive changes and identify areas needing further improvement. The team used every contact as an opportunity to reinforce NAEPP guidelines and practice improvements.

Evaluation

The PQI intervention evaluation triangulated findings using evidence from a variety of sources to achieve the following three evaluation goals: (1) to obtain information, primarily through staff questionnaires and individual interviews, on practice engagement and participation, including dissemination of information within the practice and plans for sustaining changes; (2) to document improvements in the quality of care using chart reviews for quantitative evidence, supplemented by qualitative evidence from individual interviews; and (3) to document improvements in procedures and systems using three sources of information—questionnaires, interviews, and chart reviews.

Sources of Information At the end of the one-year intervention period, all practice staff (clinical and non-clinical) received brief questionnaires (Appendix 1), distributed by an evaluator who was employed by the CARMA project but not directly involved in PQI implementation. The questionnaire focused on intervention process measures and perceived changes in practice protocol and patient management related to PQI intervention goals.Within one month after the intervention ended, the CARMA evaluator conducted 30- to 60-min semi-structured interviews (Appendix 2) with physician and nurse champions at five of the six practices. The sixth practice could not schedule interviews within the required time frame.The practices themselves pulled charts for review, resulting in non-random sampling of children seen for asthma. Charts were reviewed at both baseline (for the 12 months preceding the intervention) and post-intervention (for the 12 months after the start of the intervention). Because children often saw more than one provider over the course of the intervention, the assessment integrated the care provided by all physicians in a given practice without linking results to a specific provider. A summary of the chart review criteria used to evaluate quality of care appears in Table 2. Any indicator that occurred in fewer than 70% of charts reviewed at baseline was considered to need improvement. The standard for improvement was defined as a minimum increase of 30%. These standards, while arbitrary, represented a reasonable indication of need for improvement and a practical standard for defining improvement. Table 3 summarizes the results of baseline and post-intervention charts reviews by practice.

Table 3.

Results, by practice, of baseline and post-intervention chart reviews

Classif NAEPP asthma classification, Eductn education documented, Spiro spirometry results in chart, Action plan updated asthma action plan, F/U visit non-urgent visit documented, Trig triggers identified, Symp symptoms documented

Bold font indicates improvement by at least 30 percentage points

Normal font without shading indicates improvement of less than 30 percentage points

Dark shading indicates performance worsened in that indicator

Results

Staff Questionnaires Staff returned 44 of the 61 questionnaires distributed, for a mean response rate for the six practices of 72%. Four practices had a 100% response rate; the lowest was 35%. Staff questionnaire results were highly consistent across practices. Across all practices, 60% of staff knew who their physician and nurse champions were. Although the intent of the PQI intervention was to promote interaction between the physician and nurse champions and other staff members, more staff (70%) in the practices reported having had contact with at least one member of the PQI team. Across all practices, the majority of staff reported having learned spirometry (61%), improved patient and family education (66%) and charting and documentation practices (68%), and received asthma-related professional education (71%) during the intervention. A majority of respondents (61%) rated the level of asthma care provided by their practice as “excellent,” but only 45% rated the level of patient education as “excellent,” indicating a need for further improvement in that area.

Semi-structured Interviews with Champions Champions in each practice attributed to the PQI intervention at least four changes in keeping with best practice guidelines. The most commonly reported changes included increased spirometry use, improved systems for scheduling follow-up visits, use of a severity classification, use of asthma action plans, and patient and family education. Several respondents directly credited the intervention’s acceptability and success to the PQI team’s expertise, demonstrating the importance of having respected leaders as change agents. Several champions thought that continued contact with the PQI team would be critical for sustaining commitment to delivery system and physician behavior changes. Intervention limitations reported included the failure of the physician and nurse champions within a practice to schedule regular communication; great variation within practices in staff interest and willingness to change, particularly among physicians; and uncertainty about reimbursement for preventive visits and patient and family education.

Chart Reviews Table 3 summarizes the results of the chart reviews. At baseline, the chart reviews indicated the need for improvement in almost every indicator for all practices. Only three practices had an indicator that measured above the 70% cutoff for acceptable performance. With few exceptions, all practices improved their documentation for all indicators at follow-up. The percent of practice indicators meeting the 30% improvement goal ranged from four of the eight indicators (50%) for one practice to eight of the eight indicators (100%) for two practices. All six practices improved their documentation of severity classification and trigger identification by at least 30%. Five of the six practices met the improvement criterion for use of spacers; four of the six practices met the improvement criteria for patient and family education, updating asthma action plans, and documentation of symptoms. All four practices needing improvement in scheduled follow-up visits met the improvement criterion. Of the four practices needing improvement in spirometry use, only two met the improvement criterion despite the fact that learning spirometry was cited in the majority of staff questionnaires and increased spirometry use was commonly cited by the champions.

Personal Follow-up Only two of the six practices participating in the PQI intervention had a spirometer at enrollment, one of which was not being used. One of the two (a solo practice) obtained a better instrument as a result of the intervention. Three additional practices obtained a spirometer for the first time. Only one (a community health center) was unable to do so.

Discussion

Cabana and Flores²¹ documented many barriers to the adoption of clinical evidence-based guidelines by “real world” primary care providers. This PQI intervention was designed to address as many of those barriers as possible in a coordinated, comprehensive way while remaining sensitive to the culture, resources, and preferences of individual practices. Interactive, problem-based learning sessions for providers and staff addressed their lack of familiarity with the NAEPP guidelines. Presentation of the guidelines by a respected community leader reduced resistance caused by disagreement with the guidelines. Focus on skills development increased provider and staff self-efficacy, and positive feedback from chart audits and case and spirometry tracing reviews was designed to increase outcome expectancy or physicians’ belief that making the necessary behavior and systems changes would result in improved outcomes. Focused assistance with and regular reinforcement of good asthma care procedures and systems aimed to overcome attachment to previous practices and eliminate logistical barriers to improved care.

The three evaluation information sources provided largely complementary information, substantiating this PQI intervention’s effectiveness at achieving clinical improvements and system changes in pediatric asthma management in a community pediatric practice setting. A notable exception was the discrepancy between the chart review results and the information from questionnaires and interviews regarding improvements in spirometry training and use. One possible explanation for this discrepancy is that the questionnaire and interview results reflected the perception within the practices that spirometry was a major and consistent focus for the PQI team, while the chart review came closer to reflecting the reality of actual use. Also, the fact that many of the charts were for children too young to perform spirometry contributed to the discrepancy.

Although all practices demonstrated, by enrolling in the intervention, a willingness to change, pace of change and commitment to sustain change varied greatly. These disparities seemed largely dependent upon the confounded variables of practice size (solo versus multi-clinician), practice setting (inner city vs. other, urban vs. suburban) and patient mix (Medicaid vs. private insurance). The champions in the suburban practice, which was in the midst of adding a building dedicated to chronic conditions, were the most confident about their ability to sustain and build upon the changes introduced by the intervention. While less variation in commitment and providers made working with solo practitioners easier, their worries about the demands of the practice, the patient population, and stringent financial circumstances were obstacles. Sustainability is likely to be dependent upon continued “detailing,” which may not be realistic. Overall, the PQI team learned that while predicting which practices had the greatest potential for commitment to and success in making and sustaining change was difficult, generally, change was hardest for those practices serving children with the highest burden.

This conclusion poses an obvious dilemma for program planners and funders. The PQI intervention demonstrated that following the initial intervention phase, a subset of practices may require labor-intensive, ongoing academic detailing and support to achieve and sustain high-quality care. The practices needing the most intensive intervention might also be those with the greatest access to children most in need. One thing that these very different practices agreed upon almost unanimously was that the champions’ commitment to active advocacy for sustained change and additional needed change might falter without regular contact with the PQI team.

Limitations The primary limitations relate to the evaluation and not to the intervention itself. The evaluation represents a reasonable compromise, in a real-world setting, between a randomized-control trial design and a design that incorporates participants’ perspectives on the intervention’s effectiveness and accommodates the intervention’s flexibility and responsiveness to individual practices’ cultures and preferences. This evaluation provides a description of the process and outcome for the participating practices based on self-report with no objective validation of the reported changes. A longer-term follow-up would improve the evaluation of staff impact and sustained individual and practice changes associated with the PQI intervention. In the meantime, continued informal contact between the physician member of the PQI team and most of the participating practices suggests that despite the champions’ pessimism about sustaining change, the practices have continued practice improvements without the continued involvement of the PQI team.Although this project provides valuable evidence of outcomes that might reasonably be expected to be associated with an “academic detailing” intervention, generalizability is limited by the sampling methods for both the participating practices and chart review. Because the practices in this study chose to participate, they were probably more highly motivated to change. Within the participating practices, the chart review process was neither randomized nor in any way under the control of the evaluators. The primary purposes of the chart reviews were to provide ongoing feedback to the practices on strengths and weaknesses and to monitor progress toward reaching goals. No practice maintained a registry of patients with asthma or could easily extract such information; consequently, the charts used for evaluation could not be randomly selected from the population of patients with asthma and might have been chosen to represent a subset of patients receiving a higher quality of care. Furthermore, there were no comparison practices or providers against which to assess the impact of the intervention as opposed to the influence of secular trends. The PQI team member conducting the chart reviews may not have been impartial in measuring improvements. The structure of the project, with its emphasis on real-world intervention rather than research evaluation, precluded random sampling, obtaining inter-observer reliability or having an external evaluator conduct the chart review rather than a member of the intervention team.The absence of measures of cost and cost effectiveness also limits the generalizability of these findings. The major cost was professional time, which was relatively high, with the employment of a 0.4 FTE pediatrician and a 0.8 FTE nurse practitioner for six practices. Additionally, the cost per practice varied considerably as a function of practice size, preexisting practice policies and practices, and readiness to change.

Conclusion

Implementation of evidence-based guidelines by community practices is critical to the improvement of chronic disease management. A barrier to guideline dissemination is the variation among practices. This intervention’s greatest strength was its flexibility. Some practitioners had excellent knowledge and acceptance of the guidelines but needed help with system redesign to ensure a more planned approach to care; others benefited from one-on-one education about guideline-based management strategies. All benefited by having staff trained as partners in providing coordinated care to children with asthma.

While the strength of the intervention lay in its flexibility, that same quality makes it difficult to provide meaningful estimates of cost to a practice. This paper provides the cost of the intervention in terms of professional FTEs and cost of materials, but too many other variables influence cost to a practice. Based on this intervention’s experience, the following four conditions are necessary for success:

1. Commitment of the practice, including all staff, to a quality improvement initiative and clinical leadership for the initiative.

2. Intervention team credibility to the practitioners. The team must have solid expertise in both the content and application of evidence-based best practice guidelines and that expertise must be evident to the practitioners.

3. Intervention team skill in motivating and teaching professionals to make changes. Team members’ personalities and approaches to peer education are crucial to the success of such interventions.

4. Flexibility in adapting intervention details to each practice’s characteristics. Interventions need to be tailored to the unique features of the practices in which they are implemented.