Clinical Effectiveness Research in Managed-care Systems: Lessons from the Pediatric Asthma Care PORT

Abstract

Objective

To highlight the unique challenges of evaluative research on practice behavior change in the “real world” settings of contemporary managed-care organizations, using the experience of the Pediatric Asthma Care PORT (Patient Outcomes Research Team).

Study Setting

The Pediatric Asthma Care PORT is a five-year initiative funded by the Agency for Healthcare Research and Quality to study strategies for asthma care improvement in three managed-care plans in Chicago, Seattle, and Boston. At its core is a randomized trial of two care improvement strategies compared with usual care: (1) a targeted physician education program using practice based Peer Leaders (PL) as change agents, (2) adding to the PL intervention a “Planned Asthma Care Intervention” incorporating joint “asthma check-ups” by nurse-physician teams. During the trial, each of the participating organizations viewed asthma care improvement as an immediate priority and had their own corporate improvement programs underway.

Data Collection

Investigators at each health plan described the organizational and implementation challenges in conducting the PAC PORT randomized trial. These experiences were reviewed for common themes and “lessons” that might be useful to investigators planning interventional research in similar care-delivery settings.

Conclusions

Randomized trials in “real world” settings represent the most robust design available to test care improvement strategies. In complex, rapidly changing managed-care organizations, blinding is not feasible, corporate initiatives may complicate implementation, and the assumption that a “usual care” arm will be static is highly likely to be mistaken. Investigators must be prepared to use innovative strategies to maintain the integrity of the study design, including: continuous improvement within the intervention arms, comanagement by researchers and health plan managers of condition-related quality improvement initiatives, procedures for avoiding respondent burden in health plan enrollees, and anticipation and minimization of risks from experimental arm contamination and major organizational change. With attention to these delivery system issues, as well as the usual design features of randomized trials, we believe managed-care organizations can serve as important laboratories to test care improvement strategies.

The Agency for Healthcare Research and Quality (AHRQ) initially sponsored Patient Outcomes Research Teams (PORTs) to study, using observational designs, processes and outcomes of common health conditions (Maklan, Greene, and Cumming 1994). More recently funded PORTs respond to an interest by AHRQ in interventional research to evaluate care improvement strategies. Managed-care organizations (MCOs) may be particularly effective population laboratories in which to undertake such studies. Many MCOs have expertise in practicewide care improvement, have available data to measure utilization outcomes, and are interested in the cost-effectiveness of the care they provide. However, investigators will find new challenges in these environments in both the design and implementation of long-term interventional studies. Little has been published to help them proactively address the barriers and challenges that they are likely to face.

The Pediatric Asthma Care PORT (PAC PORT) was designed as a large, multisite, multiyear, randomized trial of care improvement strategies for childhood asthma within diverse managed-care settings (Finkelstein et al. 1998). In this report, we examine the history of the design and implementation of the PAC PORT in these “real-world” managed-care settings. Currently, analysis of trial results is still underway. Here, we focus on the challenges we encountered as investigators who needed to assure the integrity of a multiyear randomized trial of improved health services, while also accommodating the special features and program initiatives of large MCOs. Regardless of the outcomes of the trial, the lessons stand alone for those seeking to do interventional research in similar systems.

Project Rationale and Description

Childhood asthma was selected as the target condition for this trial because of recent changes in treatment recommendations, embodied in practice guidelines of the National Asthma Education and Prevention Project (NAEPP) (National Asthma Education and Prevention Program 1997), that were widely believed to have the potential to decrease morbidity and functional impact of this condition for children. Based on a conceptualization of asthma as a chronic inflammatory condition, they emphasize classifying patients by severity, use of inhaled anti-inflammatory agents to treat all but the mildest patients, and shifting care to the anticipatory management of airway inflammation (from a primary focus on medical management of symptom crises).

While there is some evidence of acceptance and use of these guidelines (Legoretta et al. 1998; Lozano and Lieu 1999), full adoption by the primary-care community remains elusive (Buchner, Carlson, and Stempel 1997; Goodman et al. 1999; Werk et al. 2000). This is not surprising given that implementation of evidence-based guidelines has been difficult and slow for many areas of medical practice (Lomas et al. 1989; Greco and Eisenberg 1993). Generic barriers to guideline adoption have been described and include: lack of awareness, familiarity, agreement, self-efficacy, and outcome expectancy, as well as inertia and external barriers (Cabana et al. 1999). The traditional organization of primary-care practice itself may be ill-suited to the adoption of comprehensive, prevention-oriented, clinical-practice guidelines. Because of these and other barriers, the literature suggests that it is unlikely that U.S. physicians will broadly adopt the NAEPP guidelines under traditional, passive-dissemination strategies (Davis et al. 1995; Soumerai, McLaughlin, and Avorn 1989). The Pediatric Asthma Care Patient Outcomes Research Team (PAC PORT) was designed to test strategies to overcome these barriers and to implement pediatric asthma-care guidelines in primary-care settings.

Forty-two practices were recruited and randomly assigned to: (1) a physician behavior change intervention, including training of a practice-based peer leader (PL), (2) the PL intervention plus an organizational change in primary-care delivery (Planned Asthma Care Intervention [PACI]), incorporating the concept of routine preventive visits with a physician and asthma nurse educator, or (3) usual care. We asked practices to select a candidate PL based on criteria used in successful behavior change interventions using trained community opinion leaders as change agents (Lomas et al. 1991; Soumerai et al. 1998). These PLs were trained in both physician behavior change techniques (e.g., academic detailing) and content of asthma guidelines. They conducted a range of clinician education and care improvement activities in their practices, supported by an educational coordinator with whom they communicated monthly. The PACI intervention was based on the work of Wagner and others who have developed multidisciplinary teams to optimize care for patients with chronic disease (VonKorff et al. 1997). The use of patient registries, integrated specialty and primary care, and a focus on patient self-management of chronic disease are all integral features of these programs. Outcomes of the PAC PORT trial will include both practice level measures of change in asthma pharmacotherapy from automated claims data and measures of morbidity for individual children assessed through bimonthly telephone contacts.

It was anticipated that setting the work of the PAC PORT in three geographic locales and within diverse health-care systems would increase the complexity of the trial but increase the generalizability of its results. These systems included Harvard Pilgrim Health Care (HPHC), Group Health Cooperative (GHC), and Rush Prudential Health Plan (RP). HPHC is a multimodel MCO in New England that provides a variety of central managed-care functions (insurance, marketing, quality assurance, and information technology among others) and contracts with a large number of physician groups. The GHC is a mixed-model consumer governed managed-care organization. The GHC's group model division, which participated in the study, serves 400,000 members in two large metropolitan areas in Washington state— Seattle and Tacoma—with 29 primary care sites and 3 specialty centers. Finally, RP was established as an alliance between Rush Presbyterian, St. Luke's Medical Center in Chicago, Illinois, and Prudential Insurance. The original delivery system was a set of primary-care practices in the greater Chicago area caring for more than 100,000 people in a staff-model environment. The alliance with Prudential led to the addition of a large network of independent practice affiliates.

The trial was designed to assess the effectiveness of strategies to implement guidelines for childhood asthma in community-based, managed-care settings. It was not to be an efficacy trial in rigidly controlled environments that would permit isolation from all other influences on clinician and patient/family actions. A prospective randomized controlled design, rather than an observational one was chosen to provide compelling evidence for intervention effects, while controlling for secular change in what was bound to be an evolving practice environment. At the outset of the PAC PORT in 1996, while the collaborating investigators were generally aware of, and prepared for, the challenges of ongoing processes of change in our “moving health-care systems,” none of us fully envisioned the many interactions of these systems changes with the final design and implementation of the PAC PORT. For this report, investigators at each health plan described the organizational and implementation challenges in conducting the randomized trial. These experiences were reviewed for common themes and “lessons” that might be useful to investigators planning interventional research in similar care-delivery settings.

Specific Challenges and PAC PORT Responses

The most important challenges faced by investigators are summarized in Table 1, along with the responses of the PAC PORT team.

Table 1.

Specific Challenges and PAC PORT Investigator Responses

Challenge	Response
Concurrent MCO care improvement initiatives  threatened contamination across study arms.	• Participated directly in MCO-wide initiatives through investigator membership on key planning and implementation groups.
	• Helped focus health plan on patients with more severe asthma than those targeted by the study.
	• Obtained agreement to roll out new initiatives to minimize unequal effects by study arm.
Network model managed-care sites contracted  with several MCOs resulting in (1) diluted  clinician attention to a project sponsored by  a single MCO; (2) fewer potentially eligible  subjects for measurement of study outcomes  in each practice.	• Encouraged sites to use materials for all patients of a practice, regardless of insurer. • Collaborated with a second large MCO, increasing the number of patients eligible for study enrollment and outcomes measurement.
Wide range of collaborative models found  within practices, some with little enthusiasm  for practice-wide change.	• Tailored roles and responsibilities of peer leader to available opportunities in each practice.
Perceived organizational demand for  provider productivity.	• Aligned goals of intervention with that of efficiency in caring for children with asthma.
	• Offered “locum tenens” staff to allow peer leaders to spend time on study initiatives.
Provider cross-site mobility threatened  contamination across arms.	• Tracked provider site changes closely so that such contamination could be accommodated in interpretation of study results.
Diversity of provider speciality-pediatricians  and family physicians.	• Designed flexible provider education to meet needs of diverse provider groups.
Changing health plan senior leadership.	• Secured support of three successive medical directors.

Harvard Pilgrim Health Care

Potential for Contamination by Contemporaneous Care-improvement Initiatives

Asthma care-improvement activities within the various regions and practice groups of HPHC varied substantially at the time of study inception. Within its staff-model division (now Harvard Vanguard Medical Associates), childhood and adult asthma had been the focus of special continuous quality-improvement initiatives. More than seven thousand members with asthma had been annually surveyed to characterize health status, treatment patterns, service utilization, and patient/family satisfaction with care. A pilot study of nurse-specialist services for children with asthma had demonstrated decreased use of emergency room and inpatient services (Greineder, Loane, and Parks 1995; 1999). Because of these activities, the PAC PORT trial avoided the staff model division, choosing to work instead in the HPHC network practices. Even here, HPHC had already offered network physicians incentives to participate in continuing medical education activities regarding asthma, and plans to disseminate the nurse program were being contemplated. Organizationwide, a multidisciplinary “Asthma Steering Committee” met regularly, tracking changes over time in survey results from the asthma cohort and guiding investments in asthma quality improvement. In the course of the PAC PORT itself, HPHC expanded its nurse specialist services to patients with repeated ER visits or hospitalizations who belonged to any group or practice site in Massachusetts.

Given the extent and diversity of these activities, it was critical to the success of the PAC PORT that investigators be aware of the goals of the health plan and of the details of care-improvement initiatives in all stages of planning. Membership of investigators on the Asthma Steering Committee allowed such communication and lent academic expertise in data interpretation and intervention planning to the health plan. It also allowed study integrity to be one consideration in the planning process of health plan administrators who came to believe that the lessons that might be learned from the PAC PORT were worth minor modification in the timing of care-improvement activities. For example, the decision was made to delay active promotion of the nurse specialist program at the PAC PORT sites, in order to prevent massive contamination of these practices. Even so, practitioners who made specific requests at these sites for the assistance of a nurse-specialist could obtain such assistance, highlighting the primacy of member health and satisfaction over strict research design integrity.

Diverse Community-based Practices

Implementation of the PAC PORT within the network practices at HPHC had many consequences. First, HPHC was not the only managed-care insurer for these practices and accounted for a minority of patients they served. Though PAC PORT investigators encouraged materials and approaches be used for any or all patients at a site, the practitioners understandably resisted office systems development that might not serve all of their patients. If an “asthma closet” of equipment, supplies, and educational materials were to be developed, for example, would this be a useful resource for any patients in the practice, whomever their insurer might be? If patient training with a particular spacer from the asthma closet were undertaken, would this particular device be listed on the formularies of all insurers? Since the available population (HPHC members) represented only a fraction of the entire practice, the number of children eligible for the study threatened the achievement of sample size targets. Ultimately, we were only able to enroll a sufficient number of patents in these practices by engaging a second large New England MCO, and recruiting additional patients insured by them.

The diversity of practice cultures and levels of collaboration among clinicians within network practice groups was surprising. Some groups were highly integrated both financially and in their clinical practice, while others acted as individual practitioners merely sharing space, support staff, and, in some cases, financial risk. This made it difficult to identify, in some practices, a “peer leader” who would be truly able to influence opinions groupwide. It also posed a challenge for the PAC PORT when attempting to achieve a high degree of enthusiasm among all practitioners for anticipatory management (“asthma checkup”) visits and other practice changes. Finally, because PAC PORT sites were widely dispersed geographically, travel time, coordination of PAC PORT visits, participation of peer leaders in centralized workshops, and all other activities requiring in-person contact were more difficult than would have been the case had these practices been located in one metropolitan area. Maintaining their focus on the PORT special services and their voluntary participation in a research project, however promising and important, was clearly a challenge.

Group Health Cooperative

Demand for Provider Productivity

The most salient challenge at GHC was the productivity-driven environment, leading to physician and staff stress. Perceived lack of time was an obstacle to participation at all stages of the study, including clinic recruitment, scheduling PL training, and participating in asthma check-ups. We also noted a broad range of levels of collaboration among physicians in a clinic. Despite the perception of uniformity of medical practice in large group or staff-model MCO settings, there were few formal or informal mechanisms to promote physician-to-physician communication. So, contrary to what might be expected, physicians in the GHC clinics were similar to those of network practices at HPHC in the diversity of their attitudes toward accepting a position of clinical leadership (such as peer leader) or embracing organizational change.

Delivery System-research Synergy and Systemwide Secular Change

The group model setting provided many advantages for conducting interventional health services research. In contrast to the network setting, all patients in a clinic were GHC patients. Therefore, all children were potentially eligible for the study. Prior to the PAC PORT study GHC had not undertaken any systemwide asthma care-improvement activities, but had implemented coordinated systems for care of other chronic illnesses (VonKorff et al. 1997). Local and regional efforts in asthma care were varied and included provider education sessions, ad hoc utilization and pharmacy reports, and clinical nurse-specialists providing patient education at certain clinics.

At the time the study was under development, GHC established an Asthma Roadmap Team to facilitate systemwide, clinical practice change in adult and pediatric asthma care. The PAC PORT study investigators acknowledged that research goals would be secondary to clinical goals and did not ask the MCO to forestall implementation of system improvements. Investigators did request that GHC commit to systemwide rollouts of any asthma-care initiatives rather than clinic-by-clinic or regional changes. Systemwide changes would be easier for the PAC PORT study to handle analytically than local changes. Furthermore, by working closely with the Roadmap Team, PAC PORT investigators ensured that the products that the Roadmap Team developed were consistent with fundamental elements of the study, such as the pharmacotherapy guidelines (National Asthma Education and Prevention Program 1997) and the chronic illness care model (VonKorff et al. 1997). Ultimately, the PAC PORT study and the Roadmap Team shared expertise and collaborated on a number of development efforts. One of the major products of the Roadmap Team's work was a website designed to facilitate physician understanding and adherence to the guidelines. The Roadmap Team's endorsement greatly facilitated acceptance of the study among primary-care providers and regional administrators.

The group model setting also allowed study staff to work more closely with providers to meet their needs. The PAC PORT study was able to respond to PL concerns about lack of time to conduct PL activities by offering to offset a portion of that time with substitute (locum tenens) physicians. Clinicians perceived study staff as working within the same organization and this may have facilitated collaboration. Clinical providers used the same internal e-mail system as the research team, simplifying communication and allowing them to rely less upon telephone messages or faxes.

Mobility and Disciplinary Diversity

GHC is a relatively fluid system where physicians move between clinics and regions with relative ease. In the first year of the study three physicians transferred among four different clinics. Because these movements threatened contamination across study arms, they were carefully tracked by the PAC PORT so that they could be taken into account in the assessment of intervention effectiveness. Family medicine physicians at GHC serve as primary-care providers for about half of all pediatric asthma patients. Individual family physicians tended to have very few eligible patients per panel (less than 10) whereas pediatricians typically had 20 to 40. We viewed the diversity of GHC providers as a strength for the clinical trial; however, this diversity required tailoring the intervention to two distinct groups of doctors involved in the study, each with different pediatric panel sizes and levels of experience with pediatric asthma.

Rush Prudential Challenges

Asthma Care-improvement Initiative

Unlike either GHC or HPHC, RP did not initiate a formal administrative team to guide improvements in asthma care. It did, however, assign one of its senior clinical staff as an asthma program leader, who worked primarily with clinicians in the staff model practices.

A pilot effort began in 1995, prior to the PAC PORT, to study the impact of a nurse case manager in reducing morbidity of children with asthma who had at least two hospitalizations or three or more emergency department visits. Interest in the program's early results led to the establishment of a full-time asthma nurse position in the health plan to assist in general education efforts and targeted case management for selected enrollees. Similar to Harvard Pilgrim, the dissemination of this service had the potential for contamination across trial arms; however, the program provided care for children with severe asthma and short-term follow-up of children who were hospitalized, rather than the children with mild and moderate asthma targeted by the PAC PORT. Additionally, in 1999 the quality management department began sending educational mailings to all patients with asthma in the RP network.

Variable Study Participation by Providers

The PORT included staff-model practices and a number of network practices in the trial at RP. Engagement of clinicians was relatively easy within the staff-model and much less easy in the more independent network offices. Staff-model practices physicians were more likely to view the interventions as a policy of the general administration and more quickly expressed openness to attempts to change practice.

Changing Senior Clinical Leadership

While the leadership at the practices remained relatively stable, the senior clinical and administrative leadership of Rush Prudential underwent change. During the study period there were three different medical directors, each with his or her own clinical agenda, and each with differing views of the relative importance of research as a corporate priority. With each change, PAC PORT investigators had to review the project's goals and methods with the new leadership to secure their support. Continuity for the project was also maintained through continuous contact with the physician responsible for planwide quality improvement programs.

Implications for Reasearch in Managed-care Systems

Analysis of the challenges described above produced a number of generalizable lessons, summarized in Table 2, for researchers contemplating trials of health delivery-system change in managed-care systems.

Table 2.

Five Lessons for Successful Intervention Research in Managed-care Systems

Understand the ecology of each health plan and actively participate in relevant care  improvement efforts.

Understand the ecology of practice sites and tailor interventions to maximize potential  benefits to participating clinicians.

Design interventions with the expectation that they will evolve over the course of the trial in a  process of continuous improvement.

Limit respondent burden.

Understand and manage sources of potential experimental group contamination.

Understand the Ecology of Each Health Plan and Actively Participate in Relevant Care-improvement Efforts

We learned just how much diversity there is in the both the “facts on the ground” and clinical cultures of modern managed-care organizations with respect to disease-management initiatives. Only by being an integral part of ongoing health plan activities could PORT investigators hold the attention of health plan managers through the lead time (of approximately two years) for research review, approval, and funding that is so different from that for quality improvement activities. Investigator participation in plan initiatives also helped minimize contamination of experimental arms by the introduction of improved services. This investment of research resources in the comanagement of health plan activities is necessary in environments in which systemic new initiatives could substantially affect the internal and/or external validity of a health systems intervention trial.

Understand the Ecology of Practice Sites and Tailor Interventions to Maximize Potential Benefits to Participating Clinicians

Staff model or exclusive-insurer environments seem to increase the receptivity of practice staff (both clinical and administrative) to the activities of intervention trials. It was clearly most difficult for the PAC PORT to get in the door of practices where only a minority of patients was insured by one organization. A diversity in provider types, self- or plan-imposed pressure for patient-care productivity, the absence of shared decision making among clinicians within a practice group, reluctance to add visits for educational and preventive-care purposes because of incentives for utilization management, and the perceptions of investigators as academics or insurance company representatives all pose special difficulties. The presence of a supportive clinical leader within the practice (e.g., senior partner or clinician-manager) can certainly open the practice to study participation. But in the end, an intervention can only succeed if front-line clinicians are convinced that the practice change being tested could substantially benefit patients, while not increasing their own work burden.

Design Interventions with the Expectation That They Will Evolve Over the Course of the Trial in a Process of Continuous Improvement

In a randomized trial, each intervention strategy must be specified in detail before starting. However, the particular activities and tools within each intervention should be expected to evolve based on the response of the practice sites. Virtually all features of the PACI and PLE interventions needed to be adapted to specific practice circumstances in order to maximize their potential. Trained peer leaders would move and need to be replaced and space constraints in particular practices would impose absolute limitations on the number of individuals who could be present for special asthma care visits in PACI practices. Data obtained during the course of the trial itself, including information on clinician practice patterns, can be used formatively, to provide feedback to clinicians within the experimental groups. For example, the PAC PORT provided feedback to practices on their rates of prescription of anti-inflammatory agents. Furthermore, across all of the trial practices, promising innovations in practice resources and approaches emerged—for example, new approaches to recording care plans, developing practice registries, and exchanging information from clinician to clinician about prescriptions issued to a patient. These innovations were made known to all practices within an intervention arm of the randomized trial in order to foster yet more care-improvement initiatives.

Limit Respondent Burden

In all of the plans, subpopulations of enrollees with childhood asthma were already receiving surveys, educational information, or other disease management materials. The PAC PORT measurements were additional impositions on children with asthma and their parents. Investigators conducting trials in such health systems need to take into consideration the respondent burden from study participation in combination with health plan measurement activities, since they are likely to draw from the same reservoir of available time and good will.

Understand and Manage Sources of Potential Experimental Group Contamination

When clinicians or patients move from one practice to another the risk of contamination of one trial arm is clearly present. Furthermore, to the extent that resources (educational materials, for example) designed for the interventional arms are used for health-planwide care-improvement efforts, contamination of the “usual care” control arm is also potentially a risk. This cross-arm contamination exists over and above the exposure to guideline dissemination, physician education, and natural diffusion of innovation that constitute the secular trends in medical care improvement. The existence of these forces underscore the need to carefully track organizational sources of cross-arm contamination.

Concluding Observations

To a limited extent, any randomized, controlled trial (even a classical trial designed to evaluate a new pharmacologic agent) is subject to biases introduced by the ecology of the practice environment in which the trial is carried out. The nature of the enrolled population, variation in measurement timing and completeness, the success of blinding, and the presence or absence of co-interventions may be affected by the study environment. In a large trial when only a modest fraction of all subjects are enrolled at any one site, it is likely that such biases would have a limited impact on the validity of the study. In health services research that features practice-level interventions carried out in a limited number of organizational and geographic environments, however, the potential for bias is increased. In such research, interventions often affect not only patients, but receptionists, nurses, physicians, administrators, information systems, and other elements of the practice environment. Under these circumstances, blinding is clearly not feasible and the number of sites is radically limited. Finally, in rapidly changing health-care delivery systems, the assumption that “usual care” will be static (the classic “control” state) is highly likely to be mistaken.

Randomized trials continue to represent the most robust design for evaluating health services improvement strategies. For this study design to achieve its promise in managed-care settings, trials will need to have many of the following features outlined here. From the perspective of classical randomized trial design principles, the most untraditional aspect is the continuous improvement strategy of the intervention arms, including use of interim information for feedback and facilitation of change in the experimental arms. Based on the experience of this project team, this “iterative-design” approach is actually required for design integrity and internal validity in MCO environments. In a moving system, anything that stands still, including a research intervention, is apt to be “run over.”