Critical Pathway Effectiveness: Assessing the Impact of Patient, Hospital Care, and Pathway Characteristics Using Qualitative Comparative Analysis

Abstract

Objective

To qualitatively describe patient, hospital care, and critical pathway characteristics that may be associated with pathway effectiveness in reducing length of stay.

Data Sources/Study Setting

Administrative data and review of pathway documentation and a sample of medical records for each of 26 surgical critical pathways in a tertiary care center's department of surgery, 1988–1998.

Study Design

Retrospective qualitative study.

Data Collection/Abstraction Methods

Using information from a literature review and consultation with experts, we developed a list of characteristics that might impact critical pathway effectiveness. We used hypothesis-driven qualitative comparative analysis to describe key primary and secondary characteristics that might differentiate effective from ineffective critical pathways.

Principal Findings

All 7 of the 26 pathways associated with a reduced length of stay had at least one of the following characteristics: (1) no preexisting trend toward lower length of stay for the procedure (71 percent), and/or (2) it was the first pathway implemented in its surgical service (71 percent). In addition, pathways effective in reducing length of stay tended to be for procedures with lower patient severity of illness, as indicated by fewer intensive care days and lower mortality. Effective pathways tended to be used more frequently than ineffective pathways (77 versus 59 percent of medical records with pathway documents present), but high rates of documented pathway use were not necessary for pathway effectiveness.

Conclusions

Critical pathway programs may have limited effectiveness, and may be effective only in certain situations. Because pathway utilization was not a strong predictor of pathway effectiveness, the mechanism by which critical pathways may reduce length of stay is unclear.

Critical pathways are being implemented in many U.S. hospitals, primarily to improve the efficiency of hospital care while maintaining or improving quality (Pearson et al. 2001). Critical pathways are structured multidisciplinary care plans that detail essential steps in the care of patients with a specific clinical problem (Campbell et al. 1998). They display goals for patients and provide the ideal sequence and timing of staff actions to achieve those goals efficiently (Pearson, Goulart-Fisher, and Lee 1995). Largely because of pressure from payers to reduce length of stay, pathways have been developed and implemented for a large variety of procedures and illnesses where there appears to be excess variation in length of stay (Campbell et al. 1998).

However, several systematic reviews of the many uncontrolled and few, mostly small, randomized, controlled trials have not supported this widespread use. These reviews conclude that evidence for reducing length of stay and costs is weak because of the preponderance of retrospective studies that use historical controls and do not adjust for risk or secular trends (Conner et al. 2000; Kim et al. 2003; Kwan and Sandercock 2003). Moreover, the use of pathways may have risks. Although most studies do not measure patient-centered outcomes, the review in patients with stroke found that patient satisfaction and quality of life were significantly lower in patients on pathways (Kwan and Sandercock 2003). These reviews also found that results may differ substantially for different types of illnesses and that there was wide variation in, and often insufficient description of, pathway interventions.

Critical pathways require substantial resources to develop, implement, and maintain, and are only one of many potential tools to improve the quality and efficiency of complex medical care. Critical pathway programs often tend to be comprehensive rather than focused, and may target the most common reasons for admission rather than patients for whom the potential for quality improvement is greatest (Darer, Pronovost, and Bass 2002). In particular, the process of pathway development, which includes forming an interdisciplinary team, reviewing data, identifying quality concerns, building consensus, and educating staff on standards, may impact hospital care independent of actual pathway use (Holtzman, Bjerke, and Kane 1998). Efficient investing in critical pathway programs would be facilitated by knowledge of why certain pathways in certain situations succeed while others do not, as well as by understanding the relative importance of the pathway development and implementation processes.

Based on Donabedian's quality assessment model of structure, process, and outcomes (Donabedian 1988), the effectiveness of critical pathways could be affected by characteristics of hospital care, pathways, and patients, and could result from both the process of pathway development and the actual use of the pathway document. We asked the following research question: what patient, hospital care, and critical pathway characteristics are qualitatively associated with pathway effectiveness for reducing length of stay for surgical procedures? We hypothesized that four factors might be associated with effective as opposed to ineffective pathways: (1) length of stay was not already decreasing at the time of pathway implementation; (2) the pathway was the first implemented on a service; (3) patients eligible for effective pathways had lower severity of illness than those eligible for ineffective pathways; and (4) pathway documentation was used more frequently as part of clinical care for effective as compared with ineffective pathways.

METHODS

Study Design and Setting

We conducted a qualitative, retrospective cohort study of a group of critical pathways using qualitative comparative analysis (QCA) (Ragin 1999). QCA is a method in comparative case-oriented research (Ragin 1987) for studying a small-to-moderate number of cases in which a specific outcome has occurred, compared with those where it has not. Using principles of both in-depth case-oriented (qualitative) and variable-oriented (quantitative) research, this method helps to identify cross-case commonalities and differences to construct a general explanation of how an outcome occurs. QCA is based on Boolean algebra and on examining the minimum combination of variables that may result in either the presence or the absence of the outcome. Based on a conceptual model, the method identifies different logical combinations of variables, using AND or OR expressions, that might be necessary and/or sufficient to produce the outcome. In addition, QCA uses the concept of “prime implicants” to test hypotheses about the factors that are associated with the outcome and to eliminate variables that are subsets of these key factors.

We studied critical pathways in the Department of Surgery in a large tertiary care center, The Johns Hopkins Hospital. Pathways were implemented between 1990 and 1996, and covered all major surgeries performed at the hospital. We included all pathways implemented in the study period that applied to a procedure primarily responsible for an admission and had more than 20 patients per year potentially eligible for the pathway. Twenty-six of the 48 surgical critical pathways (54 percent) were eligible for inclusion. Details of some of the pathways (Dardik et al. 1997; Zehr et al. 1998; Pitt et al. 1999; Berenholtz et al. 2001) are described elsewhere.

Because hospital length of stay was already decreasing preimplementation for many of the pathways, we defined pathway effectiveness as an increase in the rate of change in postoperative length of stay. In an individual regression analysis for each pathway, we evaluated whether postoperative length of stay decreased more rapidly during the 2-year post-pathway-implementation-date period than during the 2-year pre-implementation-date period. We adjusted for potential confounders that may have changed over time, including age, ethnicity, gender, health maintenance organization (HMO) or Medicaid as the primary payment source, emergency room admission, transfer from another acute care hospital, and comorbidity. We have described the methods for determining whether a pathway was effective elsewhere in more detail (Dy et al. 2003). After adjusting for confounders, including prepathway trends in length of stay, 7 (26 percent) pathways were associated with a statistically significant effect on decreasing postoperative length of stay.

Defining Pathway Characteristics and Data Collection

Using information from consultation with experts and a review of the literature on critical pathways and related quality improvement methods, we developed a list of pathway characteristics that might be associated with critical pathway effectiveness. This list was refined through consensus of the multidisciplinary research team and review by peers. We also developed a set of key hypotheses and rationales for how these factors might be associated with effectiveness or confound evaluations of pathway effectiveness (Table 1).

Table 1.

Hypotheses for Factors Associated with Critical Pathway Effectiveness Tested in Qualitative Comparative Analysis

Hypothesized Factor	Rationale
Length of stay not already decreasing at the time of implementation	If length of stay is decreasing, change in practice is already occurring; potential etiologies:
	Effects of pathway development process
	Spillover from other pathways on service or ward
	Unrelated factors causing length of stay to decrease
	Therefore, more difficult for pathway to have a significant impact
First pathway implemented in a service	More resources were invested in development and implementation for these pathways
	Effect of these pathways may have “spilled over” into care of closely related procedures and reduced impact of later pathways
Lower severity of illness	Pathways are best suited to more predictable postoperative care
	Pathways may not be well-suited to intensive care
Successful pathway implementation	Mechanism of pathway effectiveness might be related to:
	Utilization of pathway document
	Change in practice related to education and awareness surrounding pathway implementation
	Change in practice related to multidisciplinary pathway development process (i.e, unrelated to pathway implementation)

We determined patient, postoperative care, and hospital care characteristics using the Johns Hopkins Uniform Hospital Discharge Database. The Discharge Database includes International Classification of Diseases: Ninth Revision, Clinical Modification (ICD-9-CM) (U.S. Department of Health and Human Services 1997) codes for one primary diagnosis and procedure and from 4 to 14 secondary diagnoses and procedures for all hospital admissions. We included only the first four secondary diagnoses and procedures, because the number included changed over the study period. The Discharge Database also includes patient demographics, admission source, discharge disposition, hospital service and ward, and postoperative and total length of stay. We identified postoperative complications using ICD-9-CM codes defined by the Complications Screening Program for administrative data (Weingart et al. 2000). We included all surgical complications where the validation of the ICD-9-CM coding by medical record review showed a positive predictive value of >80 percent and that the event occurred in-hospital >70 percent of the time (Lawthers et al. 2000).

We defined pathway development and implementation characteristics by reviewing archived critical pathway files with local critical pathway experts. We defined pathway characteristics, including implementation dates, by reviewing these files, and validated them by reviewing a sample of medical records for each of the 26 included pathways. Two investigators (S.D. and P.D.) developed criteria for defining pathway utilization by jointly reviewing a sample medical record for each critical pathway. We then reviewed a random sample of medical records for the study period, the 2 years after the implementation date for each pathway, to determine patterns of documented pathway use and pathway characteristics. We identified eligible patients using the ICD-9-CM coding definitions for each pathway. We reviewed 26 records for each pathway, which gave us power to estimate within ±20 percent the proportion of records where pathway documentation was included.

We developed a definition for a complete pathway for each procedure. We searched the complete chart to see if the pathway was present. We were able to determine whether a pathway was missing because the pathway was also used as the form for nursing documentation. If the pathway was not used, the nursing documentation should have been present on a different set of paperwork. Therefore, if the pathway was not present, but the corresponding nursing paperwork that should have been used in place of the pathway was also not present, the pathway was coded as missing. Because both pathways and nursing documentation were missing in only 17 (2.7 percent) of charts, this was an acceptable definition of pathway use.

We also developed alternate definitions of pathway use by determining whether and how much the critical pathway document and associated paperwork were actually used. Because the pathways did not include checkboxes for the elements of the pathway, we were unable to determine the percentage of each pathway that was actually followed. We therefore evaluated the percentage of the time that a nurse on each shift for each day of the pathway initialed the pathway document. There was no initialing of the pathway in only nine (2.4 percent) of cases. We obtained similar results when pathway use was defined as pathways where the nurses initialed the pathway form on 75 percent or more of eligible shifts. We identified whether standardized orders and preprinted nursing discharge sheets were used, and whether these were pathway-specific or more general. We looked at where the pathway document was included in the chart, nursing versus progress note section. We also reviewed the pathway documents used in the chart to define pathway characteristics.

Analysis

Exploratory Analyses

For continuous variables, we examined means, medians, ranges, and skew. All continuous variables were then dichotomized for the purposes of QCA. We looked for potential breakpoints in associations with effectiveness with smoothed curves using the Lowess function (Stata Corp 1999). If there was no clear breakpoint, we dichotomized the variable at the mean or the median, depending on distribution and skew.

QCA

Our unit of analysis was the critical pathway. The outcome was whether or not the critical pathway was associated with a statistically significantly effect on decreasing postoperative length of stay. We included the characteristics of hospital care, illness, patients, and the critical pathways, as well as pathway utilization. We entered factors for each pathway into a “truth table” (Ragin 1994), organized by the categories of characteristics. We eliminated characteristics present in fewer than five pathways, because there would be insufficient numbers for analysis. We then simplified the table by evaluating for statistically significant correlations among variables. We constructed scattergrams for each combination of potentially correlated variables. We calculated Pearson's correlation coefficients for combinations of linear variables, analysis of variance for combinations of a linear and a categorical variable, and the φ coefficient for categorical variables (Goldstein 1991). We eliminated variables that were statistically significantly correlated to other variables and considered to be caused by another included variable based on our conceptual model.

We then analyzed the relationships among remaining factors, called “minimizing the truth table,” (Ragin 1994) to test our hypotheses and describe patterns more characteristic of effective than ineffective pathways. Our key a priori hypotheses and rationales for characteristics associated with critical pathway effectiveness are shown in Table 1. We included factors showing more than a 20 percent difference between effective and ineffective pathways, because this would be the margin of error if one pathway in each group were misclassified. For those factors that did appear to be more commonly associated with effective critical pathways, we looked for supporting evidence by determining whether related factors were also associated with effectiveness. Based on these patterns of association among factors with interrelated meaning, we developed a descriptive explanation of how characteristics of hospital care, illness, patients, and pathways, as well as pathway utilization, might impact pathway effectiveness.

Analyses were conducted using Stata 7.0 (Stata Corporation 2001) and QCA 3.1 (Drass 1998).

RESULTS

Pathway Effectiveness, Included Variables, and Pathway Utilization

The seven effective and 19 ineffective critical pathways included in the study, organized by service, are shown in Table 2. The included pathway characteristics and results are shown in Table 2. Because of statistically significantly correlated variables or rare occurrence, we were unable to include some of the characteristics in three primary categories: the pathway development, implementation, and maintenance processes; secular events (or potential confounders); and pathway documentation. For the pathway processes, characteristics such as leadership and resources invested in development and support were strongly correlated with being the first pathway implemented in a surgical division or with being implemented after 1994. Most secular events occurring around the time of pathway implementation, such as changes in surgical or nursing leadership or the nature of the procedure, occurred in only a few pathways and were too rare to be analyzed. Finally, characteristics of the pathway documents could not be included in the analysis because they were significantly correlated with and related to either calendar time or hospital care characteristics. The distinctiveness of the pathway document from the rest of the chart (using a three-page foldout on yellow, thicker paper that was easy to identify) was augmented in 1994. The complexity and length of the pathway document were significantly correlated with the complexity of care and length of stay for the procedure.

Table 2.

Truth Table, after Removing Variables Statistically Significantly Correlated with Other Variables of Similar Meaning

	Pathways†
	Effective								Not Effective
Domains and Variables*	CEA	HEJ	MAS	RRP	RTP	TES	WHI	Total (%)	AAA	AMV	CAB	CBV	COL	DPA	LCH	LRE	LTP	NEP	PAR	RCY	TAR	THO	THY	THR	TKR	TRA	TUR	Total (%)
Patient characteristics
Age: mean >55‡	1	1	1	1	0	1	1	86	1	1	1	1	1	0	0	1	0	0	0	1	1	1	0	1	1	0	1	37‡
Ethnicity: >85% white	1	0	0	1	0	1	1	43	1	0	1	1	0	0	0	1	0	0	0	1	0	1	0	0	0	1	0	63
Medicaid: >4% of patients	1	1	1	0	0	0	0	43	0	1	0	0	0	0	1	0	1	1	0	0	0	0	1	1	0	1	0	37
HMO: >5% of patients‡	0	1	0	0	0	0	0	14	0	0	1	1	1	1	1	1	1	1	0	0	1	0	1	1	0	0	1	63‡
Comorbidity: >50% with score >0	1	0	0	0	0	0	0	14	1	1	1	1	0	0	0	0	1	0	0	0	1	0	0	0	0	0	0	32
Admitted from emergency room: >4%	1	1	0	0	1	0	0	43	0	0	1	0	1	0	0	0	0	1	0	1	1	0	0	1	0	0	1	42
Postoperative care characteristics
Postoperative LOS: median >8 days	0	1	0	0	1	1	1	57	1	1	0	1	0	0	0	1	1	0	0	1	1	0	0	1	1	0	0	47
Variance in PLOS >100	0	0	0	0	0	0	1	14	1	0	0	1	0	0	0	0	0	1	0	0	0	0	1	0	1	1	0	32
Intensive care unit LOS: median >0‡	0	0	0	0	1	0	1	29	1	1	1	1	1	0	1	1	0	1	0	1	0	0	1	0	1	1	1	68‡
Complication rate: >5%	0	0	0	0	0	1	0	14	1	1	1	1	0	0	0	0	1	0	0	0	1	0	0	0	0	0	0	32
Mortality rate: >=1%‡	0	0	0	0	1	0	1	29	1	1	1	1	1	1	0	0	0	1	0	1	0	0	1	0	1	1	0	58‡
Hospital care characteristics
No prepathway trend towards decreasing length of stay‡	1	1	0	0	1	1	1	71	0	0	1	0	1	0	0	0	0	0	0	0	1	0	0	0	0	1	1	26‡
Procedure volume:>50 cases/year	1	0	1	1	0	0	1	57	1	1	1	1	1	0	1	0	0	1	0	0	0	1	1	1	0	0	1	58
Wards: >1 with >10 cases/year	0	1	1	1	0	1	1	71	0	0	0	0	1	1	1	1	0	1	1	0	0	1	1	1	1	1	1	63
Surgeons: >2 with >10 cases/year	0	0	1	0	1	0	1	43	1	1	1	1	1	1	1	1	1	0	0	0	1	0	0	0	0	1	1	63
Secular events§
Increase in % of patients with HMO: >4%‡	0	0	1	0	1	1	1	57	0	1	0	0	0	0	1	0	0	0	1	0	0	1	1	0	0	1	0	32‡
Increase in procedure volume by >10%‡	1	1	0	1	1	1	1	86	0	0	0	0	1	1	0	1	1	1	1	0	0	1	1	0	1	1	0	52‡
Timing of pathway implementation
First pathway in division‡	0	0	1	1	1	1	1	71	1	0	1	0	0	0	0	1	0	0	1	0	0	0	0	1	0	0	0	26‡
Implementation date in or after 1994	1	1	0	0	1	1	0	43	0	1	0	1	1	1	0	0	1	0	0	0	0	1	1	0	1	1	0	53
Pathway use documentation
Pathway in record: >72% of records	1	1	0	1	0	0	0	43	0	1	0	1	0	1	0	0	0	0	1	1	1	1	1	0	1	1	0	53
Preprinted orders in record: >75% of records	0	0	0	1	1	1	0	43	0	1	1	1	0	0	0	0	1	0	1	1	1	1	1	1	0	1	0	63

^*For all categorical variables: 0=no, 1=yes. Dichotomized variables: 0=lower, 1=higher.

^†Key to abbreviations for pathways:

Effective: CEA, carotid endarterectomy; HEJ, hepatojejunostomy; MAS, mastectomy; RRP, radical retropubic prostatectomy; RTP, renal transplant; TES, transhiatal esophagectomy; WHI, Whipple procedure (pancreaticoduodenectomy).

Not effective: AAA, abdominal aortic aneurysm repair; AMV, aortic/mitral valve replacement; CAB, coronary artery bypass grafting; CBV, coronary artery bypass with aortic/mitral valve replacement; COL, colon resection; DPA, distal pancreatectomy; LCH, laparoscopic cholecystectomy; LRE, liver resection; LTP, liver transplant; NEP, nephrectomy; PAR, parathyroidectomy; RCY, radical cystectomy; TAR, thoracic aneurysm repair; THO, thoracotomy; THY, thyroidectomy; THR, total hip replacement; TKR, total knee replacement; TRA, transverse rectus abdominus myocutaneous (TRAM) breast reconstruction; TUR, transurethral prostatectomy (TURP).

^‡Variables showing more than a 20 percent difference.

^§2-year postpathway period compared with the 2-year prepathway period.

HMO, health maintenance organization; LOS, length of stay, PLOS, postoperative length of stay.

QCA

The primary hypotheses tested in our analysis and their rationales are shown in Table 1. The truth table after removing variables with significant correlations and results of the QCA are shown in Table 2. Our primary finding was that all seven effective pathways had one of the following characteristics: length of stay was not already decreasing for the procedure at the time of pathway implementation (71 percent of pathways), or it was the first pathway implemented on a service (71 percent of pathways). Both of these characteristics were less common among the 19 ineffective pathways (only 26 percent had length of stay not already decreasing at implementation, and 26 percent were the first pathway implemented on a service). Of the two effective pathways that were not the first implemented on a service, one (hepaticojejunostomy) was on the only service with two effective pathways, and the other (carotid endarterectomy) was on a service with a pathway developed several years earlier (abdominal aortic aneurysm) that had faced some difficulties in implementation.

We also found that effective pathways tended to be for conditions with lower complexity/severity of illness. This was manifested by lower mortality (only 29 percent of effective versus 58 percent of ineffective pathways with mortality ≥1 percent) and lower intensive care unit length of stay (29 percent of effective versus 68 percent of ineffective pathways with median intensive care unit stay >0 days). In addition, effective pathways tended to have somewhat lower complication rates, patient comorbidities, and variation in postoperative length of stay.

Although there were trends toward higher rates of pathway inclusion in the medical record and documentation of pathway completion for effective compared with ineffective pathways, these differences were relatively small and high rates of documented pathway use were not necessary for pathway effectiveness. For patients treated for conditions associated with effective pathways, 43 percent had pathway documentation included in >72 percent of medical records reviewed, as compared with 53 percent of ineffective pathways.

Finally, we found several characteristics that appeared to show potential associations with effectiveness unrelated to our primary hypotheses. Effective pathways were less likely to have more than 5 percent of patients with HMO insurance (14 percent of effective versus 63 percent of ineffective pathways) and were more likely to have more than a 4 percent increase in the percentage of patients with HMO insurance (57 percent versus 32 percent of ineffective pathways) between the 2-year pre- and postpathway periods. Effective pathways were also more likely to have greater than 10 percent increase in procedure volume from the 2-year pre- to the 2-year postpathway period (86 versus 52 percent of ineffective pathways). Finally, effective pathways were more likely to have a mean patient age of over age 55 (86 percent of effective versus 37 percent of ineffective pathways).

DISCUSSION

In this exploratory analysis, we found that the effectiveness of a hospital's comprehensive critical pathway program may have been limited to certain types of situations. We found that the key characteristics qualitatively associated with surgical critical pathways' effectiveness for reducing postoperative length of stay were that they were either the first pathway implemented on a surgical service or did not have a preimplementation trend toward decreasing length of stay. Although these characteristics were much more frequently associated with the seven effective pathways than with the 19 that were ineffective, neither characteristic was necessary for pathway effectiveness. We also found that lower severity of illness may be an important factor for pathway success. Because high rates of pathway use were not strongly associated with effectiveness, the mechanism by which critical pathways may reduce length of stay is unclear.

Our finding that most of the pathways associated with reducing length of stay were the first pathway implemented on a service has several potential explanations. The first pathway in each service was likely to have more resources invested in its development and in building support, awareness, and knowledge, both of the pathway and of the goal for a change in practice. Because the first pathway chosen tended to be implemented when there was greatest interest among clinicians for pathway development, this may also be a marker for elements of clinician support, such as having a strong physician or nurse champion, that may be important for effective quality improvement (Thomson O'Brien et al. 2000; Bradley et al. 2001). In addition, changes in structure and practice that developed through the implementation of the first pathway may have diffused to other related procedures and contributed to trends toward lower length of stay, therefore reducing the apparent impact of later pathways.

Several of our findings suggest that the pathway development and implementation process might have a role in reducing length of stay. High rates of documented pathway use were not necessary for pathway effectiveness, and some effective pathways had very low rates of pathway documentation. Therefore, other mechanisms of pathway implementation, such as changing the culture regarding expectations about length of stay and standard of care for patients undergoing a procedure, may have been responsible for increased pathway effectiveness. These findings are consistent with a previous study that found that much of the change from a critical pathway occurred during the development period (Holtzman, Bjerke, and Kane 1998). It is also important to note that the changes attributed to critical pathways were small compared with the trends in length of stay over the 10-year study period. Change in medical practice can be the most important factor reducing length of stay (Sloan and Valvona 1986).

We found that rates of documented critical pathway use were highly variable, and that high rates of documented pathway use were rare, despite our conservative definition of pathway use. Because most pathways did not require that the nurse fill in any patient information on the pathway document, it was impossible to tell whether the nurse actually read or used the pathway or simply initialed in the appropriate box. The relatively low rates of pathway utilization are consistent with rates of use of quality improvement methods in other clinical, nonresearch settings (Cohen 2000) and may have reduced the impact of the pathways. The critical pathway interventions did not include a nurse to monitor use of the critical pathway for eligible patients, as occurs in many research studies of quality improvement (Marrie et al. 2000). Other factors may have contributed to low rates of documented use for certain pathways, including inadequate physician leadership in the implementation process, inadequate education, or physician unawareness of length of stay norms or disagreement with the recommended shorter length of stay (Morgan and Beech 1990).

Postoperative and hospital care characteristics may also have affected the success of the critical pathways, although results that were not associated with one of our key hypotheses should be interpreted with more caution. As we hypothesized, pathways were more likely to be effective when applied to procedures with lower severity/complexity of illness. Because pathways tend to be relatively inflexible and oriented toward patients with the predictable course of care, they may not work well where care is more variable, as demonstrated by a related study in the intensive care unit setting (Berenholtz et al. 2001).

Some of the factors we studied may also have been potential confounders of determinations of pathway effectiveness and highlight limitations of before:after observational studies. For example, decreasing length of stay at the time of implementation might potentially reflect effects of the pathway development process and decrease the power to determine effect at the time of pathway implementation. Managed care may also be a confounder of the determination of pathway effectiveness. Critical pathway effectiveness was associated with both initially low baseline rates and with upward trends in the proportion of patients with HMO insurance. In other words, new managed care pressures where they had not existed previously, such as standards for length of stay for surgical procedures, might have provided an outside stimulus to reduce length of stay, coincidental in timing to pathway implementation. This is consistent with the results of Pearson et al. (2001), which suggest that outside managed care pressures (or perceptions of these pressures) may be an important influence on length of stay. However, rates of managed care use were quite low, and these pressures might be mitigated in the state of Maryland because of the regulated, all-payer reimbursement formulas utilized here (Health Services Cost Review Commission 2002). Substantial increases in procedure volume were also more common among effective than ineffective critical pathways. This is consistent with a study of variation in length of stay for myocardial infarction showing that hospitals with lower caseloads tended to have higher lengths of stay (Chen and Naylor 1994).

Our study has several limitations as well as significant strengths. Although the qualitative comparative, case-based approach allowed us to explore several hypotheses and many pathways, the qualitative approach and sample size make our results exploratory at best. We did take several measures to try to maximize the validity of our results, including testing a priori hypotheses rather than just exploring the data, looking for confirmatory results to support our findings, and evaluating potential confounders. Our results are based on individual pathway assessments of effectiveness, and sample sizes for some pathways were relatively small. Therefore, some less common pathways may have been misclassified as ineffective, and there may be potential bias against these pathways. The small number of cases also limited our power for evaluating factors associated with effectiveness; there are certainly many more factors than those we have identified.

Although limiting the study to a single institution may decrease the applicability of our findings to other settings and did not allow us to study organizational factors or characteristics of pathway documents, it allowed us to study pathways developed in similar environments as part of a comprehensive pathway program. The large numbers of procedures addressed by the pathway program allowed us to conduct what is to our knowledge the first study of factors influencing pathway effectiveness. Some of the factors that we studied had several potential etiologies, and we were unable to evaluate some important factors because they were significantly correlated with other factors in this program. Finally, because of lack of documentation on the pathway forms, we were not able to assess any of the pathways' actual mechanisms of effectiveness. These mechanisms, particularly whether there was an actual change in the structure or process of care that affected length of stay, may be an important factor associated with pathway effectiveness (Pearson 1998).

Finally, we only assessed one outcome: length of stay. This is an important outcome and the most frequent measurement in published studies of critical pathway effectiveness. However, pathways may be designed to improve and may affect many other aspects of patient care. The factors we have identified might not be associated with other important outcomes, such as patient satisfaction and long-term health, which are important for assessing the quality of care (Morgan and Beech 1990).

The results of our study suggest that critical pathway programs may have limited effectiveness and may be more applicable to certain situations than others. Given the exploratory nature of our analysis, the factors we have identified require additional investigation in other settings to provide more evidence on whether they are associated with effectiveness. However, our results may provide some guidance to institutions deciding whether or where to invest their resources in implementing critical pathways. The success of quality improvement efforts may depend as much on the circumstances under which they are undertaken as on the particular quality improvement method chosen (Bradley et al. 2001). Our results also provide further evidence that the development process of a quality improvement program may lead to changes in practice, and suggest that more research into the mechanism of the effects of quality improvement initiatives might help improve the efficiency of these programs.