The Limited Incorporation of Economic Analyses in Clinical Practice Guidelines

Abstract

BACKGROUND

Because there is increasing concern that economic data are not used in the clinical guideline development process, our objective was to evaluate the extent to which economic analyses are incorporated in guideline development.

METHODS

We searched medline and HealthSTAR databases to identify English-language clinical practice guidelines (1996–1999) and economic analyses (1990–1998). Additional guidelines were obtained from The National Guidelines Clearinghouse Internet site available at http://www.guideline.gov. Eligible guidelines met the Institute of Medicine definition and addressed a topic included in an economic analysis. Eligible economic analyses assessed interventions addressed in a guideline and predated the guideline by 1 or more years. Economic analyses were defined as incorporated in guideline development if 1) the economic analysis or the results were mentioned in the text or 2) listed as a reference. The quality of economic analyses was assessed using a structured scoring system.

RESULTS

Using guidelines as the unit of analysis, 9 of 35 (26%) incorporated at least 1 economic analysis of above-average quality in the text and 11 of 35 (31%) incorporated at least 1 in the references. Using economic analyses as the unit of analysis, 63 economic analyses of above-average quality had opportunities for incorporation in 198 instances across the 35 guidelines. Economic analyses were incorporated in the text in 13 of 198 instances (7%) and in the references in 18 of 198 instances (9%).

CONCLUSIONS

Rigorous economic analyses may be infrequently incorporated in the development of clinical practice guidelines. A systematic approach to guideline development should be used to ensure the consideration of economic analyses so that recommendations from guidelines may impact both the quality of care and the efficient allocation of resources.

Evidence-based practice guidelines have been advocated as a mechanism for reducing practice variation and improving the quality of care. Systematic approaches to guideline development have been defined and advocated by leading medical journals and authorities.^1–5 Recently, a review of the guideline development process has outlined several deficiencies, including the omission of economic data.^6,7

In an era of increasingly constrained financial resources, allocation of scarce medical resources has become an important factor in health policy decision making. Such policy should rely on the best available evidence as promulgated in evidence-based practice guidelines.⁸ Evidence-based medicine and economic analyses address the values of effectiveness and efficiency critical to allocation decision making,⁹ and economic analyses are an important type of evidence that could inform the practice of evidence-based medicine.¹⁰ Several sources, including the Consensus Statement on the Role of Cost-Effectiveness Analysis in Health and Medicine, recommend that cost-effectiveness analyses be used as an aid to decision makers¹¹ and that economic data should be incorporated into guidelines where possible.¹²

Because 80% or more of healthcare costs may be directly related to clinical decisions, safely controlling health care costs will require influencing clinical decisions. Guidelines are often used to inform and improve clinical decision making. If guidelines are to address economic issues relevant to clinical decision making, they should be informed by the highest quality economic evaluations available in the published literature. Our objective was to evaluate the extent to which economic analyses are incorporated in the guideline development process.

METHODS

Literature Review and Selection Process

A systematic search of the medline and HealthSTAR computerized bibliographic databases was performed to identify English-language clinical practice guidelines using the MESH heading “Practice Guidelines” for the years 1996–1999. Guidelines were also obtained from The National Guidelines Clearinghouse Internet site (http://www.guideline.gov).

English-language economic analyses—articles reporting data on direct and indirect costs—were identified through a systematic search of the databases using the MESH headings “Cost-Benefit Analysis” or “Costs and Cost Analysis” for the years 1990–1998. The search time frames for economic analyses and guidelines were chosen such that the economic analyses would likely predate the clinical guidelines. Duplicates, editorials, letters, and reviews were excluded. Then, on the basis of titles, the clinical practice guidelines and economic analyses were grouped by condition within 5 clinical categories: acute therapy, chronic therapy, risk factor reduction, screening, and surgical therapy. These categories were selected in an attempt to include guidelines and economic analyses across a broad spectrum of medical conditions. The medical condition with the greatest number of guidelines and at least 10 economic analyses was selected from each of the 5 categories.

After the 5 conditions were selected, a second literature search was conducted using the same search strategy as before but also including the medical conditions' MESH headings. Four researchers under the supervision of 2 health economists reviewed the identified guidelines and economic analyses and finalized the list for further testing on the basis of several criteria. These 4 researchers included a pharmacist trained in pharmacoeconomics (who was the lead investigator), a physician/health services researcher, a health economist, and an epidemiologist. All 4 researchers had training and experience in health services research methods. Guidelines were eligible for analysis if they: 1) met the Institute of Medicine definition of “systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances,”⁶ 2) were described clearly and made recommendations as to appropriate care,^13–15 and 3) addressed a topic included in an economic analysis. Economic analyses were eligible for analysis if they met the following inclusion criteria: 1) assessed interventions specifically addressed in the guideline and 2) predated the guideline publication by 1 or more years. Hand searches of bibliographies of the guidelines that were accepted for analyses further identified additional economic analyses not identified in our search.

Scoring of Economic Analyses

Eligible economic analyses for the 5 conditions were then assessed for methodological quality using a structured scoring system (Table 1). Since no published checklist or appraisal system enables a quantitative ranking of the quality of economic analyses, we developed a compendium of questions from the most well-known published checklists for evaluating economic studies.^16–20 Items were selected from the existing instruments if they represented a domain present in all existing checklists, and if the item could be operationalized into a yes/no question without changing the content of the item. To ensure content validity, each domain represented in the published checklists was also represented in the final composite instrument. The 4 researchers who finalized the list of guidelines and economic analyses also evaluated the quality of economic analyses with the scoring system. In a pilot test, pairs of reviewers (the lead investigator paired with each of the 3 other reviewers) assessed articles together and resolved the discrepancy among their ratings through several rounds of discussion overseen by the health economist. The agreement between 3 pairs of reviewers was assessed on 3 separate samples of 8 randomly selected economic analyses. The κ values for the samples were 0.87, 0.88, and 0.78. Subsequently, the economic analyses from the 5 conditions were divided among the 4 reviewers, with the lead investigator evaluating articles from 2 conditions. Mean and median scores were calculated for all economic analyses. Analyses scoring higher than the mean were defined as above-average quality.

Table 1.

Scoring System for Economic Analyses^*

1.	Was the study objective clearly defined and measurable?	Y / N
2.	Was the perspective for the analysis stated clearly?	Y / N
3.	Was the pharmacoeconomic tool used appropriate for the study?	Y / N
4.	Was this the tool that was actually used?	Y / N
5.	Did the study provide a comparison of alternative treatments for patients with the same clinical condition?	Y / N
6.	Was a complete description of the alternatives given?	Y / N
7.	Was the evidence of efficacy established through randomized trials?	Y / N
8.	Was this evidence of efficacy supplemented by evidence of effectiveness applicable to the patient population or subgroups considered in the study?	Y / N
9.	Were the methods and analysis displayed in a clear and transparent manner?	Y / N
10.	Were the components of the numerator and denominator displayed?	Y / N
11.	Were costs and consequences measured in the appropriate physical units?	Y / N
12.	Were costs and outcomes relevant to the analysis tool chosen?	Y / N
13.	If the study time was greater than one year, were costs and consequences that occur in the future discounted to their present value?	Y / N
14.	Equity assumptions made during analysis: if quality-adjusted life years gained by an individual were considered equal, is this acceptable?	Y / N
15.	Were the results practical for medical decision makers?	Y / N
16.	Were sensitivity analyses performed by incorporating ranges of values for variables with uncertainty?	Y / N
17.	Were the assumptions and limitations of the study discussed?	Y / N
18.	Was an incremental analysis performed?	Y / N
19.	Were the conclusions of the study justified?	Y / N
20.	Can the conclusions be generalized to other populations?	Y / N

^*Questions adapted from guidelines for economic evaluation.¹⁶–²⁰

Analysis

Each guideline was then evaluated as to whether eligible economic analyses were incorporated into the eligible guidelines on the basis of the following criteria: 1) the economic analysis or the result of the economic analysis was mentioned in the body of the text, or 2) the economic analysis was listed as a reference. When neither of these criteria was met, the developers of each guideline were then contacted to determine if any economic analyses were incorporated in the guideline development process. Four telephone or e-mail attempts were made to contact the guideline committee/task force chairperson or other members knowledgeable about the guideline development process. A standardized questionnaire was administered to each contact to determine if there were records of additional references used in the development process that could be reviewed for the analysis. If no additional records were available, the contact was asked if there had been any consideration of economic issues by members of the task force. If the contact indicated that there had been such consideration, a list of eligible economic analyses was sent for review. Successful contact (yes/no), consideration of economic issues (yes/no/no recollection), and consideration of eligible economic analyses (yes/no) were documented.

The incorporation of economic analyses in guidelines was evaluated in 2 ways. The first approach used guidelines as the unit of analysis or denominator, and determined the proportion of guidelines that incorporated economic analyses in the text or references. The second approach used economic analyses as the unit of analysis, with a denominator equal to the number of opportunities for economic analyses to be incorporated into guidelines. Economic analyses with opportunities for incorporation were those that met the inclusion criteria described above, i.e., addressing a topic within, and predating, the guideline. Some economic analyses had opportunities for incorporation in more than 1 guideline within each clinical category.

We tested whether there was variation between the conditions studied in the degree to which economic analyses were incorporated into guidelines. Differences between groups were tested using χ² and Fisher exact tests. Because there may be a time lag between publication of an economic analysis and the incorporation into a guideline, we performed a secondary analysis comparing the rates of incorporation of economic analyses into guidelines for those economic analyses published 1 or more years prior to guideline development and 2 or more years prior to guideline development.

RESULTS

The systematic literature review resulted in 6,722 candidate practice guidelines (356 from the National Guideline Clearinghouse and 6,366 from medline) and 8,096 candidate economic analyses (Fig. 1). After excluding duplicates, letters, editorials, and reviews and classifying by medical condition on the basis of the title, there were a total of 2,121 candidate practice guidelines and 2,377 candidate economic analyses. The medical conditions with the most candidate practice guidelines and at least 10 candidate economic analyses for each category were acute myocardial infarction,^21–56 asthma,^57–90 smoking cessation,^91–113 colorectal cancer,^114–140 and breast cancer.^141–166 The second literature search with the addition of each condition's MESH heading resulted in the identification of additional guidelines and economic analyses. A total of 180 guidelines and 353 economic analyses were subjected to further review (Fig. 1).

FIGURE 1.

Results of literature search and grouping process to identify the top medical conditions with the most guidelines and at least 10 economic analyses for each category.

A total of 35 guidelines across the 5 conditions met the inclusion criteria (Table 2). Hand searches of their bibliographies resulted in 14 additional economic analyses. Among the 367 economic analyses, 112 met the inclusion criteria (Table 2). Fourteen of these guidelines (40%) were developed by a governmental agency, 11 (31%) were developed by a medical society, 8 (23%) were developed by a nonprofit nonmedical society, and 2 (6%) were published in peer-reviewed medical journals but were not sponsored by an agency. The mean quality score for the 112 eligible economic analyses was 69% (median, 72%). Sixty-three of the 112 eligible economic analyses (56%) had scores greater than 69% and were defined as above-average quality. The distribution of quality scores was similar to that found in previous work.¹⁶⁷

Table 2.

Number of Guidelines and Economic Analyses Identified and Included in the Analysis

	Category
	Acute Therapy	Chronic Therapy	Risk Factor Reduction	Screening	Surgical Therapy	Total
Condition	Myocardial infarction	Asthma	Smoking cessation	Colorectal cancer	Breast cancer
Included
Guidelines, n	621–23,28,49,50	557–60,63	691–93,106,112	6115–118,130,131	12141–147,149,153,161,162,164	35
Economic analyses, n	3024–27,29–48,51–56	2961,62,64–90	1894–105,107–111,113	21114,119–129,132–140	14148,150–152,154–160,163,165,166	112
Economic analyses scoring > mean*	17	13	10	16	7	63

^*The mean score for all economic analyses is 69% (median, 72%).

Incorporation of Economic Analyses by Guidelines

Using the guidelines as the unit of analysis, the incorporation of economic analyses by guideline is reported across the 5 conditions in Table 3. Eleven of 35 guidelines (31%) incorporated at least 1 eligible economic analysis in the text, and 15 (43%) incorporated at least 1 eligible economic analysis in the references. Nine (26%) of the guidelines incorporated at least 1 economic analysis of above-average quality in the text, and 11 (31%) incorporated at least 1 eligible economic analysis of above-average quality in the references. The 5 conditions were also analyzed individually, revealing statistically significant (P < .001) variation in the degree to which economic analyses were considered in guideline development (Table 3). We also evaluated variation when only including economic analyses of above-average quality. The percentage of economic analyses incorporated by guidelines across the 5 conditions ranged from 0 of 12 (0%) for breast cancer to 4 of 6 (67%) in smoking cessation and colorectal cancer and the variation was statistically significant (P < .005).

Table 3.

Incorporation of Economic Analyses by Guidelines

	Category
	Acute Therapy	Chronic Therapy	Risk Factor Reduction	Screening	Surgical Therapy	Total
Condition	Myocardial infarction	Asthma	Smoking cessation	Colorectal cancer	Breast cancer
Any economic analysis
Guidelines, n	6	5	6	6	12	35
With economic analyses in text, n (%)	3 (50)	0 (0)	4 (67)	4 (67)	0 (0)	11 (31)
With economic analyses in references, n (%)	4 (67)	3 (60)	4 (67)	4 (67)	0 (0)	15 (43)
With economic analyses considered by task force, n (%)	5 (83)	3 (60)	4 (67)	4 (67)	0 (0)	16 (46)
Economic analyses scoring > mean*
Guidelines, n	6	5	6	6	12	35
With economic analyses in text, n (%)	1 (17)	0 (0)	4 (67)	4 (67)	0 (0)	9 (26)
With economic analyses in references, n (%)	2 (33)	1 (20)	4 (67)	4 (67)	0 (0)	11 (31)
With economic analyses considered by task force, n (%)	3 (50)	1 (20)	4 (67)	4 (67)	0 (0)	12 (34)

^*The mean score for all economic analyses is 69% (median, 72%).

Of the 20 guidelines failing to incorporate an eligible economic analysis into the body of the text or in the references, 12 of the chairpersons (60%) were successfully interviewed. For 1 of 12 guidelines (8%), the chairperson reported consideration of economic analyses by the task force in guideline development

Incorporation of Economic Analyses by Opportunities for Incorporation

Using the economic analyses as the unit of analysis, the incorporation of economic analyses by opportunities for incorporation is reported across the 5 conditions in Table 4. One hundred twelve eligible economic analyses had an opportunity for incorporation in 300 instances across the 35 guidelines. Economic analyses were incorporated in the body of the text and in the references in 16 of 300 (5%) and 29 of 300 (10%) instances, respectively. Similarly, 63 eligible economic analyses of above-average quality had an opportunity for incorporation in 198 instances across the 35 guidelines. Economic analyses were incorporated in the body of the text and in the references in 13 of 198 (7%) and 18 of 198 (9%) instances, respectively. Twenty-two of the 112 economic analyses (20%) and 16 of the 63 economic analyses of above-average quality (24%) were incorporated into any of the guidelines meeting our inclusion criteria.

Table 4.

Economic Analyses Guidelines by Opportunities for Incorporation

	Category
	Acute Therapy	Chronic Therapy	Risk Factor Reduction	Screening	Surgical Therapy	Total
Condition	Myocardial infarction	Asthma	Smoking cessation	Colorectal cancer	Breast cancer
Economic analyses
Opportunities for economic analyses to be incorporated in guidelines, n	94	35	36	92	43	300
Economic analyses incorporated in text, n (%)	4 (4)	0 (0)	4 (11)	8 (9)	0 (0)	16 (5)
Economic analyses incorporated in references, n (%)	9 (10)	3 (9)	7 (19)	10 (11)	0 (0)	29 (10)
Economic analyses scoring >mean*
Opportunities for economic analyses to be incorporated in guidelines, n	61	18	25	68	26	198
Economic analyses incorporated in text, n (%)	2 (3)	0 (0)	4 (16)	7 (10)	0 (0)	13 (7)
Economic analyses incorporated in references, n (%)	3 (5)	1 (6)	6 (24)	8 (12)	0 (0)	18 (9)

^*The mean score for economic analyses is 69% (median, 72%).

As shown in Table 4, risk reduction guidelines (as represented by smoking cessation) had the highest percentage of economic analyses incorporated (i.e., 19%). Eighteen economic analyses had opportunities for incorporation in 36 instances across the 6 risk reduction guidelines. Economic analyses were incorporated in the text and in the references in 4 of 36 (11%) and 7 of 36 (19%) instances, respectively. Smoking cessation also had the highest percentage of analyses of above-average quality incorporated. Ten analyses of above-average quality had opportunities for incorporation in 25 instances across the 6 smoking cessation guidelines. Economic analyses were incorporated in the text and in the references in 4 of 25 (16%) and 6 of 25 (24%) instances, respectively.

Analysis by Number of Years Between Publication of Economic Analysis and Guideline

We also compared the rates of incorporation of economic analyses into guidelines for economic analyses published 1 or more years prior to guideline development with those published 2 or more years prior to guideline development. There were no significant differences between economic analyses published 1 or more years prior to guideline development and those published 2 or more years prior to guideline development for incorporation in the body of guideline (5.3% vs 4.6%; P = .69) or in the references (9.6% vs 10.5%; P = .76).

COMMENT

Economic analyses appear to be infrequently incorporated in the development of clinical practice guidelines. Twenty of the 35 guidelines (57%) failed to incorporate any eligible economic analysis into the text or in the references. Ninety of the 112 economic analyses (80%) failed to be incorporated into any of the guidelines meeting our inclusion criteria. These findings are inconsistent with recommendations that economic analyses should be taken into account when developing guidelines.

The paucity of high-quality economic data within the clinical guideline development process makes it increasingly difficult for decision makers to allocate limited health care resources efficiently. It is unlikely that this is due to the lack of higher quality, published economic analyses, based on the number identified in this study. Certain clinical conditions were more likely to incorporate economic analyses than others. Conditions amenable to risk factor reduction or preventative care, such as smoking cessation and colorectal cancer screening, were more likely to incorporate economic analyses than, for example, surgical therapy for breast cancer. This suggests there may be greater rationale to justify program benefits economically when the benefits will occur in the future compared to treatments with more immediate impact.

There are some limitations to this analysis. We attempted, a priori, to stratify our analysis by the quality of the economic analysis because we do not believe that guidelines should incorporate the results of economic analyses that do not meet standards for reporting and methodological rigor. Because we were unable to identify a valid scoring system for quality,¹⁶⁷ we developed one from existing instruments.^1–20 This composite system assumes that each item carries equal weight, and it has not been formally validated. While construct validity was not tested in this study, we believe that the composite instrument has face, or content, validity since each domain from existing checklists is represented and the distribution of quality scores obtained with this instrument appears similar to the distributions found in other studies using different methods. Moreover, there was high inter-rater agreement, suggesting adequate reliability. A limitation of our composite instrument is that items that are relevant to only specific types of economic analyses may be lacking, such as a specific item about utility values. Further study of the construct and discriminant validity of the composite scoring system appears warranted. One reason for the lack of incorporation of economic analyses in guidelines may be that there is no standardized method for developers of guidelines to determine which economic analyses merit incorporation on the basis of their quality.

Another limitation of our analysis is that we did not grade the quality of the practice guidelines. While appraisal systems exist to identify certain elements that guidelines should include, there is no acceptable system for ranking or comparing the relative quality of clinical practice guidelines. In light of this, a priori we assumed that if economic analyses were considered in the guideline development process, consideration would most likely occur in high-profile guidelines developed by national societies and published in the peer-reviewed literature.

A third limitation is that we used a very specific (MESH headings) rather than sensitive (keywords) search strategy, recognizing that we wanted a “sample” of the universe of published economic analyses and guidelines. Thus, our sample may not be completely representative of all relevant articles. Moreover, because this analysis included only guidelines published in peer-reviewed journals or on the National Guidelines Clearinghouse website, it may be susceptible to publication bias.

Furthermore, although we attempted to analyze a representative sample of published guidelines, we nonetheless submitted a sample of 5 conditions to evaluation. Since we only examined 1 condition in each category, all of the analyses performed in the current study were based on relatively small samples. This may influence the generalizability of the findings. However, the conditions chosen were those with the greatest number of clinical guidelines and at least 10 published economic analyses predating the guidelines. We also recognized that the recollection of the guideline committee regarding the consideration of economic analyses by the group might not be completely reliable. Nevertheless, we felt that this was the only available method for obtaining information regarding whether the group developing the guideline did not consider any economic analyses or simply could not identify any economic analyses.

Finally, it may be considered unlikely that economic analyses published only 1 year prior to guideline development would be incorporated into a practice guideline. We explored this by comparing the rates of incorporation of economic analyses into guidelines for economic analyses published 1 or more years prior to guideline development and 2 or more years prior to guideline development. When we limited our analysis to only economic analyses published 2 or more years prior to guideline development, the results were unchanged.

In an era of increasing scrutiny of the costs and benefits of medical interventions, clinical practice guidelines are needed to inform practitioners about the most effective and efficient management strategies. Because there is a proliferation of economic evaluations and methodological standards for these evaluations in the literature, economic findings should be incorporated in the guideline development process. However, incorporating economic analyses of poor quality in the guideline could potentially decrease the value of the guideline and mislead the users. Therefore, practical standardized methods for assessing the quality of economic analyses appear to be needed. We are concerned that guidelines and ensuing policy recommendations may not be informed by economic evaluations that elucidate and promote the most efficient utilization of health care resources. Further research is necessary to identify barriers to the incorporation of economic analyses in the guideline development process.