Improving the quality of abstract reporting for economic analyses in oncology

Abstract

Background

The increasing cost of cancer drugs underscores the importance of economic analyses. Although guidelines for abstract reporting of randomized controlled studies and phase i trials are available, similar recommendations for conference abstracts of economic analyses are lacking. Our objectives were

• to identify items considered to be essential in abstracts of economic analyses;

• to evaluate the quality of abstracts submitted to the American Society of Clinical Oncology (asco), the American Society of Hematology (ash), and the International Society for Pharmacoeconomics and Outcomes Research (ispor) meetings; and

• to propose guidelines for future abstract reporting at conferences.

Methods

Health economic experts were surveyed and asked to rate each of 24 possible abstract elements on a 5-point Likert scale. A scoring system for abstract quality was devised based on elements with an average expert rating of 3.5 or greater. Abstracts for economic analyses from asco, ash, and ispor meetings were reviewed and assigned a quality score.

Results

Of 99 experts, 50 (51%) responded to the survey (average age: 53 years; 78% men; 54% from the United States, 28% from Europe, 18% from Canada). In total, 216 abstracts were reviewed: asco, 53%; ash, 14%; and ispor, 33%. The median quality score was 75, but notable deficiencies were observed. Cost perspective was reported in only 61% of abstracts, and time horizon was described in only 47%. Abstracts from recent years demonstrated better quality scores. We also observed disparities in quality scores for various cancer sites (p = 0.005).

Conclusions

The quality of conference abstracts for economic analyses in oncology has room for improvement. Abstracts may be enhanced using the guidelines derived from our survey of experts.

1. INTRODUCTION

Economic analyses are helpful in evaluating the relative costs and benefits of a health intervention. In the face of budget constraints, results of economic analyses can assist decision-makers in allocating resources effectively to better maximize net public health benefit^1–3.

Economic analyses are particularly important in the field of oncology because of the dramatic rise in the cost of many new cancer treatments. For instance, the addition of new molecularly targeted agents such as bevacizumab to conventional chemotherapy regimens has been associated with an increase in the cost of treatment by as much as US$700 to US$1000 per 100 mg of drug.

In oncology, economic analyses are frequently presented first in abstract form at major conferences, such as the annual meetings of the American Society of Clinical Oncology (asco), the American Society of Hematology (ash), and the International Society for Pharmacoeconomics and Health Outcomes Research (ispor). Importantly, data presented in such abstracts may be accepted or endorsed by the general oncology community before actual publication of the full report in a scientific journal, in part because of time delays incurred with manuscript preparation and peer review and also because of a sense of urgency to adopt new interventions that demonstrate cost-effectiveness^4–6. Studies have confirmed that there can be significant delays between abstract presentation and full publication. Krzyzanowska et al.⁷ found that the median time between abstract presentation of a randomized clinical trial and its final publication to be approximately 2.7 years; at 5 years, 26% of trials remain unpublished. The reason most frequently cited by authors for not publishing is lack of time, funds, or other resources. Although data presented in conference abstracts reflect preliminary findings, they may ultimately affect patient care, especially if final results are never published.

Despite the valuable information that economic analyses provide, there are currently no guidelines for abstract reporting of economic analyses at various clinical conferences. Reporting guidelines are available for abstracts of randomized controlled trials, meta-analyses, and phase i studies^8–12. Guidelines for the reporting of economic analyses were proposed by Drummond and various colleagues as early as 1980^13–16. Unfortunately, the proposed guidelines are not specific to oncology, and thus they have not been adopted for routine use by the clinical community. However, quality assurance measures are clearly needed for abstracts of economic analyses, particularly given the growing number of these abstracts being presented at major clinical conferences in recent years. We therefore set out to identify essential elements of such abstracts and to evaluate the quality of economic analysis abstracts submitted to the annual meetings of asco, ash, and ispor based on the identified elements. We then proposed guidelines that can be used in the future to improve the quality of abstract reporting for economic analyses in oncology.

2. METHODS

2.1. General Overview of Study Design

We began by surveying health economic experts, asking respondents to rate each of 24 possible economic analysis elements on a 5-point Likert scale (1 = unnecessary to include; 3 = should include if possible and space permitting; 5 = absolutely required in the abstract). Based on items receiving an average expert rating of 3.5 or more out of 5, we devised a scoring system for abstract quality that ranged from 0 (“poor”) to 100 (“excellent”). Using the scoring system, we evaluated the quality of abstracts submitted to recent asco (1997–2009), ash (2004–2009), and ispor (1997–2009) meetings. Reporting guidelines for future abstract reporting were then developed based on the results from our survey of experts. Research ethics approval was obtained before the start of this study.

2.2. Survey of Health Economic Experts

We used e-mail to conduct an electronic survey of health economic experts selected from the membership directory of ispor. All members of ispor were included in our survey panel because we wanted to consider a broad spectrum of economic experts and not to limit our sample to a subset, which could lead to selection bias. The contact information for each potential respondent was obtained from the official Web page of the expert’s affiliated institution. We used recommendations from the Panel on Cost-Effectiveness in Health and Medicine as the primary source for developing a list of items that might potentially be important to include in an economic analysis abstract. The Panel consisted of experts in economic analyses, clinical medicine, ethics, and health outcomes measurement and had been convened by the U.S. Public Health Service in the early 1990s¹⁰.

A list of 24 possible abstract elements was identified (Table i). Survey recipients were asked to rate each item on our 5-point Likert scale. To improve the survey response rate, an e-mail reminder to complete the survey was sent at 2 and at 4 weeks after the initial contact.

TABLE I.

Potential items for abstracts of economic analyses in oncology

Potential abstract item	Average expert rating
Background and rationale of the cost problem	3.80
Type of cost analysis (for example, cea, cua, cba)	4.63
Description of intervention	4.54
Description of comparator	4.57
How and why this particular comparator was selected	3.06
Target population for the intervention	4.23
Time horizon	4.04
Perspective of the cost analysis	4.00
The costs that were included	3.61
How the cost data were derived	3.04
How the denominator was measured (for example, utility/qaly/daly)	4.33
Actual cost-effectiveness or cost–utility ratios (for example, $100,000/lyg)	4.41
Use of incremental cost-effectiveness ratios	4.04
Whether sensitivity analyses were conducted	3.58
Type of sensitivity analyses (for example, deterministic, probabilistic)	3.25
Whether a cost-effectiveness acceptability curve was derived	2.56
A prior definition of budget threshold	2.67
Concluding statement and interpretation of cost-effectiveness	4.15
Source of funding	3.33
Location and setting of the study	3.67
Currency (for example, CA$/US$)	4.15
Whether discounting was used	3.69
Base year of cost adjustment	3.25
Limitations of the analysis	3.30

cea = cost-effectiveness analysis; cua = cost–utility analysis; cba = cost–benefit analysis; qaly = quality-adjusted life year; daly = disability-adjusted life year; lyg = life-year gained.

2.3. Development of Quality Scoring System and Evaluation of Conference Abstracts

The mean expert rating for each of the 24 possible abstract items was determined by summing the ratings for a particular item and then dividing the total by the number of responses received. A scoring system for abstract quality was subsequently devised by categorizing the abstract elements into four groups:

• Items that are absolutely essential for inclusion in abstracts (mean expert rating: ≥4.50)

• Items that should be reported (mean expert rating: 4.00–4.49)

• Items that might be reported if space permits (mean expert rating: 3.50–3.99)

• Items that are unlikely to be informative (mean expert rating: <3.50).

We subsequently reviewed conference abstracts of economic analyses from the asco, ash, and ispor meetings and graded their quality based on whether the abstracts contained the 24 essential and nonessential items. As a first step toward grading the quality of the abstracts, 3 points were assigned for the presence of absolutely essential items, 2 points for items that should be reported, 1 point for items that might be reported if space permits, and 0 points for items that are unlikely to be informative. The overall quality score of each abstract was then converted to a normative score out of 100 to facilitate interpretation (0 = poor quality; 100 = excellent quality).

To review the proceedings from the asco (1997– 2009), ash (2004–2009), and ispor (1997–2009) annual meetings, we visited the conference Web sites and their corresponding abstract databases. Abstracts were identified using key words such as “economic analysis,” “cost-effective analysis,” “cost–utility analysis,” “cost-minimization analysis,” and “cost–benefit analysis.”

All abstracts were reviewed using a standardized form containing the 24 abstract elements (Table i). We piloted-tested the form on a sample of 10 abstracts. One author (MYH) performed the data collection; a second author (WYC) validated the accuracy for a 5% random sample of abstracts. Disagreements were resolved by consensus.

Parameters collected from each abstract included the name of the meeting, the year of presentation, cancer type, study site, type of economic analysis, description of methods and models used, currency, definition of the budget threshold that was considered cost-effective, cost-effective analysis or cost–utility analysis ratios, use of discounting, base year for cost adjustment, types of costs included, sensitivity analyses, target population, description of intervention and comparator, reasons for selecting the comparator, time horizon, and cost perspective. Based on the findings of our expert survey and abstract review, essential elements for abstract reporting were identified and incorporated into our proposed guidelines.

2.4. Statistical Considerations

Summary statistics are used to describe the characteristics of the health economic experts, their survey responses, and the quality of the conference abstracts reviewed. In multivariate analyses, we used logistic regression to determine independent factors that were associated with the quality of reporting, with adjustments for covariates. The main explanatory variables were

• year of presentation (divided into 4 categories: 1997–2000, 2001–2003, 2004–2006, 2007–2009);

• type of economic analysis (cost-effective analysis, cost–utility analysis, cost-minimization analysis, cost–benefit analysis);

• cancer site (breast, lung, gastrointestinal, genitourinary, gynecologic, hematologic, or other);

• geographic location of the study (Europe, North America, or other); and

• name of the meeting (asco, ash, ispor).

We used the SAS software application (version 9.2: SAS Institute, Cary, NC, U.S.A.) to perform all statistical analyses, in which a p value less than 0.05 was considered statistically significant.

3. RESULTS

3.1. Characteristics of Health Economic Experts

Of 104 health economic experts invited to complete the survey, 5 opted out of participation, leaving 99 potential respondents. Of those 99 experts, 50 replied, yielding an overall response rate of 51%. Table ii describes the demographic characteristics of the survey respondents (briefly, 78% men; median age 53 years; 50% economists). The average experience in the field was 24 years. Approximately 72% of survey respondents worked in an academic institution. About 54% of respondents were from the United States; only 2% were from Asia.

TABLE II.

Characteristics of health economic experts

Variable	Value [n (%)]
Sex
Male	39 (78)
Female	11 (22)
Location
Canada	9 (18)
United States	27 (54)
Europe	11 (22)
Asia	1 (2)
Other	2 (4)
Occupation
Physician	6 (12)
Economist	25 (50)
Policymaker	3 (6)
Other	16 (32)
Work setting
Academic	36 (72)
Community	1 (2)
Other	13 (26)

3.2. Characteristics of Economic Analysis Abstracts

A total of 216 abstracts were included in the review. Table iii presents their characteristics.

TABLE III.

Characteristics of economic abstracts reviewed

Variable	Frequency	Percentage
Meeting type
asco	114	53
ash	31	14
ispor	71	33
Meeting year
1997	8	3.7
1998	3	1.4
1999	6	2.8
2000	11	5.1
2001	11	5.1
2002	7	3.2
2003	7	3.2
2004	15	6.9
2005	17	7.9
2006	26	12.0
2007	38	17.6
2008	30	13.9
2009	37	17.1
Study location
Europe	57	26
North America	136	63
Missing	10	4.6
Other	13	6.0
Cancer site
Breast	66	31
Gastrointestinal	31	14
Genitourinary	30	14
Gynecologic	5	2.3
Hematologic	43	20
Lung	17	7.9
Other	24	11
Type of analysis
cea	81	38
cma	3	1.4
cua	104	48
Missing	28	13

asco = American Society of Clinical Oncology; ash = American Society of Hematology; ispor = International Society for Pharmacoeconomics and Outcomes Research; cea = cost-effectiveness analysis; cma = cost-minimization analysis; cua = cost–utility analysis.

The number of economic analysis abstracts per year increased between 1997 and 2009, with significantly higher representation in more recent years. Specifically, 17% of all abstracts reviewed were presented in 2009; only 4% had been presented in 1997. About 53% of abstracts were presented at asco, and 63% were based in North America. Almost 31% of the abstracts were associated with breast cancer, but only 2% were in the field of gynecologic malignancies. Approximately 50% of the abstracts involved cost–utility analyses.

3.3. Quality of Economic Analysis Abstracts

Table iv describes the quality of the abstract reporting. Median quality score was 75 (range: 48–93), but notable deficiencies were observed. For instance, the cost perspective of the economic analysis was reported in only 61% of abstracts, the time horizon was described in only 47%, and the types of costs included in the analysis were clearly described in fewer than 25% of abstracts. Reporting of parameters that were absolutely essential was better, with more than 90% of abstracts stating the type of cost analysis conducted and providing a clear description of the intervention being examined. However, a clear description of the comparator was present only in 56% of abstracts.

TABLE IV.

Reporting of key abstract parameters among abstracts reviewed

Abstract item	Frequency	Percentage
With scores ≥4.50
Type of cea
No	0	0
Yes	216	100
Description of intervention
No	1	0.5
Partial	1	0.5
Yes	214	99
Description of comparator
No	13	6.0
Partial	83	38
Yes	120	56
With scores 4.00–4.49
Target population
No	18	8.3
Yes	198	92
Time horizon
No	114	53
Yes	102	47
Perspective
No	81	38
Partial	4	1.9
Yes	131	61
How denominator was derived
No	17	7.9
Yes	199	92
cea ratio
No	0	0
Yes	216	100
Conclusion
No	0	0
Yes	216	100
Currency
No	18	8.3
Yes	198	92
With scores 3.50–3.99
Background
No	7	3.2
Partial	100	46
Yes	109	50
Costs that were included
No	63	29
Partial	103	48
Yes	50	23
Sensitivity analysis
No	66	31
Yes	150	69
Study site
No	10	4.6
Yes	206	95
Discounting
No	113	52
Yes	103	48

cea = cost-effectiveness assessment.

3.4. Factors Associated with Abstract Quality

Table v shows the relationship between various abstract characteristics and the abstract quality scores. Independent factors that were associated with higher abstract quality scores included year of presentation (p = 0.0014) and tumour site (p = 0.0053). Specifically, abstracts presented in more recent years demonstrated higher quality than earlier abstracts. Furthermore, we found that abstracts dealing with particular tumour sites (such as hematologic malignancies and genitourinary cancers) had higher quality scores.

TABLE V.

Factors associated with abstract quality scores

Parameter	Estimatea	p Value	Median quality scoreb
Meeting
ispor	Ref	0.3100	75.0
asco	1.61		82.1
ash	−2.27		75.0
Year
1997–2000	Ref	0.0014	67.9
2001–2003	3.32		73.2
2004–2006	6.58		76.8
2007–2009	8.38		78.6
Site
Other	Ref	0.1034	72.3
Europe	−0.15		76.8
North America	0.39		75.0
Cancer
Other	Ref	0.0053	73.2
Breast	2.28		75.9
Gastrointestinal	3.10		75.0
Genitourinary	4.71		78.6
Gynecologic	0.82		64.3
Hematologic	7.45		82.1
Lung	−5.45		69.6
Type
Other	Ref	0.7315	79.5
cea	−2.19		73.2
cma	−3.24		75.0
cua	−0.93		76.8

^aBeta estimates are presented in this linear regression model.

^bAbstract scores have been normalized.

ispor = International Society for Pharmacoeconomics and Outcomes Research; Ref = reference variable; asco = American Society of Clinical Oncology; ash = American Society of Hematology; cea = cost-effectiveness analysis; cma = cost-minimization analysis; cua = cost–utility analysis.

3.5. Proposed Abstract Reporting Guidelines

Based on the findings of the expert survey, we identified essential elements for economic analysis abstracts and proposed guidelines to improved future reporting (Table vi).

TABLE VI.

Guidelines for reporting of economic analysis abstracts in oncology

Items that are absolutely essential

Type of economic analysis (cost–utility, cost-effective, cost benefit, cost-minimization)

Description of intervention

Description of comparator

Items that should be reported

Target population for intervention

Time horizon

Perspective

Method in which denominator (lyg, qaly) was derived

Cost-effectiveness ratio

Conclusion about cost-effectiveness intervention

Currency

Items that may be reported if space permits

Background of cost problem

Costs that were included

Sensitivity analysis

Setting of study

Discounting

lyg = life-years gained; qaly = quality-adjusted life years.

4. DISCUSSION

In an era of increasing restrictions on health care expenditures, legislative bodies more frequently require information about the cost-effectiveness of health care interventions. The purpose of an economic evaluation is to identify the costs and consequences of the alternatives being considered with the aim of informing value-for-money judgments about an intervention or program. Consequently, to promote proper and consistent conduct of evaluations of new health interventions, guidelines pertaining to economic analyses have been proposed in both Canada and the United States^17–19.

In the present study, we surveyed a panel of health economic experts to identify elements considered essential for the abstract reporting of economic analyses at major clinical oncology conferences. We subsequently evaluated the quality of abstracts that had been presented at such meetings since the late 1990s, and we rated each one using a quality score derived from our survey. To improve future reporting of such abstracts, we also developed a set of abstract guidelines. The proposed checklist of essential items is long. Thus, it may be more reasonable expect abstracts to include mainly the “absolutely essential” or “should be reported” items, which would improve concision and optimize the feasibility of our guidelines.

Although the quality scores of abstracts reviewed for the present study were favourable (median score: 75; range: 48–93), we observed deficiencies that could be improved. For instance, the cost perspective and time horizon of the economic analysis were not reported in a substantial number of abstracts. The latter deficiencies are particularly concerning because the cost perspective determines the relevant costs and health outcomes that should be included in the economic analysis. Without that information, the reliability of the findings may be called into question. Likewise, the time horizon reflects the length of time over which costs and outcomes are considered. For the reference case, the timeframe should be long enough to capture all significant costs, benefits, and harms associated with the intervention. Without those parameters being clearly detailed in the abstracts, accurate interpretation of the results can be difficult.

Poor abstract quality has been well documented in other oncology settings. Just as we did, Krzyzanowska et al.¹² and Siu et al.²⁰ noted that asco abstracts involving randomized controlled studies and phase i trials were poor in quality. Multiple mechanisms likely underlie those findings. Strict word limitations placed on conference abstract submissions may obligate authors to omit key information. It is noteworthy, however, that a small proportion of the abstracts reviewed for our study were able to achieve a quality score of 90 or more out of 100, suggesting that it is possible to devise concise abstracts that contain most of the essential elements.

Without standardized abstract guidelines, one problem may be that authors do not have a clear idea of the information that would be considered a high priority to abstract reviewers and readers. Interestingly, earlier research indicates that economic analyses published in peer-reviewed journals were also poor in quality, but improved after endorsement of the publication guidelines in the late 1990s^21,22. That finding suggests that the quality of conference abstract reporting might also be improved through the dissemination of similar guidelines²³.

Although the quality of economic analysis abstracts from major conferences over the study period was suboptimal, our study indicates that the abstracts presented in more recent years were of higher quality than those presented in earlier years. This trend might be related to increasing economic pressures within health care systems that have encouraged the use of economic analyses when introducing new interventions. Improved abstract quality may also be a result of increased competitiveness in abstract acceptance at major conferences in recent years. Improvement in quality over time might also be an indirect result of increasing resource constraints, the exponential rate of growth in the costs of cancer drugs, and the advent of national organizations—for example, the U.K. National Institute for Health and Clinical Excellence, the Canadian Agency for Drugs and Technologies in Health—that promote the development of better methods for economic appraisals. Most likely, it is the result of a combination of those factors.

Notably, we observed disparities in the quality scores between economic analyses for different cancer sites (p = 0.005). Abstracts pertaining to hematologic malignancies, for instance, demonstrated better quality scores than did abstracts involving other tumour sites. That finding may be rooted in the fact that molecularly targeted therapies were introduced significantly earlier for hematologic cancers than for nonhematologic cancers. For example, the biologic agent imatinib was adopted more than a decade ago for the treatment of chronic myelogenous leukemia, but similar agents were implemented for the management of solid malignancies only starting in about 2005^24,25. Because biologic agents are costly, their earlier use in hematology may also have contributed to an earlier adoption of economic analyses in the field and thus resulted in better quality abstracts over time.

Although economic analyses have become more common as new interventions are introduced in oncology, they have not consistently been accepted by the broader oncology community or uniformly adopted by all health care organizations. For instance, most participants in six California-based health management organizations indicated that economic analyses should ideally be used to guide Medicare coverage decisions, but they have not fully incorporated economic analyses into their review processes. Cited concerns about implementing economic analyses include the fear of litigation, the failure of cost analyses to address shorter-horizon cost implications, and the quality and accuracy of the analyses (especially commercially sponsored ones)²⁶. Those data suggest a need to develop guidelines to standardize the way in which abstract reporting of economic analyses are first presented, framed, and discussed at various clinical conferences, because the first presentation can have important implications for the acceptability of the analyses to decision-makers. To that end, we identified abstract elements that were considered important by a worldwide panel of health economic experts, a process that assisted in the development of guidelines that may help to improve future reporting.

Past studies have demonstrated that the establishment of reporting guidelines can result in improvements in the quality of full articles for randomized controlled clinical trials and meta-analyses^8–11,27. Because conference abstracts may have a significant impact on medical decision-making about novel interventions, measures to improve quality—such as our proposed guidelines—are essential to ensure that accurate and pertinent information are conveyed. Better-quality reporting has been found to be associated with a higher likelihood of acceptance for conference presentations and also broader acceptance by the target audience^28–30. Notably, guidelines concerning economic evaluations already exist, and they emphasize comprehensive data collection and complete reporting of all elements for full manuscripts. Given that the main goal of the present study was to propose guidelines for abstract reporting of economic analyses at conferences and meetings, our guidelines for abstracts may not be reliably compared to the guidelines for manuscripts. Details about elements such as “how data were derived” and “what costs are included” are very important always to include in the final manuscript, but the results of our expert survey indicate that those details may not be as pertinent in the initial abstract, in which the word count is consistently more limited.

Our study has several limitations. First, the quality of abstract reporting does not necessarily correlate with the actual quality of the methods used to conduct the economic analysis. Second, the quality of the abstract is not an accurate reflection of the quality of the final manuscript. Previous studies have shown significant discrepancies between data presented in abstract form and data presented in final manuscript form. Third, we reviewed only abstracts related to oncology and limited our selections to abstracts presented at three major conferences. We therefore cannot comment on the quality of abstract reporting in other specialties of medicine or of abstracts discussed at other meetings. Fourth, the guidelines may be subject to author bias because the original list of 24 abstract items that was distributed to health economic experts for their ratings were generated by the authors, even though an exhaustive inventory of items was devised after a thorough review of the cost literature. Lastly, because the proposed abstract guidelines were derived based on the rankings from survey respondents, a main limitation is survey response bias, whereby rankings from survey non-respondents may have differed from those of respondents, potentially limiting generalizability. However, our initial survey was distributed to experts across North America, Europe, and Asia, and so our results may still have utility and applicability in jurisdictions in which the general demographics of the population are largely similar to those of our respondents.

5. CONCLUSIONS

Our study revealed a need to enhance the quality of abstract reporting of economic analyses in oncology, because data presented in conference proceedings represent formal study records that may ultimately affect health care decisions. The guidelines proposed here may help to improve the quality of future economic analysis abstracts by standardizing the format and prioritizing the content to be included.

6. CONFLICT OF INTEREST DISCLOSURES

The authors have no financial conflicts of interest to declare.