6-STEPPPs: A Modular Tool to Facilitate Clinician Participation in Fair Decisions for Funding New Cancer Drugs

Abstract

Purpose:

To design a tool to assist clinician participation with cancer drug funding decisions. Public policy-makers and insurers are struggling with funding decisions regarding increasingly expensive new cancer drugs. Increasingly, oncologists are contributing to the process of review that leads to such decisions. We were asked to design a system for ranking new cancer drugs for priority-based funding decisions.

Methods:

The “Accountability for Reasonableness” framework informed the design of a six-module multistakeholder decision process blending evidence-based traditional technology assessment methods with individual and cultural values elicitation. The tool was piloted in three settings: (1) videotaped simulated multistakeholder deliberation sessions; (2) clinical oncology leaders; and (3) a regional (Canadian provincial) pharmacy and therapeutics committee making formulary decisions. The modules involve: decision clarification, drug eligibility screening (filtering), clinical performance scoring index, cost modeling, data integration and values clarification, and process evaluation.

Results:

The tool was feasible to use, acceptable to participants, and able to rank candidate drugs. The pharmacy and therapeutics committee with whom it was tested used the tool as a part of their deliberations, and the tumor group leaders requested its incorporation into organization-based decision making.

Conclusion:

The decision tool can facilitate priority-based cancer drug funding decisions that meet the conditions of fairness as perceived by participants, including oncologists.

Introduction

Pricing trends for cancer drugs are challenging decision makers and insurers to examine the health gains to justify rising costs.^1,2 Oncologists are becoming more involved in these issues as health services researchers and as developers of guidelines used in priority-based rationing decisions.^1–3 In Canada and Europe, variations in public funding are raising concerns about equity of access.^4–6 Recent studies have examined physicians' attitudes toward rationing as a principle of distributive justice.^7–10

Although public debate about cancer drug costs has intensified recently, proposed fair-decision frameworks^11–14 and approaches to ethical priority setting are not new.^15–19 The “accountability for reasonableness” (A4R) framework has garnered interest as a basis for fair and acceptable decisions about cancer drug funding.^17,20,21

In response to drug budget pressures, we were commissioned by a Canadian provincial cancer agency to help prioritize new drugs for funding using a ranking system. In the absence of an established tool that met our needs, we initiated a project to develop a decision tool within the context of Canada's publicly financed health system. The tool, 6-STEPPPs (Systematic Tool for Evaluating Pharmaceutical Products for Public Funding Decisions), is arranged in a modular format; the modules are included in the Appendix (Tables A1 and A2, online only). While the tool has and continues to be piloted among key decision makers and continues to evolve, its current form has been sufficiently developed and refined to warrant broader exposure and application to further its improvement.

Table A1.

Decision Clarification Module

Decision Model	Decision Scenario	Decision Type	Comparator	Costing	Intended Use Stratification*
					Tier 1	Tier 2
A	Single technology/drug under review, de-listing of previously approved candidate drugs not an option	Appropriateness only (ie, does not involve a funding decision or recommendation)	Benchmark against credible alternative(s)	Cost models not required
B	Single technology/drug under review, de-listing of previously approved candidate drugs not an option	Appropriateness and funding	Benchmark against credible alternative(s)	At least one cost model required	Curalib (A)
C	Several competing new drugs under review, de-listing of previously approved candidate drugs not an option	Appropriateness only	Rank against one another	Cost model not required
D	Several competing new drugs under review, de-listing of previously approved drugs not an option	Appropriateness and funding	Rank against one another	At least one cost model required		Pallialib (B)
E	Single drug only, de-listing of prior approved drugs is an option	Appropriateness only	Benchmark against a prior approved candidate drug or rank against several prior drugs	Cost model not required
F	Single drug only, de-listing of prior approved drugs is an option	Appropriateness and funding	Benchmark against prior approved drug or rank against several prior drugs	At least one cost model required
G	Several competing drugs, de-listing of prior approved drugs is an option	Appropriateness only	Rank against currently competing drugs and against prior approvals	Cost model not required
H	Several competing drugs, de-listing of prior approved drugs is an option	Appropriateness and funding	Rank against currently competing drugs and against prior approvals	At least one cost model required

NOTE. Tier 1 indicates adjuvant or curative treatment; tier 2, palliative treatment for advanced metastatic disease.

Table A2.

Criteria Filtering (screening) Module

Drug name	Pass/Fail/Defer	Fail (indicating reason using code* below)	Defer (indicate reason using code* below)
Drug A (curalib)	D		NOC
Drug B (pallalib)	P
Drug C (oxymoralib)	P

* Codes: NOC (Notice of Compliance); E (Minimal evidence standard not met); O (inadequate outcomes reported); UR (under review elsewhere); NEG (sensitive negotiations ongoing); OTH (other).

Methods

Assumptions

This tool is designed to be used under the constraints of a budget envelope, which is common within health care organizations. Hence, it deals mainly with the purchase prices of new technologies. A more comprehensive approach would include a formal economic evaluation considering all costs and consequences of their adoption, including indirect costs and benefits. Such approaches are limited by lack of data and budgeting contingencies. Our tool is intended to assist those making day-to-day decisions under existing real-world constraints. The use of the tool could be complemented by formal economic evaluation methods that capture broader costs and consequences.

Tool Design

The 6-STEPPPs tool design was informed by the principles of A4R,²¹ which specifies the following conditions: “publicity” (ie, transparency of process and decisions); “reasons” (ie, the decision logic, what might be called content validity); “appeals” (ie, opportunity for decisions to be challenged); and “enforcement” (ie, a mechanism to ensure the other conditions are met). To this we added “consistency,” or ensuring that decisions made at different times used similar processes, or that decisions that were apparently inconsistent could be defended based on new circumstances²²; and “efficiency” (timely decisions). The A4R framework is intended to lend legitimacy to difficult funding decisions where priority setting will naturally fail to meet the wishes of all competing interests.¹⁷

The following section describes the 6-STEPPPs tool. An example is introduced in Table 1 and carried through in the Appendix. A similar approach has recently been published for priority setting in primary care.¹⁹

Table 1.

Baseline Product Profile Template

Generic Product Name(brand name)	Response Options	Product A(curalib)	Product B(pallalib)	Product C(oxymoralib)
Manufacturer	Name of sponsor	Company X	Company Y	Company Z
Product purpose	Tier 1(adjuvant/curative v tier 2 (metastatic/ palliative)	Tier 1	Tier 2	Tier 2
Notice of compliance status (see text)	Pending, approved, not submitted	Not submitted	Pending	Approved
Clinical indication requested	List indication	Adjuvant colorectal cancer	Advanced plasma cell myeloma	Advanced breast cancer
Approved for marketing for other indication(s) in this jurisdiction?	Yes or No	No	No	No
Approved for this indication in another Canadian jurisdiction? (specify)	Yes or no; if yes, list jurisdictions	Yes (provinces of … )	No	No
a) Recommended dose and duration of treatment	Dose per exposure/No. of exposures	130 mg per exposure/6 treatments	12 mg per exposure/4 treatments	80 mg per exposure/12 treatments
b) No. of eligible patients expected/year*	No. patients with condition in population	70	30	115
c) Cost per unit of drug exposure ($)	Cost per single treatment	4,500	8,000	11,000
d) No. of treatments per year (year 1)	Average No. of exposures/patient	5	3	9
e) Total annual acquisition cost ($ in millions)	b × c × d	1575	0.720	11.385
f) Total acquisition cost of treatment being replaced (annual; $ in millions)	Price per unit × No. of units per patient × No. of patients	0.54	0.215	4.50
g) Incremental acquisition cost (annual; $ in millions)	e-f	1035	0.505	6.885
Other resource utilization impacts (favorable/unfavorable; staffing, drug delivery, equipment, testing)	List resources affected	Oral replaces intravenous	More prolonged intravenous infusion	Weekly intravenous replaces every-3-weeks intravenous; additional laboratory test for drug eligibility
Status of other decision processes (eg, NICE, common drug review)	Under review/not under review/don't know	None	Common drug review	NICE

NOTE. Please note that Products A, B, and C are hypothetical, and are used here for purposes of illustration only.

Abbreviation: NICE, National Institute for Health and Clinical Excellence.

*For the advanced metastatic setting annual costs for drugs are expected to increase over time as patients surviving may require ongoing treatment (assume ≤ 50% patients surviving to second year).

Overview of Decision Process for Tool Application

The 6-STEPPPs tool is composed of a series of modules embedded within a decision process involving three sequential phases with distinct attributes (Table 2): clinical evaluation phase, clinical/administrative evaluation phase, and the policy decision phase. These occur sequentially, with each phase informing the next one.

Table 2.

Outline of Decision Process Phases and Their Attributes

Attributes	Clinical Evaluation Phase	Clinical/Administrative Evaluation Phase	Policy Decision Phase
Objective	To evaluate the clinical appropriateness, and relative clinical value of a new drug technology for its introduction into the formulary for cancer treatment	To evaluate financial and systems-related factors in relation to the relative clinical value of new drug technologies for introduction into the formulary for cancer treatment; to compare the relative merits of competing drugs for funding.	To ensure fair process in evaluation of new drug technologies; to evaluate policy-related factors; and to integrate clinical, administrative, and policy determinants in making decisions about public financing of new drug technologies
Expertise required	Clinical knowledge Methodological skills Workplace familiarity, practical	Pharmacy policy Clinical knowledge Financial Systems flow	Governance Population perspective Relative value and affordability Health systems factors Political factors
Participants	Clinical care teams (tumor groups)Pharmacist Methodologists (may be clinicians)Patient representative (preferred)	Pharmacy and therapeutics committee (clinicians, clinical administrators, pharmacist)Patient representative (preferred)	Organizational Executives Trustees as appropriate, clinical, and methodological resource person Patient representative (optional)May include physicians
Responsibilities	Provide critical evaluation of strength of evidence, relative clinical benefit in relation to current standards, likely impacts on patients, availability of reasonable alternatives to patients, and feasibility of delivery of a new drug technology	Provide comparative estimates of projected usage and cost implications of competing new drugs if approved, benchmark comparators, determine relative value for degree of benefit against benchmarks	Ensure appropriate decision process. Weigh relative value to patients, opportunity costs to the population/ society, and mechanisms for funding
Outcome	A critical evaluation of the drug technology's clinical performance and relative clinical value, prioritize indications for use	A priority-based recommendation regarding inclusion of drugs in the formulary, and appropriate conditions for funding	A decision regarding funding conditions and mechanisms of funding, or recommendations to government

The clinical evaluation phase involves mainly clinicians assisted by methodologists. The clinical/administrative phase involves both clinicians and administrators; for example, a pharmacy and therapeutics (P&T) committee. Patient representation is encouraged as an important attribute of legitimacy. Within the context of this work, the policy decision phase is undertaken by organizational executives and a board of directors, which could include physician representation.

To facilitate the decision-making process, each phase is divided into a preparation stage, where data are synthesized and organized by trained staff, and a deliberation stage, where the panel examines and discusses the prepared data. Table 1 presents the type of data that can be prepared in advance using a baseline template (Table 1).

Following is a description of how to apply the 6-STEPPPs tool, with examples of theoretical products (drugs A, B, and C), information for which is included in Table 1, and carried forward in the Appendix. The drug names used as well as the data in Table 1 are fictitious, although the data closely resemble real situations based on our experience.

The tool is composed of a user guide and a companion workbook (not shown), through which deliberations are recorded and carried forward to create a record of decisions at each phase. The entire tool, including the user guide and the workbook can be made available through correspondence with the authors.

Modules

The six modules of the 6-STEPPPs process are: (1) decision clarification; (2) criteria filtering (screening); (3) clinical performance evaluation; (4) cost modeling; (5) data integration and values clarification; and (6) process evaluation.

1. Decision clarification module.

The decision clarification module (Appendix, Table A1) invites decision panels to explicitly identify the type of decision to be made. There are nine possible decision types based on a combination of three variables: (1) Appropriateness only versus appropriateness and funding: all decisions about whether or not to approve a new medicine address the clinical appropriateness of the product's claims, and its indications for use. In some circumstances, this may be the only factor considered; in other circumstances, decision makers are asked to determine whether the product should be publicly funded and under what conditions. (2) Ranking versus benchmarking: in some circumstances, several new products may be competing for the same budget envelope, and the decision may require a relative ranking among the competing products. This is more likely with tier 1 as opposed to tier 2 (palliative) drugs. In other circumstances, only one product is being considered, with its attributes benchmarked against other previously evaluated products. (3) Delisting option: the approach taken may be influenced by whether previously approved products are allowed to be removed from the approved list in order to make room for a new product. For tier 2 (palliative) products, other than chemotherapy drugs, it is more likely that new products would be added to the list of available ones, rather than replacing existing drugs.

The decision clarification module is to be completed at the start of the deliberation process, and provides the initiation of the record of decisions that contributes to transparency.

Table A1 also shows stratification of candidate products according to their intended purpose, with tier 1 products intended to provide opportunities for cure or longer term survival, and tier 2 products intended for advanced or metastatic disease where palliative benefit or a shorter term survival gain is the clinical goal. In the model, products within each tier are compared and ranked, but products are not directly compared across tiers. This stratification was introduced after initial pilot testing.

In the example of the three candidate drugs included in Table 1, all are being evaluated for a funding decision without the option for delisting. A is the only product in the adjuvant or curative treatment tier; therefore, it is to be benchmarked against previously evaluated products (decision model B, Table A1). Products B and C are intended for palliative treatment (tier 2), and are to be compared with one another (decision model D).

2. The criteria filtering module.

The criteria filtering module (Appendix Table A2, online only) is intended to lend efficiency to the process by allowing panels to either defer, or drop consideration of a product that fails to meet obvious criteria (ie, filters).

Reasons for filtering, or screening products early can be coded (panels can devise their own reasons or codes) with an example shown in the Appendix (Table A2). Coding is done explicitly and contributes to creating the permanent “record of decision.”

In the example, consideration of product A is deferred in the Canadian context because it has not been submitted to government for approval by the marketing sponsor (known in Canada as Notice of Compliance, which was not filed as per the baseline information in Table 1), while drugs B and C pass the filtering process for further consideration. However, in the modules that follow, product A is carried forward for illustrative purposes.

3. The clinical performance evaluation module.

The clinical performance evaluation module (Appendix Tables A3 and A4, online only) is designed to meet the “reasons” condition of A4R and is where clinicians are most influential. The content is framed purposefully to focus panel members on the clinical attributes of the candidate products, with room left for judgment and compromise in order to elicit individual perspectives and values. The seven clinical performance criteria and their definitions (Table A3) are drawn from published studies of priority decision processes for funding new cancer drugs.^11,13,14 The GRADE framework was used to inform criteria scoring related to strength of evidence.²⁵

Table A3.

Clinical Performance Index Scoring Guide

Scoring Index Criterion/Score	Scoring Guide
Intended purpose of intervention (most relevant clinical benefit[s] to be experienced by patient)
5	To provide a realistic opportunity for an important improvement in the cure rate compared with current standard
4	To provide a realistic opportunity for longer term survival free of disease compared with current standard, with the possibility for improved cure rate
3	To provide more effective palliation or symptom control compared with the current standard, with a realistic opportunity for improved survival
2	To provide more effective symptom control or palliation, without a realistic opportunity for clinically meaningful survival improvement
1	To produce a tumor response or improved tumor marker status only
Strength of evidence (in relation to most appropriate and feasible study design)
5	It is unlikely for legitimate clinical and/or scientific reasons that stronger or more persuasive evidence of higher quality will emerge from future studies
4	Future studies of similar quality are likely to be possible to confirm current evidence, but with significant delay in availability of results
3	Future studies of higher quality are feasible and likely to be done, but with significant delays in availability of results
2	It would have been feasible to conduct higher quality studies than what has been reported to date; but, there are no such active trials and higher quality evidence is not anticipated to be available in the near future (ie, within 3 years)
1	Evidence is currently being collected through ongoing higher quality studies, with results likely to be available within a reasonable time period
Consistency of results (in terms of magnitude and direction of effect)
5	There are several different RCTs* across which direction and magnitude of benefit are similar
4	There is only one well-powered, high quality RCT but with very good results making further trials unnecessary or unlikely
3	There are several different trials across which direction but not magnitude of benefit are similar
2	There is inconsistency in the direction and/or magnitude of benefit across trials
1	There is insufficient data to assess the magnitude of benefit (eg, small sample sizes)
Clinical impact (in terms of size and direction of effect across studies)
5	The magnitude and duration of benefit in terms of tumor control for clinically important outcomes compared with current standard of care is large and important from the individual and societal perspectives
4	The magnitude or duration of benefit in terms of tumor control for clinically important outcomes compared with current standard are modest, but most would agree important
3	The magnitude or duration of benefit in terms of tumor control is modest and of debatable importance to either the individual or to society
2	The degree of benefit in terms of tumor control is real but small
1	The degree of benefit in terms of tumor control is of debatable clinical importance
Appropriateness of measures used to assess outcomes (validity, credibility of measures of effect used in trials)
5	Valid and reliable outcome measures were used that are appropriate to the intended purpose of the treatment
4	Outcome measure(s) used did not directly measure the clinical endpoint of interest, but has been shown to be a valid surrogate of this endpoint
3	Outcome measures used may translate into improvements in the main clinical endpoints of interest, but this has not been directly validated in rigorous studies
2	Outcome measures used are secondary to the main clinical endpoints of interest as defined by the purpose of the treatment
1	Outcome measures have not been formally or sufficiently validated in the context of the studies reported
Toxicity/convenience (importance of adverse effects including toxicity, safety and inconvenience)
5	Toxicity/inconvenience is much less than current standard or best alternative
4	Toxicity/inconvenience is equivalent to the current standard or best alternative
3	Toxicity of grade 1–3 is somewhat greater than current standard or best alternative, without increased life-threatening toxicity
2	Toxicity grade 1–3 is significantly greater than current standard or best alternative
1	Increased chance of life-threatening toxicity for no, small or modest anti-tumor benefits compared with current standard or best alternative
Availability of alternatives (would a patient have access to reasonably effective alternative interventions?)
5	The patient would be otherwise ineligible to receive a treatment that has shown any benefit
4	Alternatives are significantly less effective; or with significantly more toxicity or inconvenience
3	Alternatives are somewhat less effective; or equally effective but with slightly more toxicity or inconvenience
2	Alternatives are equivalent in anti-tumor effectiveness, but with slightly more toxicity or inconvenience
1	Alternative is equivalent or superior in anti-tumor effect, with no or slight increase in toxicity

Table A4.

Pattern and Clinical Performance Scoring Index Results

Evaluation Criteria (score range 1–5)
Product Name (generic)	Intended Purpose of Treatment	Strength of Evidence	Consistency of Results	Clinical Impact	Appropriate Outcome Measured	Toxicity Level/Risks/Convenience	Alternatives Available	Total Score
Tier 1
Curalib (A)	5	4	4	4	4	4	2	27
Tier 2
Pallalib (B)	3	4	4	3	4	4	3	25
Oxymoralib (C)	3	3	3	3	5	3	4	24

NOTE. Tier 1 indicates drug technologies used in the adjuvant or curable setting; tier 2, drug technologies intended for the palliative or metastatic setting. Product A (tier 1) is not compared with B and C (tier 2). B scores better than C overall, but scoring patterns differ. B studies were more rigorous, with more consistent results and a better toxicity profile versus comparators. C studies used more appropriate outcomes, and acceptable alternatives exist. This fosters debate that elicits panel members' implicit values.

Table A3 presents a guide for assigning scores to each criterion, where 5 represents most favorable and 1, least favorable. For priority setting, we suggest that the criterion scores be aggregated to yield a scoring index. The purpose of the scores is to focus panel members on common assumptions about the meaning of the attributes. The scores have not been validated in other contexts, but have been found to be a useful vehicle in our pilot studies. Similarly, the process allows for judgments by decision makers about the relative importance of products, independent of the index scores (see footnote to Table A4). The scores force panel members to explicitly justify rankings that are adjusted despite the scores, to surface implicit values. By encouraging panel members to assign clinical performance scores, the tool provides an opportunity for comparative analyses of the different attributes of different products.

Table A4 presents a sample scoring pattern and derived index for products A, B, and C with a brief notation explaining how such scores can be assigned and used in deliberations. We stress the importance of the scoring pattern, since products may have close or tied scores, and how these are derived would need to be discussed among panel members. The scoring index and the relative weights that panels assign to different criteria can be used to rank these products.

The scoring criteria are purposely not weighted a priori in order to elicit the necessary discussion among panelists that reveals individual and collective values. Even the scoring criteria themselves are worded purposefully to encourage rather than stifle discussion in order to balance the objective data with opportunities for more subjective inputs that give panelists scope for argument and lend legitimacy to the decision.

The clinical performance evaluation module worksheet also contributes to a permanent record of decisions that can be scrutinized by a third party.

4. Cost-modeling module.

As presented in the Appendix (Tables A5 and A6), while classic economic evaluation techniques²⁴ within technology assessment are extensively used for informing resource allocation decisions, there are well-known limitations associated with their use, including the fact that decision makers may not comprehend the results, and their use for drugs for which clinical information is available only for the drug's impact on surrogate end points (often the case for new cancer therapies) is problematic.²⁶ One of the main issues is that high-quality economic evaluations are often not available when policy assessments regarding a new drug are performed.

Given these issues, and in the attempt to make the cost models more comprehensible to members of multistakeholder panels, two cost models were designed to help express the worthwhileness of investments in cancer drugs. In these models, only drug costs are included; the impact on other health care resource use is not estimated. While the use of these cost models requires certain assumptions, their use is meant to inform panelists and to support transparent decision making.

Examples of the two models are shown in the Appendix. They involve: (1) an expression of the cost ($) per average unit of time gained (eg, months) in delaying an adverse event (eg, mortality − $/months of survival gained; recurrence − $/months of disease-free survival gained; or progression − $/months of progression-free survival gained; Appendix, Table A5); and (2) an expression of the cost per single adverse event avoided (eg, death or recurrence), or per favorable event achieved (eg, tumor response) using the number needed to treat (NNT) methodology^27,28 (Appendix, Table A6).

Table A5.

Cost Model: Cost per Average Month Gained Before Experiencing an Adverse Event (event may be mortality, recurrence, or progression)

	A	B	C	D	E	F	G	H
Product Name (generic)	Annual Cost/Patient ($ in thousands)	No. of Patients Eligible/Year	Total Annual Cost ($ in millions; A × B)	Intended Benefit (event avoided)	Outcome Measured	Avg. Benefit (months gained)	Person- Months of Benefit (B × F)	Cost per Month of Benefit ($ in thousands; A/F)
Tier 1
Curalib (A)	22.5	70	1.575	Death	Recurrence	5	350	4.5
Tier 2
Pallalib (B)	24.0	30	0.72	Progression	Progression	4	120	6.0
Oxymoralib (C)	99.0	115	11.385	Progression	Progression	4	460	24.75

NOTE. Cost estimates are taken from Table 1, and clinical outcomes from clinical trials. Product A (tier 1) intends to improve long-term survival, but disease recurrence is used as a surrogate. The estimated cost is $4,500/month of benefit. For B and C (tier 2), progression-free survival was measured-estimated costs are $6000 and $24,750/month gained, respectively. Approving C costs more, but produces more person-years of benefit. Columns D, G, and E focus discussion on the relative importance of outcomes used.

Table A6.

Cost Model: Cost per Single Event Avoided at a Designated Period of Follow-Up

	A	B	C	D	E	F	G	H
Product Name (generic)	Annual Cost/Patient ($ in thousands)	No. of Patients Eligible/Year	Total Annual Cost ($ in millions; A × B)	Event Avoided at Follow-Up Time	Absolute Event Rate Difference (%)	NNT 1/(E/100)	No. of Patients Benefiting per Year (B/F)	Cost per Event Avoided or per Single Patient Benefiting ($ in thousands; A × F)
Tier 1
Curalib (A)	22.5	70	1.575	Recurrence at 1 year	7	14.3	4.9	321.75
Tier 2
Pallalib (B)	24.0	30	0.72	Progression at 1 year	8	12.5	2.4	400.80
Oxymoralib (C)	99.0	115	11.385	Progression at 1 year	12	8.3	13.8	821.70

NOTE. Cost estimates are taken from Table 1. Estimates of clinical benefit are determined from clinical trials. A (tier 1) costs $321,000 to avoid a single recurrence at 1 year, with 4.9 patients in the population benefiting. For B and C, costs are $400,000 and $821,000 respectively to avoid one progression at 1 year, with 2.4 and 13.8 patients benefiting, respectively. With wider application of the technique, panel members' ability to judge the “worthwhileness” of such investments will improve.

Abbreviation: NNT, number needed to treat.

Incremental rather than total drug acquisition costs per unit benefit can be calculated by subtracting the costs of treatments to be replaced. The calculations can be derived from pricing estimates and outcomes from published clinical trials. The drug products could then be ranked by both costing models taking into account the differences in the outcome measures reported. Panels are still free to commission more formal economic evaluations.

Our experience suggests that the ranking results of new products using modules 3 (clinical performance index) and 4 (two cost models) may differ. While those desperate for an unequivocal “answer” may be frustrated by such discordance, this has emerged as an important part of the values clarification process as panels struggle to reconcile conflicting data.

The footnotes for Tables A5 and A6 highlight important considerations for making decisions, especially where available data for different drugs involves outcomes of varying clinical importance.

5. The data integration and values clarification module.

The data integration and values clarification module (Appendix Table A6) provides a matrix display of key data, scoring patterns and index, and rankings across modules 3 and 4. The display assists deliberations by contrasting the ranking patterns. Panel members are encouraged to examine and weigh different criteria and relative costing data. These deliberations are recorded in order to reveal how differences of opinion and interpretations were handled.

6. Process evaluation module (Appendix Figure A1, online only).

Figure A1.

Process and Tool Evaluation Module

By definition, A4R is achieved when those participating in, or are affected by, decisions are satisfied that the processes leading to them were fair and reasonable. This module is designed to survey participants about their satisfaction that A4R conditions were met.

Pilot Testing

The 6-STEPPPs tool has undergone several modifications through pilot testing. Pilot tests were conducted with three groups: clinical leaders of tumor groups, usually oncologists in their professional roles; the multidisciplinary P&T committee responsible for priority-related formulary decisions; and a multistakeholder group who participated in a role-playing exercise to determine how they approached the decision process from an unfamiliar assigned role. The latter experience was video-taped with the participants' permission and the results will be published separately by Sinclair S et al.²⁹

The tool was received favorably in all three venues. The P&T committee used it several times to deal with previously outstanding submissions outside of the formal testing exercise. It is too early to determine whether the goals of improving acceptance of priority setting decisions by all stakeholders and improving the efficiency of priority setting will be achieved.

Discussion

Clinical oncologists are becoming more involved, either directly or indirectly, in resource allocation decision making because of their ability to evaluate the clinical performance of alternative interventions or because of their roles as guideline developers or health services researchers. It is in the interests of participating clinicians on behalf of their patients to promote decision processes that are transparent to the public and to providers, and that meet the conditions of fairness.

The main concern of those affected by rationing decisions has been the apparent lack of transparency of decision processes, debate about the utility of traditional technology assessment methods for budgeting purposes, and lack of understanding by the public of the methods used.

Our project was a direct response to concerns expressed by our regional policy makers about the tools available to them for setting priorities for cancer drug funding within a restricted budget envelope.

The 6-STEPPPs tool was designed to meet the conditions of A4R, and to incorporate more intuitive processes within traditional technology assessments that elicit the values at play in such deliberations. The tool was purposefully designed to create an ongoing ‘record of decisions’ to meet the condition of transparency.

The 6-STEPPPs tool is currently designed as a user guide and companion workbook. The workbook is composed of a series of worksheets, completion of which forms the ultimate “record of decisions,” which will contribute to transparency. Examples are shown in the Appendix.

The 6-STEPPPs tool has undergone pilot testing and seems sufficiently developed in the context of other published frameworks to be of value to decision makers currently struggling with priority setting focused on cancer drug funding.

Table A7.

Data Integration Matrix

Product Name (generic)	Index Score	Index Rank by Score	Adjusted Index Rank	NNT	Cost per Event Avoided or Patient Benefiting ($ in thousands)	NNT Cost Model Rank	Cost per Average Month Gained ($ in thousands)	Outcome of Record	Cost Model Rank	Final Rank
Tier 1
Curalib (A)	27	NA	NA	14.3	321.75	NA	4.5	R	NA
Tier 2
Pallalib (B)	25	1	2	8.5	400.80	1	6.0	P	1
Oxymoralib (C)	24	2	1	12.3	821.70	2	24.75	P	2

NOTE. The matrix displays relevant data in a single field. In tier 2, B ranks better than C by scoring index alone, but receives a lower adjusted rank after considering other factors (see Table A3). Both costing models favor B over C. Depending on the social, economic, and political context in which they operate, decision makers could still favor C. The module illustrates the tool's function in focusing on data without restricting flexibility in judgments. The main gain is in transparency.

Abbreviations: NNT, number needed to treat; NA, not available; R, recurrence-free survival; P, progression-free survival.