Consensus in controversy: The modified Delphi method applied to Gynecologic Oncology practice

Abstract

Objectives

To determine the degree of consensus regarding the probabilities of outcomes associated with IP/IV and IV chemotherapy.

Methods

A survey was administered to an expert panel using the Delphi method. Ten ovarian cancer experts were asked to estimate outcomes for patients receiving IP/IV or IV chemotherapy. The clinical estimates were: 1) probability of completing six cycles of chemotherapy, 2) probability of surviving five years, 3) median survival, and 4) probability of ER/hospital visits during treatment. Estimates for two patients, one with a low comorbidity index (patient 1) and the other with a moderate index (patient 2), were included. The survey was administered in three rounds, and panelists could revise their subsequent responses based on review of the anonymous opinions of their peers.

Results

The ranges were smaller for IV compared with IP/IV therapy. Ranges decreased with each round. Consensus converged around outcomes related to IP/IV chemotherapy for: 1) completion of 6 cycles of therapy (type 1 patient, 62%, type 2 patient, 43%); 2) percentage of patients surviving 5 years (type 1 patient, 66%, type 2 patient, 47%); and 3) median survival (type 1 patient, 83 months, type 2 patient, 58 months). The group required three rounds to achieve consensus on the probabilities of ER/hospital visits (type 1 patient, 24%, type 2 patient, 35%).

Conclusions

Initial estimates of survival and adverse events associated with IP/IV chemotherapy differ among experts. The Delphi process works to build consensus and may be a pragmatic tool to inform patients of their expected outcomes.

1. Introduction

The standard treatment for advanced ovarian cancer is surgical cytoreduction followed by cytotoxic chemotherapy. While the mainstay for adjuvant chemotherapy has been treatment with intravenous (IV) platinum and taxane agents, several phase III clinical trials have demonstrated improved survival with the use of these drugs both IV and intraperitoneally (IP) [1]. Consistent positive results from these studies led to a National Cancer Institute clinical announcement [2] in 2006 supporting the use of IP chemotherapy in selected patients with advanced ovarian cancer following cytoreduction with limited residual disease. The NCI announcement stated that patients “…should be counseled about the clinical benefit associated with combined IV and IP administration of chemotherapy”. Despite this statement from the NCI, the use of IP chemotherapy has not become widespread, either by gynecologic oncologists or by medical oncologists [3,4], mainly due to the perception of increased toxicity and complexity of administration compared with IV chemotherapy. An additional reason for the lack of acceptance of IP chemotherapy may be clinician bias against this treatment strategy. This bias may have far-reaching implications, since recent literature indicates that the survival advantage of IP/IV over IV chemotherapy extends beyond 10 years [5]. Therefore, although patients and clinicians may differ in the perceived benefits of treatments, as well as the most important treatment-related side effects [6,7], it is reasonable to open the IP versus IV discussion through a shared decision making model.

The Affordable Care Act Section 3506 is a “program to facilitate shared decision making” whose purpose is to “facilitate collaborative processes between patients, caregivers or authorized representatives, and clinicians that engages the patient, caregiver or authorized representative in decision making, provides patients, caregivers or authorized representatives with information about trade-offs among treatment options, and facilitates the incorporation of patient preferences and values into the medical plan.” In an effort to enhance the process of shared decision making regarding the route of administration (IV versus IP/IV) of chemotherapy in women with advanced ovarian cancer, we developed a decision aid using estimates of clinical parameters from the literature and from completed and ongoing phase III randomized clinical trials of IV versus IP/IV chemotherapy. Given the differences between populations enrolled in clinical trials and those seen in general practice, we sought to determine the degree of professional consensus regarding the probabilities of specific patient outcomes associated with IV and IP/IV chemotherapy. We administered a web-based survey to an expert panel of clinicians, using the modified Delphi method, described below, to obtain consensus in ovarian cancer patient outcomes that would be expected in usual practice. This technique allows clinicians to base their decisions and responses on more then just their own experience in their own practice, but rather to benefit from the additional experience of a larger community of clinicians.

2. Methods

As part of a PCORI-funded project, 10 ovarian cancer experts (9 gynecologic oncologists and 1 medical oncologist) who administer both IV and IP/IV chemotherapy were asked to provide estimates pertaining to four process or outcome events for two hypothetical patients. They were asked to provide these probabilities for each of the patients and each of the four processes or outcomes, one assuming the patient received IV chemotherapy, and one assuming the patient received IP/IV chemotherapy. In this setting, an expert is defined as a board-certified clinician who has been involved in the chemotherapy management of ovarian cancer for over five years, and who has a willingness to prescribe both IP/IV and IV therapy for advanced ovarian cancer for their patients. They were further selected based on their strong records of clinical trial participation, and representation across the country from both rural and urban settings (which was intended to allow for diversity of responses based on geographic and clinical variation. Experts all practiced at academic medical centers. The purpose of this definition of expertise was to maximize the likelihood that experts were highly knowledgeable regarding the outcomes and toxicities associated with both modes of chemotherapy administration. The regimens and schedules used for IP/IV and IV chemotherapy were not explicitly defined in an effort to allow for responses patterning what is seen in usual practice (for example, with some patients being given bevacizumab or weekly paclitaxel). Experts were queried on the outcomes of two “types” of patients: “type 1” was a patient with limited comorbidities, with a performance status of 1 (prior to initiation of chemotherapy but after cytoreduction) and who underwent complete cytoreduction without bowel resection; “type 2” was a patient with moderate comorbidities, with a performance status of 2 (prior to initiation of chemotherapy but after cytoreduction) and who underwent an optimal cytoreduction with small volume residual disease requiring rectal resection with anastomosis. The panel was asked to provide estimates for the following four events: 1) probability of ER/hospital visits during chemotherapy treatment, 2) probability of completing 6 cycles of the prescribed chemotherapy, 3) probability of surviving 5 years, and 4) median overall survival time (Table 1). The survey was administered in three rounds. Subsequent to round 1 and round 2, a summary of responses and individuals' anonymous explanatory comments were circulated back to the panel, and the experts were asked to resubmit their probability estimates in light of the (anonymous) replies of other members of the panel.

Table 1.

Questions posed to expert panel regarding IV versus IP/IV chemotherapy for advanced ovarian cancer.

Probability of ER/hospital visits during treatment Probability of completing six prescribed cycles of chemotherapy Probability of surviving 5 years Median survival time

This methodology is known as the Delphi survey technique [Fig. 1], which was developed in the 1950s by research scientists working at the RAND Corporation [8]. The original Delphi technique provided open-ended questions; modifications of this technique as used in this study (known as the modified Delphi technique) allow for the process to begin with a set of carefully selected items drawn from various sources (including synthesized reviews of the literature, and interviews with selected content experts). The modified Delphi strategy provides a highly structured, transparent process to obtain anonymous feedback. The approach allows participants to reassess their own judgments as recommendations which are revised according to feedback received through the process. In addition, quantitative data can be collected, allowing for the application and reporting of statistical analyses [9]. Through a series of rounds (typically three), the process is designed to yield consensus. The anonymity of the expert panel is maintained throughout this process to prevent the authority, personality, or reputation of some participants from dominating others in the process. Anonymous participation also allows free expression of opinions, encourages open critique, and facilitates admission of errors when revising earlier judgments [10,11].

Fig. 1.

Flow diagram of modified Delphi technique for establishing consensus in the absence of definitive data. Two (or three if needed to achieve consensus) rounds of surveying are generally performed.

Responses from the experts were summarized using descriptive statistics (mean, median, standard deviation, range) for each patient type, treatment, and outcome after each of the three Delphi surveys. Power analysis and statistical significance are appropriate only for studies that test a hypothesis; these do not apply to descriptive studies such as the Delphi technique. Responses were also examined graphically to identify outliers. Optional comments provided by the experts following each round enabled revision of questions for improved clarity and greater consensus at subsequent rounds.

3. Results

Administration of the survey to 10 ovarian cancer experts who utilize both IV and IP/IV chemotherapy in their practices was performed in three rounds. In the first round, experts estimated that patients undergoing IV chemotherapy had a lower probability of requiring a hospitalization or emergency department visit compared to patients receiving IP/IV chemotherapy, regardless of whether the patient had a worse performance status and residual disease (type 2 patient) or had a better performance status and no residual disease (type 1 patient), as shown in Table 2 and depicted in Fig. 2. When comparing the type 1 to the type 2 patient, the more complex (type 2) patient was estimated to have a higher probability of requiring a hospitalization or an emergency department visit. Similar trends were seen for the probability of completing all 6 prescribed cycles of chemotherapy, though there was less difference between type 1 and type 2 patients receiving either IV or IP/IV chemotherapy compared with differences when stratified by route of administration of chemotherapy. Estimates of the percentage of patients surviving at least 5 years ranged from 67% (type 1 patient undergoing IP/IV chemotherapy) to 41% (type 2 patient receiving IV chemotherapy) in round 1, and median survival from 81 months (type 1 patient undergoing IP/IV chemotherapy) to 47 months (type 2 patient receiving IV chemotherapy) in round 1. Table 2 describes the responses by the expert panel, stratified by patient type, route of administration of chemotherapy, and the round of administration of the survey. Fig. 2 illustrates the responses of the expert panel at round 2 for median survival in a type 2 patient. Comments received from panel members included references to clinical trials used as a source of information and explanations and exceptions for the response given. For example, comments on median survival time included: “Obesity may decrease survival” and “Macroscopic residual has much worse survival no matter how the patient is treated”.

Table 2.

Results (mean, median, range and standard deviation, by survey round) of questions posed to expert panel for type 1 and type 2 patients.

		Type 1 patient — IV therapy			Type 1 patient — IP/IV therapy			Type 2 patient — IV therapy			Type 2 patient — IP/IV therapy
		Round 1	Round 2	Round 3	Round 1	Round 2	Round 3	Round 1	Round 2	Round 3	Round 1	Round 2	Round 3
% with ED or hospital visits	Mean	15.3	14.4	9.6	40.4	33.6	23.5	31.1	28.4	21.2	52.7	47.7	34.9
	Median	10	10	10	20.5	21.5	20	27.5	25.5	23.5	40	40	35
	SD	14.7	10.1	3.5	36.2	25.9	8.0	15.2	13.9	4.8	30.0	21.1	4.1
	Range	5–50	5–33	5–15	10–100	12–100	15–40	11–60	10–62	14–26	20–100	30–100	30–40
% completing 6 cycles of chemotherapy	Mean	89.9	92.6		63.8	62.4		84.5	82.6		40.6	43.1
	Median	95	95		64	60		85.5	83		40	47.5
	SD	8.4	3.3		14.0	9.1		8.7	6.3		16.6	13.6
	Range	75–99	86–95		40–80	50–80		71–95	75–90		14–67	10–60
% surviving 5 years	Mean	51.7	52.7		67.3	66.1		40.5	39.1		49.1	47.1
	Median	50	52		68.5	66		37	40		51	50
	SD	9.6	7.2		9.0	5.5		9.5	5.8		13.1	8.0
	Range	40–70	40–63		55–80	55–74		30–62	30–50		31–72	30–55
Median survival (months)	Mean	58.4	57		81.4	82.5		47.4	47.7		56.1	57.7
	Median	57.5	59.5		72	71		49	48		60	60
	SD	11.6	7.0		20.0	21.6		9.0	5.1		11.6	7.1
	Range	43–82	48–68		65–110	65–120		30–60	40–58		35–70	48–68

Abbreviations: IV, intravenous; IP, intraperitoneal; ED, emergency department; SD, standard deviation.

Fig. 2.

Distribution of responses to the question of the expected median survival (in months) in a type 2 patient undergoing IV (panel A) or IP/IV (panel B) during round 2.

Overall, the range of estimates was smaller for IV than IP/IV therapy (Table 2). When comparing the ranges with each subsequent round of the survey, ranges and standard deviations decreased with each round. Ultimately, the majority of the group opinions converged within tighter ranges, although some outlier opinions remained. Consensus converged around the following values: 1) percentage of patients expected to complete 6 cycles of IP/IV chemotherapy as prescribed (type 1 patient = 62%, range 50%–80%; type 2 patient = 43%, range 10%–60%); 2) percentage of patients expected to survive 5 years after IP/IV (type 1 patient = 66%, range 55%–74%; type 2 patient = 47%, range 30%–55%); and 3) median survival after IP/IV (type 1 patient = 83 months, range 65–120; type 2 patient = 58 months, range 48–68). Given a continued lack of consensus on the probability of hospitalization or emergency department visits after IP/IV chemotherapy after two rounds of the modified Delphi technique, a third round was performed, leading to a narrowing of the range (type 1 = 24%, range 15%–40%; type 2 = 35%, range 30%–40%). Regarding median survival after IP/IV, there were three persistent outlier respondents, whose comments justifying their responses led the investigators to believe that no further consensus could be achieved; thus, the range in responses to this question remained broad and no additional rounds of this question were undertaken. In cases where there was one or more outlier responses resulting in wider response range, the median may give a truer reflection of the consensus response than the mean. Due to the usual small number of experts used with the Delphi technique, the impact of length of practice, type of training, or geographic location on responses could not be assessed.

4. Discussion

We utilized a modified Delphi technique to establish consensus clinical estimates from an expert panel in the absence of definitive data. The use of expert opinion was necessary since data from randomized clinical trials may not be applicable to the general ovarian cancer population due to the restrictive eligibility and exclusion criterion required for entry onto the clinical trial. In developing a decision tool to aid women preparing to undergo chemotherapy for advanced ovarian cancer, estimates were required regarding toxicity and survival associated with different routes of administration of chemotherapy in patients who are generally healthy as well as in those who are considered less healthy (but still candidates for IP/IV chemotherapy). Through the process of surveying a panel of experts, with anonymous responses of the panel being shared between rounds of surveys (and, when appropriate, modification of questions to improve comprehension between rounds), ranges and standard deviations decreased. Thus, “best estimates” for pertinent uncertainties were established for use in the decision aid.

Shared decision making and the use of decision aids have been recognized as being critically important in helping patients understand complex healthcare options. Decision aids have been shown to increase patients' knowledge about their medical options and potential risks, and to make the patient's role in treatment decision-making more active [12]. The decision aid currently being tested by our group incorporates the expert opinions gathered in this study about the outcomes of treatment with IP and IV chemotherapy, clinical trial data regarding the likelihood of specific adverse events of treatment, and patients' personal preferences for acceptable tradeoffs between survival time and severe toxicities of treatment. We hope both to educate women about their treatment choices and to increase the likelihood that they are satisfied with their choice.

There are clearly limitations in the use of the modified Delphi technique in this study, and in any study where there is a lack of established data from the results of randomized clinical trials. Since there is no “correct answer” to the questions asked of the expert panel, the consensus responses are different than those reported from clinical trial of IP/IV versus IV chemotherapy [1]; it will be interesting to contrast these results from those from GOG 252 when published. It is important to note that randomized controlled clinical trials like GOG 252 are, by definition, “explanatory” trials, aimed to test whether an intervention works under optimal situations, in contrast to “pragmatic” trials, which aim to evaluate the effectiveness of interventions in real-life routine practice. While the results from a well-designed phase III randomized controlled clinical trial are critically important for establishing standards of care in patient management, they are not always applicable to patients seen in routine practice; thus opinions from experts regarding the outcomes of patients who may not fit into the inclusion and exclusion criteria of a clinical trial may offer more realistic outcome data that is relevant to the accurate counseling of patients. With this modified Delphi process, it is interesting to hypothesize as to the reason for outlier responses. Once can envision that an initial estimate for any clinical variable may be “anchored” by results from a recent clinical trial, modified by previously reported clinical trials, and refined by the personal experience of the expert. More weight given to personal experience (over a recent clinical trial, for example), may led to disparate answers to the same clinical scenario. Given the small sample size that is typical for the Delphi technique, an assessment of the impact of the background of the experts themselves (for example, practice location, length of practice, or type of training) on responses was unable to be assessed.

Through this iterative survey process, narrowing of ranges and standard deviations are seen, with final results converging around a single value. In our survey, the response to the question of the “probability of emergency room or hospital visits during treatment” continued to have a very large range of responses for patients undergoing IP/IV chemotherapy (for both the type 1 and type 2 patients), with standard deviations exceeding 30% (with a median response of 20–40% for both patient scenarios) after the first round. Comments were provided by experts to qualify these ratings, and were useful in identifying issues that seemed to influence a lack of consensus (for example, possible regional practice differences) [13]. In practice, the comments both enabled the revision of questions for clarity and improved consensus and also helped us to understand the outliers and why there was a greater difference of opinion for some questions. For example, for the question addressing the “probability of emergency room or hospital visits during treatment”, comments from the experts led us to add the word “unexpected” to describe the emergency room or hospital visit (since some patients routinely receiving IP/IV chemotherapy go to a hospital setting for hydration following chemotherapy); this resulted in a narrowing of the range of responses (with the standard deviation being less than 8%, with a median response of 20–35% in both patient scenarios) after the third round of surveys. However, we recognized that the question addressing “median survival time” for type 1 patients receiving IP/IV chemotherapy had three outlier responses that were not going to change with subsequent rounds of the survey. Thus, we accepted a wider range for this question rather than administering a third round of the survey.

In summary, this modified Delphi survey technique enabled us to derive expert opinion-based estimates of survival and the probability of severe adverse events associated with IP/IV chemotherapy. Estimates of median survival following optimal primary debulking ranged from 82 months (good health, no macroscopic residual disease following debulking, IP chemotherapy) to 48 months (fair health, minimal macroscopic residual disease, IV chemotherapy). In the face of controversial therapies, the Delphi process can elucidate areas of controversy and build consensus to improve clinical practice. It may also be a useful tool to inform patients and practitioners of expected outcomes in the typical patient (who may not fit into the perfect mold of an “explanatory” clinical trial), when randomized clinical trials do not exist or when only limited data from clinical trials exist.

Highlights.

• The Delphi technique provides expert opinion when accurate data does not exist.

• The Delphi technique achieves consensus with multiple iterations of the same question.

• The Delphi technique is a useful tool for modeling studies.