Abstract
Background
Several challenges presently impede the conduct of prospective clinical studies in orthopaedic oncology, including limited financial resources to support their associated costs and inadequate patient volume at most single institutions. This study was conducted to prioritize research questions within the field so that the Musculoskeletal Tumor Society (MSTS), and other relevant professional societies, can direct the limited human and fiscal resources available to address the priorities that the stakeholders involved believe will have the most meaningful impact on orthopaedic oncology patient care.
Questions/purposes
The purpose of this study was to use a formal consensus-based approach involving clinician-scientists and other stakeholders to identify the top priority research questions for future international prospective clinical studies in orthopaedic oncology.
Methods
A three-step modified Delphi process involving multiple stakeholder groups (including orthopaedic oncologists, research personnel, funding agency representation, and patient representation) was conducted. First, we sent an electronic questionnaire to all participants to solicit clinically relevant research questions (61 participants; 54% of the original 114 individuals invited to participate returned the questionnaires). Then, participants rated the candidate research questions using a 5-point Likert scale for five criteria (60 participants; 53% of the original group participated in this portion of the process). Research questions that met a priori consensus thresholds progressed for consideration to an in-person consensus meeting, which was attended by 44 participants (39% of the original group; 12 countries were represented at this meeting). After the consensus panel’s discussion, members individually assigned scores to each question using a 9-point Likert scale. Research questions that met preset criteria advanced to final ranking, and panel members individually ranked their top three priority research questions, resulting in a final overall ranking of research priorities.
Results
A total of 73 candidate research questions advanced to the consensus meeting. In the end, the consensus panel identified four research priorities: (1) Does less intensive surveillance of patients with sarcoma affect survival? (2) What are the survival outcomes over time for orthopaedic oncology implants? (3) Does resection versus stabilization improve oncologic and functional outcomes in oligometastatic bone disease? (4) What is the natural history of untreated fibromatosis?
Conclusions
The results of this study will assist in developing a long-term research strategy for the MSTS and, possibly, the orthopaedic oncology field as a whole. Furthermore, the results of this study can assist researchers in guiding their research efforts and in providing a justified rationale to funding agencies when requesting the resources necessary to support future collaborative research studies that address the identified orthopaedic oncology priorities.
Electronic supplementary material
The online version of this article (doi:10.1007/s11999-017-5482-7) contains supplementary material, which is available to authorized users.
Introduction
Sarcomas are a rare and heterogeneous group of cancers that represent < 1% of all malignancies [24]. However, as a result of their clinical behavior, multidisciplinary management, and complex multimodal treatment, the impact to patients is significant and the cost of care is substantial [7, 13, 27]. Moreover, the skeleton is the third most common organ affected by metastatic cancer [36]. In the advent of improved medical treatment of many cancers, bone metastases are becoming increasingly prevalent because patients with cancer are living longer with their disease [1, 2]. Many critical questions surrounding the surgical management of patients with bone and soft tissue tumors and metastatic bone disease remain unanswered, but funding to support prospective clinical research is meager in comparison to basic science research [34]. Furthermore, a recent qualitative study determined that orthopaedic oncology collaborators are enthusiastic to conduct further research, provided that the research questions are feasible and address clinically relevant endpoints [28]. Therefore, it is important to identify research priorities through a systematic and thoughtful process.
An effective research system should address health issues, interventions, and outcomes of concern within a given field that are considered important by both clinicians and patients. The Delphi method is one approach that can be utilized to amass the expertise and opinions from various stakeholders in an area and use it to determine a suitable set of research priorities through consensus. This method has been previously used in the development of research agendas in other specialties. For example, the Acute Kidney Injury Network (AKIN) utilized a Delphi process to construct a research agenda to guide future clinical and translational acute kidney injury (AKI) research [17, 21]. This collaborative network has since conducted multiple studies that address many of the identified priorities and has generated evidence to improve the care of patients with AKI [11, 16, 25]. Their work also exemplifies the impact a formal research agenda can have on clinical care in a given field. However, research priorities have yet to be identified in the field of orthopaedic oncology.
The purpose of this study was to use a consensus-based approach to prioritize research questions within orthopaedic oncology so that efforts and available resources can then be streamlined toward addressing these priorities.
Materials and Methods
The protocol for this initiative was previously reported elsewhere in further detail [29]. Briefly, a three-step modified Delphi process was conducted between April and November 2016, consisting of a qualitative assessment (Phase I), a rating evaluation (Phase II), and a consensus panel meeting (Phase III) (Fig. 1). During the qualitative assessment, participants were asked to identify a maximum of three research questions that they believed most urgently needed answering to guide patient care. In the rating evaluation, participants were asked to individually rate each candidate research question based on the definitions provided of five criteria that are considered crucial to the development of a realistic yet compelling research question: scientific merit, clinical significance, innovation, relevance, and feasibility (Table 1). Research questions from the rating evaluation that met the a priori consensus thresholds outlined in detail subsequently were brought forward to the consensus panel meeting for facilitated discussion. Thereafter, each consensus panel member anonymously scored each candidate research question; those that met the predetermined criteria were then brought forward for final ranking. The consensus panel members were then asked to rank their top three research questions, which were then distributed to all participants for validation.
Fig. 1.
A flow diagram shows the Moving Forward Through Consensus Orthopaedic Oncology Research Program planning initiative.
Table 1.
Phase II criteria definitions
| Criteria | Criteria definitions |
|---|---|
| Scientific merit | Clear, concise research question with a strong rationale (eg, clinical equipoise) and approach |
| Clinical significance | Potential impact on clinical practice |
| Innovation | Explores new scientific avenues, has a novel hypothesis, and will create new knowledge |
| Relevance | Importance to patient populations and key strategic research priorities |
| Feasibility | Potential to recruit participants in a timely manner, obtain funding, and expand internationally |
Before conducting this consensus-based process, ethics approval was received from the Hamilton Integrated Research Ethics Board (Project No. 1765). For the qualitative assessment and rating evaluation, written informed consent was not obtained from participants because the completion and submission of questionnaires were considered implied consent. For the in-person consensus meeting participants, written informed consent was obtained before the commencement of the meeting; however, for remote participants, written informed consent was not obtained because access to the remote response system and submission of responses were again considered implied consent. All suggested research questions, scores, and rankings were kept strictly confidential and all identifiers were removed before any analysis. Data with direct identifiers were available to only one member of the planning committee (PJS).
An initial invitation to gauge interest in participating in this research program planning initiative was distributed to 351 individuals by email on May 24, 2016. The invitation was sent to all clinicians on the Prophylactic Antibiotic Regimens in Tumor Surgery (PARITY) network distribution list, which includes both individuals who have either expressed an interest or are actively participating in the PARITY trial (NCT No. 01479283) [6] as well as members of the Musculoskeletal Tumor Society (MSTS). Of the 351 individuals, 117 responded to the email (33%). Almost all of these 117 respondents (114 individuals) indicated that they would be interested in participating in all phases of this initiative with only two individuals indicating that they did not wish to participate and one individual wanting to participate only in the consensus meeting.
Phase I: Qualitative Assessment–Soliciting Research Questions of Interest
Participants
To be eligible for the qualitative assessment, participants had to be a clinician who was either: (1) interested or actively participating in the PARITY trial; or (2) an active or candidate MSTS member. An Orthopaedic Research and Education Foundation (OREF) representative and a clinical research manager of an orthopaedic oncology unit at a major clinical site (with extensive practical experience in orthopaedic oncology clinical research) were also included. A total of 114 individuals were invited to participate. Sixty-one individuals (54% response rate) provided complete responses to the Phase I questionnaire (Fig. 1). Of the participants, all but one were orthopaedic surgeons (98%), and almost all (90%) were men. Of the orthopaedic surgeons who participated, 88% had completed a fellowship in orthopaedic oncology. Participants represented orthopaedic oncology practices from 16 countries (Table 2).
Table 2.
Participant demographics
| Characteristic | Total number (%) | Phase I, number (%) | Phase II, number (%) | Phase III, number (%) |
|---|---|---|---|---|
| Number of participants | 82 | 61 | 60 | 44 |
| Gender | ||||
| Male | 69 (84) | 54 (89) | 51 (85) | 36 (82) |
| Female | 13 (16) | 7 (11) | 9 (15) | 8 (18) |
| Age (years) | ||||
| Younger than 30 | 0 (0) | 0 (0) | 0 (0) | 0 (0) |
| 30–40 | 34 (41) | 26 (43) | 25 (42) | 21 (48) |
| 41–50 | 25 (30) | 17 (28) | 21 (35) | 13 (30) |
| 51–60 | 18 (22) | 15 (25) | 10 (17) | 9 (20) |
| Older than 60 | 5 (6) | 3 (5) | 4 (7) | 1 (2) |
| Country | ||||
| Africa | 1 (1) | 1 (2) | 1 (2) | 1 (2) |
| South Africa | 1 | 1 | 1 | 1 |
| Asia | 4 (5) | 3 (5) | 4 (7) | 3 (7) |
| India | 1 | 1 | 1 | 1 |
| Israel | 1 | 1 | 1 | 1 |
| Japan | 2 | 1 | 2 | 1 |
| Europe | 11 (13) | 8 (13) | 9 (15) | 4 (9) |
| Austria | 1 | 1 | 1 | 1 |
| Denmark | 1 | 0 | 1 | 1 |
| Italy | 2 | 2 | 1 | 0 |
| Netherlands | 2 | 1 | 2 | 1 |
| Norway | 1 | 1 | 1 | 0 |
| Spain | 2 | 2 | 2 | 1 |
| Sweden | 1 | 0 | 1 | 0 |
| United Kingdom | 1 | 1 | 0 | 0 |
| North America | 58 (71) | 44 (72) | 42 (70) | 32 (73) |
| Canada | 6 | 5 | 3 | 3 |
| United States | 52 | 39 | 39 | 29 |
| Oceania | 1 (1) | 1 (2) | 0 (0) | 0 (0) |
| Australia | 1 | 1 | 0 | 0 |
| South America | 7 (9) | 4 (7) | 4 (7) | 4 (9) |
| Argentina | 1 | 1 | 1 | 1 |
| Brazil | 3 | 1 | 1 | 3 |
| Colombia | 3 | 2 | 2 | 0 |
| Occupation* | ||||
| Orthopaedic surgeon | 79 (96) | 59 (97) | 58 (97) | 41 (93) |
| Orthopaedic oncology fellow | 1 (1) | 1 (2) | 1 (2) | 1 (2) |
| Clinical research manager | 1 (1) | 1 (2) | 1 (2) | 1 (2) |
| Patient representative | 1 (1) | 0 (0) | 0 (0) | 1 (2) |
| OREF representative | 1 (1) | 0 (0) | 1 (2) | 1 (2) |
| Completion of orthopaedic oncology fellowship | ||||
| Yes | 73 (89) | 53 (87) | 52 (87) | 37 (84) |
| No | 3 (4) | 3 (5) | 3 (5) | 3 (7) |
| Not applicable | 6 (7) | 5 (8) | 5 (8) | 4 (9) |
| Years in practice | ||||
| Less than 5 | 17 (21) | 13 (21) | 12 (20) | 12 (27) |
| 5–10 | 21 (26) | 15 (25) | 18 (30) | 13 (30) |
| 11–15 | 14 (17) | 13 (21) | 11 (18) | 5 (11) |
| 16–20 | 8 (10) | 3 (5) | 6 (10) | 5 (11) |
| Over 20 | 20 (24) | 16 (26) | 12 (20) | 7 (16) |
| Not applicable | 2 (2) | 1 (2) | 1 (2) | 2 (5) |
| Type of institution | ||||
| Academic | 75 (92) | 56 (92) | 55 (92) | 42 (95) |
| Nonacademic | 6 (7) | 5 (8) | 5 (8) | 1 (2) |
| Not applicable | 1 (1) | 0 (0) | 0 (0) | 1 (2) |
| Proportion of practice constitutes management of orthopaedic oncology patients | ||||
| 0%–25% | 2 (2) | 2 (3) | 2 (3) | 1 (2) |
| 26%–50% | 9 (11) | 6 (10) | 8 (13) | 4 (9) |
| 51%–75% | 17 (21) | 13 (21) | 8 (13) | 10 (23) |
| 76%–100% | 50 (61) | 38 (62) | 39 (65) | 26 (59) |
| Not applicable | 4 (5) | 2 (3) | 3 (5) | 3 (7) |
*In instances where the OREF representative participated, the total will equal more than the total number of participants because the OREF representative was also an orthopaedic oncologist (and, therefore, was counted twice); OREF = Orthopaedic Research and Education Foundation.
Interventions
The invitation to complete a web-based, open-ended questionnaire (with unrestricted answers) was distributed by email on June 14, 2016. The questionnaire was not accompanied by any additional facilitators or literature reviews. Rather, participants were asked to review the literature and consult with colleagues as they saw fit before proposing their ideas. The questionnaire also requested some demographic data and any potential financial or intellectual conflicts of interest. The questionnaire remained online for 4 weeks, and reminder emails were sent approximately every 7 days after the initial invitation to those who had not yet completed the questionnaire.
Phase II: Rating Evaluation–Rating of Research Questions
Participants
The same eligibility criteria for qualitative assessment were maintained for the rating evaluation. In total, 113 individuals (one individual withdrew) were invited to participate in this part of the study. Sixty individuals (59 complete, one incomplete; 53% response rate) responded to the rating evaluation questionnaire (Fig. 1). Of the participants, all but one were orthopaedic surgeons (97%); however, one of these orthopaedic surgeons specialized in veterinary medicine. Over two-thirds of the participants (85%) were men. Of the orthopaedic surgeons who participated, 88% had completed a fellowship in orthopaedic oncology. Participants represented orthopaedic oncology practices from 16 countries (Table 2).
Interventions
The second phase of the modified Delphi process used a web-based questionnaire that asked participants to rate each candidate research question individually on a 5-point Likert scale for five criteria: scientific merit, clinical significance, innovation, relevance, and feasibility (Table 3). Participants were also provided with the opportunity to further clarify or add to research questions, if necessary, and recommend additional research questions. Participants were invited to complete this questionnaire by email on August 10, 2016. The questionnaire remained online for 4 weeks, and reminder emails were sent approximately 7 days after the initial invitation to individuals who had not yet completed the questionnaire.
Table 3.
Phase II 5-point Likert scale
| 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|
| Strongly disagree | Disagree | Neither agree nor disagree | Agree | Strongly agree |
Phase III: Consensus Meeting–Vetting and Ranking of Research Priorities
Participants
Rather than utilize a random sample that is representative of the target population, the Delphi method uses a consensus panel of those invested in the process and its outcomes. There is little agreement on the definition of an expert [3]. Therefore, for the purposes of the current study, all participants from the qualitative assessment and rating evaluation were invited to participate as well as representatives from patient advocacy groups, the MSTS, and OREF. One hundred one individuals were invited to participate. In response to numerous requests, mostly from international clinicians, we opted to also provide a web-based response system and conferencing capabilities, which allowed us to ensure that the consensus panel membership was geographically diverse. The consensus panel included a total of 44 individuals from 12 countries (Fig. 1). The panel was comprised of one patient representative, one clinical research manager, and 42 were orthopaedic surgeons. The OREF selected a representative to attend the meeting, who is also an orthopaedic oncologist. Of the orthopaedic surgeon members, half (50%) were between the ages of 30 and 40 years, and almost two-thirds (57%) had been in practice for no more than 10 years (Table 2).
Interventions
The consensus meeting was held on October 7, 2016. The meeting was facilitated by an experienced and independent meeting facilitator with extensive prior involvement in strategic planning initiatives. At this meeting, the members of the consensus panel were given the opportunity to discuss the eligible candidate research questions as a group. A semistructured agenda that had flexible time parameters to allow for discussion and questions was followed, as previously published [29]. The consensus meeting was recorded, and all consensus panel members were informed that it was being recorded before the start of the meeting.
Absolute Scoring Stage
Onsite participants were provided with an audience response system device to anonymously score research questions throughout the meeting. Remote participants were provided with access to anonymously score research questions through the Internet in real time throughout the meeting. All candidate research questions were individually discussed by members of the consensus panel, thereby providing an opportunity for members to reconsider their Phase II ratings in light of other members’ views. However, priority was given to the discussion of higher ranking tiers. For each question, the facilitator first asked the consensus panel for any remarks in support of the research question; then the facilitator asked for any comments against the research question before asking for any additional comments. When differences in the ratings from Phase II appeared to have resulted from ambiguity in the wording of a research question, the members also used this time to agree on revised wording. After each question’s discussion, the members anonymously assigned a score using the response system.
Final Ranking Stage
Utilizing a web-based questionnaire, the candidate research questions that met the predetermined criteria as outlined below were subsequently distributed to the consensus panel members who were asked to rank their top three research questions [29]. All individuals (100%) involved in the consensus meeting provided their final rankings.
Analysis
Phase I: Qualitative Assessment–Soliciting Research Questions of Interest
The questionnaire responses were compiled for review. Responses from the qualitative assessment were initially reviewed by an orthopaedic oncologist (MG) on the planning committee. Similar ideas were clustered together into emerging research questions, and duplicate responses were removed. This review was independently repeated by a health research methodology expert (NE) on the planning committee. The two reviewers then met to discuss any differences produced from these independent reviews and reach a consensus on the list of candidate research questions that would progress to the next phase.
Phase II: Rating Evaluation–Rating of Research Questions
The planning committee compiled the ratings for each research question as well as any additional research questions. The results of the questionnaire were reviewed to determine whether each candidate research question met the predetermined consensus thresholds (Table 4). Research questions that met either the inclusion or nonconsensus thresholds progressed to Phase III for review by the consensus panel. Candidate research questions that met the exclusion consensus threshold were not brought forward for review.
Table 4.
Phase II consensus thresholds
| Consensus threshold | Threshold definition |
|---|---|
| Inclusion | Greater than 75% of respondents provide a positive result (four or five) on the Likert scale for all criteria |
| Exclusion | Greater than 75% of respondents provide a negative result (one or two) on the Likert scale for all criteria |
| Nonconsensus | When the candidate research question has met neither the inclusion nor exclusion consensus thresholds |
Phase III: Consensus Meeting–Vetting and Ranking of Research Priorities
Absolute Scoring Stage
Consensus panel members anonymously assigned a score on a 9-point Likert scale for each candidate research question (Table 5). A larger Likert scale was used during this stage because it provided a better opportunity for respondents to adequately report their opinions on the relative importance of each candidate research question. This was especially important during this stage considering the large number of candidate research questions that progressed for in-person vetting.
Table 5.
Phase III scoring scale
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|
| Should not be studied | Lowest priority | Very low priority | Low priority | Medium priority | Slightly high priority | Moderately high priority | High priority | Highest priority |
Once the scores were compiled, questions meeting one of the following predetermined criteria were brought forward for final ranking: 100% of respondents scored the candidate research question as a 7, 8, or 9 or at least 10% of respondents scored the research question as a 9. If none of the research questions met these criteria, it was also decided a priori that the top 10 scoring candidate research questions would be brought forward for final ranking [29].
Final Ranking Stage
Consensus panel members then ranked their top three research questions. The corresponding point system was used to determine the highest ranking candidate research questions (Table 6). We decided a posteriori that a multiple regression analysis of the consensus meeting data, as stated in the study’s protocol [29], was not appropriate given the relatively small final sample size and the large number of priority research questions brought forward by the participants.
Table 6.
Phase III final ranking scoring system
| Rank | First | Second | Third |
|---|---|---|---|
| Points | 3 | 2 | 1 |
Results
Phase I: Qualitative Assessment–Soliciting Research Questions of Interest
In total, 175 candidate research questions were proposed by the respondents. After the qualitative assessment review, during which 106 duplicates were removed, it was determined that 69 candidate research questions would progress to the rating evaluation phase (Appendix 1 [Supplemental materials are available with the online version of CORR ®.]).
Phase II: Rating Evaluation–Rating of Research Questions
In total, 60 individuals (59 complete, one incomplete) participated in the rating evaluation, rating the 69 candidate research questions on five criteria. Four additional research questions were proposed (Note: after the consensus meeting, none of these four questions were ranked highly.). As per the consensus thresholds (Table 4), only one candidate research question met the inclusion threshold. Sixty-eight research questions met the nonconsensus threshold, and no research questions met the exclusion threshold. Therefore, 73 candidate research questions (69 candidate research questions plus four new suggestions) progressed to the final phase for consideration and assessment at the consensus meeting (Appendix 2 [Supplemental materials are available with the online version of CORR ®.]).
As a result of the large number of candidate research questions that met the conditions for advancement, we divided them into the following seven tiers in preparation for the consensus meeting: Tier 1–met inclusion threshold; Tier 2–met nonconsensus threshold (average consensus score ≥ 70%); Tier 3–met nonconsensus threshold (average consensus score = 65%–69%); Tier 4–met nonconsensus threshold (average consensus score = 60%–64%); Tier 5–met nonconsensus threshold (average consensus score = 50%–59%); Tier 6–met nonconsensus threshold (average consensus score < 50%); and Tier 7–new suggestions. This was done to minimize time constraints and prioritize discussion of candidate research questions that had higher degrees of consensus in the rating evaluation phase.
Phase III: Consensus Meeting–Vetting and Ranking of Research Priorities
Final Research Priorities
The final stage identified four research priorities with proposed study methodologies (Table 7): (1) Does less intensive surveillance of patients with sarcoma affect survival? (2) What are the survival outcomes over time for orthopaedic oncology implants? (3) Does resection versus stabilization improve oncologic and functional outcomes in oligometastatic bone disease? (4) What is the natural history of untreated fibromatosis?
Table 7.
Priority research questions for future research in the field
| Rank | Priority research question |
|---|---|
| First rank | Does less intensive surveillance of sarcoma patients affect survival? (methodology: randomized controlled trial) |
| Second rank | What are the outcomes over time for orthopaedic oncology implants? (methodology: prospective registry) |
| Third rank | Does resection versus stabilization improve outcomes in oligometastatic bone disease? (methodology: randomized controlled trial) |
| Fourth rank | What is the natural history of untreated fibromatosis? (methodology: prospective cohort) |
Absolute Scoring Stage
The final top research priorities were identified through the scoring of each research question. Two candidate research questions met the criteria for progression to the final ranking stage. This was considered a priori an insufficient number of candidate research questions to conduct the final ranking, so the mean score was calculated for each research question and the top 10 were brought forward as per protocol [29]. Because one research question that met the criteria for progression did not score in the top 10 by mean score, we decided that 11 candidate research questions would advance to the final stage of this phase (Table 8).
Table 8.
Phase III: priority research questions advancing to final ranking stage
| Question number | Mean score | SD | Priority research question |
|---|---|---|---|
| 58 | 7.0 | 1.3 | What is the natural history of untreated fibromatosis?*,† |
| 15 | 6.5 | 2.3 | Does less intensive surveillance of patients with sarcoma affect survival?† |
| 46 | 6.4 | 2.1 | Does resection versus stabilization improve oncologic outcomes in oligometastatic bone disease without compromising functional and surgical outcomes?† |
| 14 | 6.0 | 2.4 | What are the outcomes over time for orthopaedic oncology implants?† |
| 49 | 5.9 | 2.2 | Do circulating tumor cells in patients with osteosarcoma predict outcome?† |
| 4 | 5.9 | 1.9 | Do local bisphosphonates and/or cementation reduce local recurrence rates in resectable GCT?† |
| 35 | 5.9 | 1.9 | What are the functional and surgical outcomes of resection versus internal fixation for metastatic bone disease of the proximal femur?† |
| 1 | 5.7 | 1.6 | Does denosumab and/or systemic bisphosphonates reduce recurrence rates in resectable GCT?† |
| 38 | 5.5 | 2.5 | What are the patient reported outcomes in rotationplasty versus growing prostheses in children?† |
| 40 | 5.2 | 2.1 | What is the optimal filling agent for cavitary bone defects?† |
| 10 | 4.9 | 2.4 | Development of decision-making algorithm for metastatic bone disease and validation of existing prediction models.* |
*Met the a priori inclusion criteria for progression to final ranking stage; †scored in the top 10 candidate research questions (by mean score); GCT = giant cell tumor.
Final Ranking Stage
After each consensus meeting participant provided their final rankings, the top priority research questions were identified as outlined previously (Table 7). The research question with the fourth highest score was also included in this list because the total number of panel members that ranked this question in their top three was higher than that of both the second and third ranking questions with a final score only marginally lower than that of the third ranking question.
Discussion
Previous work has demonstrated a lack of high-quality evidence to guide clinical decisions in orthopaedic oncology [5]. As a result of the rarity of bone and soft tissue tumors, multicenter prospective collaboration is imperative for broadly meaningful research and evidence-based advances in patient care [18]. However, although agreement exists pertaining to the importance of collaborative research and the need for higher quality research in orthopaedic oncology, research priorities remain unclarified. Consensus methods are being increasingly used to develop research agendas in various medical and surgical specialties [17, 20, 23]. Research agendas can assist professional groups in allocating finite research resources to clinical investigations likely to provide the greatest value. They can also provide individual researchers with guidance to help prioritize their own endeavors. To establish a research agenda specific to orthopaedic oncology, we brought together international stakeholders and conducted a modified Delphi process, which identified the following four priorities: (1) Does less intensive surveillance of patients with sarcoma affect survival? (2) What are the survival outcomes over time for orthopaedic oncology implants? (3) Does resection versus stabilization improve oncologic and functional outcomes in oligometastatic bone disease? (4) What is the natural history of untreated fibromatosis?
Our study has some limitations that should be recognized. Although diverse, our sample’s composition was still dominated by individuals from North America, so our results may reflect a North American perspective that is not applicable to all regions as a result of differences in access to health care, referral patterns, availability of medical interventions, and cultural acceptability of treatment. However, these imbalances in perspective may have been resolved through discussions with international colleagues during the consensus meeting. Furthermore, underrepresentation of distinct groups may have resulted from our sampling method. We invited all interested or active PARITY investigators and MSTS members to participate in this initiative. However, a substantial number of orthopaedic oncologists are not affiliated with either the PARITY trial or MSTS; therefore, they may not be adequately represented in our study. Although we made numerous attempts to contact orthopaedic oncology patient advocacy groups for their input, we were only able to connect with one patient advocate, who was contacted directly and who was interested in supporting orthopaedic oncology research. It was, therefore, fruitful to directly contact patients rather than patient advocacy groups. Nevertheless, future opportunities exist to include a greater number of patient advocates and underrepresented stakeholders such as during the development process of specific research protocols to address the identified priority research questions. Finally, our response rates (33% [117 of 351] for the initial invitation and 54% [61 of 114] and 53% [60 of 113] for the qualitative assessment and rating evaluation questionnaires, respectively) were somewhat lower than those achieved in similar studies in other medical specialties [20, 23]. However, our initial invitation allowed us to specifically select a population that is interested in collaborative research in the field. Considering that our objective was to determine the priority research questions in orthopaedic oncology to drive future collaborative prospective research, this bias may have been favorable and resulted in inherently collaborative participants providing many insightful ideas. Individuals who did not respond to the initial invitation may not have been interested in participating in any collaborative research, may not have expected this initiative to be fruitful, or did not prioritize this initiative above other academic interests. However, response bias was minimized by allowing individuals to participate in any phase of the study. Future studies that are aimed at more rigorously evaluating potential sampling biases should include European and Asian orthopaedic oncology societies.
Despite these limitations, the design of the current initiative was structured, thorough, transparent, and aimed to include all invited individuals who expressed an interest. Furthermore, the use of a modified Delphi process for this initiative maximized the benefits of two common consensus approaches—the classic Delphi method and the Nominal Group Technique. Throughout the initiative, anonymity in scoring was maintained, even while the moderated discussion took place; this provided the opportunity for the causes of disagreement to be explored without particularly vocal participants dominating the discussion and overpowering the opinions of others [8, 15]. Therefore, it is unlikely that consensus was forced on participants nor that the priority research questions identified reflect the perspective of any one participant. Finally, the composition of the participant group is also thought to influence the outcome of the consensus process. Other studies have previously demonstrated that homogenous samples select different options than heterogeneous ones considering the same choices [31]. To maximize the feasibility and generalizability of the identified research priorities, we assembled a group of participants from a broad range of healthcare systems, geographic regions, career focus and stage, and prior research experience.
The natural succession of the identification of research priorities is the development of an action plan to answer the priority research questions and to identify viable funding mechanisms to support these research efforts. Surveillance after sarcoma treatment is a subject that spans all disciplines in cancer care. The balance between the cost of intense surveillance with respect to resources and patient quality of life and the potential benefit to identify relapsed disease in a curable stage must take into account many important factors [7, 10, 19, 22, 35]. This is, therefore, a complex question that will require coordinated protocol development among a long list of stakeholders, the most important being the patients themselves. Puri et al. [26] have published the only relevant randomized controlled trial (RCT) to date in the sarcoma field. This single-center study concluded that overall 3-year survival and disease-free survival were no different between patients with sarcoma who had more intensive surveillance (CT scans) and those with less (chest radiographs) [26]. However, as a result of the sample size, this trial could not conclusively demonstrate noninferiority in overall survival for a 6-monthly followup visit interval against a 3-monthly interval [26]. In addition, because this was a single-center study, generalizability of the results to other centers and countries is limited. A large international collaborative network will be required to implement a RCT protocol that addresses both the implications for healthcare systems and the preferences of patients with sarcoma. Government-level funding and large-scale grants from cancer research funding agencies would presumably be required to ensure the success of this effort.
Patients with sarcoma are often teenagers and young adults [24]. Therefore, those who survive after treatment may live for many decades. The implants used to reconstruct the extremities after tumor excision are prone to failure for a variety of reasons including infection, fracture, and aseptic loosening [14, 30]. Identifying emerging trends in the characteristics of those requiring revision surgery specific to the available implants may help identify risk factors so that the appropriate resources can be allocated to mitigate those risks. The success of the National Joint Registry (NJR) for England, Wales, Northern Ireland, and the Isle of Man in identifying implant designs that were failing at a proportionately higher rate illustrates the benefits of long-term observational data collection in other orthopaedic specialties [33]. An industry- or government-funded prospective implant registry specific to orthopaedic oncology would likely be the ideal approach to address this research priority.
With newer targeted systemic therapy, many patients with metastatic bone disease from primary carcinomas such as breast and renal cancers are living months and years longer with their disease [4, 32]. Therefore, a more aggressive surgical approach to resect entire bone metastases en bloc, as opposed to stabilization without tumor excision in its entirety, may be warranted to improve survival and possibly quality of life. This research priority could be answered with a multicenter RCT with a focus not only on survival rates, but also patient-reported outcomes specific to quality of life. However, such an endeavor would likely require generous government-level and nonprofit support. Similarly, there has been a paradigm shift in the treatment of fibromatosis (desmoid tumors), a benign diagnosis with aggressive local behavior. Once thought to be a surgical disease, it is now understood through retrospective data that with unacceptably high recurrence rates, “watchful waiting” may be the most appropriate management strategy for most patients [9, 12]. However, patients must be convinced that, if left untreated, the natural history of this disease is benign. A prospective multicenter cohort study that follows patients for symptoms and progression would assist in directing the care of patients in future generations. The Desmoid Tumor Foundation would be an ideal funding mechanism for this important study in orthopaedic oncology.
We have identified research priorities for international prospective research in orthopaedic oncology by conducting a three-step modified Delphi process. Top research priorities in orthopaedic oncology include evaluating different postoperative surveillance regimens in patients with extremity sarcoma, understanding the survival outcomes of orthopaedic oncology implants over time, evaluating whether resection versus stabilization improves outcomes in patients with oligometastatic bone disease, and understanding the natural history of untreated fibromatosis. These priority research questions highlight areas where international stakeholders have agreed by consensus that further knowledge would have a significant impact on the clinical care of orthopaedic oncology patients. Therefore, the MSTS and other professional orthopaedic oncology societies whose missions are to promote the advancement of orthopaedic oncology science and patient care may enable and support research efforts that address these priorities. These societies could do so by helping to facilitate the establishment of Working Groups and coordinating Working Group meetings, preferably by designating specific meeting times at annual conferences, to explore the next steps and develop action plans. The limited research funds of these professional societies could also be preferentially allocated to studies concentrating on one of the identified priorities. This research agenda could also be used by researchers to focus their research efforts and provide a rationale in competitive grant applications when applying for the financial resources to support endeavors directed at answering these priority research questions in orthopaedic oncology.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Acknowledgments
We thank Mr Bruce Withrow of Meeting Facilitators International for his objective advice on executing a strategic planning initiative and his impartial and unbiased facilitation of the consensus meeting. Reaching an agreement on these research priorities in orthopaedic oncology would not have been possible without his expertise. We would also like to thank the following individuals, who participated in at least one phase of this orthopaedic oncology research program planning initiative: Albert Aboulafia MD, MBA (MedStar Franklin Square, Georgetown University, Baltimore, MD, USA); John Abraham MD (The Rothman Institute, Thomas Jefferson University, Philadelphia, PA, USA); Brock Adams MD (MedStar Franklin Square, Georgetown University, Baltimore, MD, USA); Alexandre Arkader MD (The Children’s Hospital of Philadelphia, University of Pennsylvania, Philadelphia, PA, USA); Annie Arteau MD (L’Hôtel-Dieu de Québec, CHU de Québec, Université Laval, Québec, Canada); Raffi Avedian MD (Stanford University, Redwood City, CA, USA); Tessa Balach MD (University of Chicago, Chicago, IL, USA); André Mathias Baptista MD, PhD (Hospital de Clínicas de São Paulo, Universidade de São Paulo, São Paulo, Brazil); Ricardo Gehrke Becker MD, PhD (Hospital de Clínicas de Porto Alegre, Instituto do Câncer Infantil do Rio Grande do Sul, Porto Alegre, Brazil); Joseph Benevenia MD (University Hospital, Rutgers, The State University of New Jersey, Newark, NJ, USA); Marko Bergovec MD (Medical University Graz, Graz, Austria); Nicholas Bernthal MD (University of California Los Angeles, Los Angeles, CA, USA); B. Hudson Berrey MD (Baptist Medical Center, Jacksonville, FL, USA); Justin Bird MD (University of Texas MD Anderson Cancer Center, Houston, TX, USA); Michele Boffano MD (AOU Città della Salute e della Scienza, Torino, Italy); Scot Brown MD (University of Wisconsin, Madison, WI, USA); George Calvert MD (Norton Healthcare, Louisville, KY, USA); Edward Cheng MD (University of Minnesota, Minneapolis, MN, USA); Mark Clayer, MBBS (retired; Royal Adelaide Hospital, Adelaide, Australia); Sheila Conway MD (University of Miami, Miami, FL, USA); Benjamin Deheshi MD, MSc (McMaster University, Juravinski Hospital and Cancer Centre, Toronto, Ontario, Canada); P. D. Sander Dijkstra MD, PhD (Leids Universitair Medisch Centrum, Leiden, The Netherlands); Gregory Domson MD (Virginia Commonwealth University Health System, Richmond, VA, USA); Yee-Cheen Doung MD (Oregon Health and Science University, Portland, OR, USA); Nicole Ehrhart VMD, PhD (Colorado State University, Fort Collins, CO, USA); Cynthia Emory MD (Wake Forest University, Winston-Salem, NC, USA); Lisa Ercolano MD (Allegheny Health Network, Drexel University, Pittsburgh, PA, USA); Gary Friedlaender MD (Yale University, New Haven, CT, USA); Marcos Galli Serra MD (Hospital Universitario Austral, Universidad Austral, Pilar, Argentina); Barbara Gilligan, Krista Goulding MD (McGill University, Montreal, Quebec, Canada); David Greenberg MD (St Louis University, St Louis, MO, USA); Anthony Griffin MSc (Mount Sinai Hospital, Toronto, Ontario, Canada); James Hayden MD, PhD (Oregon Health and Science University, Portland, OR, USA); John Healey MD (Memorial Sloan-Kettering Cancer Center, Cornell University, New York, NY, USA); Robert Henshaw MD (MedStar Washington Hospital Center, Georgetown University, Washington, DC, USA); Asle Charles Hesla MD (Karolinska Institutet, Karolinska University Hospital, Stockholm, Sweden); Werner Hettwer MD (Rigshospitalet, Københavns Universitet, Copenhagen, Denmark); Ronald Hillock MD (Nevada Orthopedic & Spine Center, Las Vegas, NV, USA); Kelly Homlar MD (Augusta University, Augusta, GA, USA); Shintaro Iwata MD, PhD (National Cancer Center Hospital, Tokyo, Japan); Paul Jutte MD, PhD (Universitair Medisch Centrum Groningen, Groningen, The Netherlands); David King MD (Medical College of Wisconsin, Milwaukee, WI, USA); Daniel Lerman MD (University of Maryland, College Park, MD, USA); Adam Levin MD (Johns Hopkins University, Baltimore, MD, USA); Patrick Lin MD (University of Texas MD Anderson Cancer Center, Houston, TX, USA); Francisco Linares MD (Hospital Universitario San Ignacio, Pontifica Universidad Javeriana, Bogota, Colombia); Adam Lindsay MD (University of Connecticut, Farmington, CT, USA); Dieter M. Lindskog MD (Yale University, New Haven, CT, USA); Kevin MacDonald MD (Virginia Mason Medical Center, Seattle, WA, USA); Leonard Marais MBChB, MMed (Grey’s Hospital, University of KwaZulu-Natal, Durban, South Africa); Joel Mayerson MD (Wexner Medical Center, The Ohio State University, Columbus, OH, USA); Richard McGough MD (University of Pittsburgh, Pittsburgh, PA, USA); Nathan Mesko MD (The Cleveland Clinic, Cleveland, OH, USA); Benjamin Miller MD, MS (Holden Comprehensive Cancer Center, University of Iowa, Iowa City, IA, USA); David G. Mohler MD (Stanford University, Redwood City, CA, USA); Michael P. Mott MD (Henry Ford Health System, Detroit, MI, USA); Michael E. Mulligan MD (University of Maryland, Baltimore, MD, USA); Miguel Murcia Hernández MD, PhD (Hospital Pablo Tobón Uribe, Universidad Pontificia Bolivariana, Medellín, Colombia); Jorge Navia MD (deceased; Centro Medico Imbanaco, Universidad del Valle, Cali, Colombia); Lukas Nystrom MD (The Cleveland Clinic, Cleveland, OH, USA); Eduardo J. Ortiz-Cruz MD, PhD (Hospital Universitario La Paz, Universidad Autónoma de Madrid, Madrid, Spain); Michael Parry MBChB, MD (Royal Orthopaedic Hospital NHS Foundation Trust, Birmingham, Birmingham, UK); Joshua Patt MD, MPH (Carolinas Medical Center, Charlotte, NC, USA); Terrance D. Peabody MD (Northwestern Memorial Hospital, Northwestern University, Chicago, IL, USA); Daniel E. Prince MD, MPH (Memorial Sloan Kettering Cancer Center, Cornell University, New York, NY, USA); Ajay Puri MS (Tata Memorial Centre, Homi Bhabha National Institute, Mumbai, India); R. Lor Randall MD (Huntsman Cancer Institute, University of Utah, Salt Lake City, UT, USA); Pietro Ruggieri MD, PhD (Università di Padova, Padova, Italy); Chigusa Sawamura MD, MPH (Tokyo Medical and Dental University, Tokyo, Japan); Amir Sternheim MD (Tel Aviv Sourasky Medical Center, Tel Aviv University, Tel Aviv, Israel); Richard Terek MD (Providence VA Medical Center, Brown University, Providence, RI, USA); Joachim Thorkildsen MD, PhD (Oslo University Hospital - Radiumhospitalet, Oslo, Norway); Steven Thorpe MD (UC Davis Comprehensive Cancer Center, University of California, Davis, Sacramento, CA, USA); Robert Turcotte MD (McGill University Health Centre, McGill University, Montreal, Quebec, Canada); Wakenda Tyler MD, MPH (Columbia University Medical Center, Columbia University, New York, NY, USA); Roberto Vélez MD, PhD (Hospital Universitario Vall d’Hebron, Universidad Autònoma de Barcelona, Barcelona, Spain); Matthew Wallace MD, MBA (MedStar Cancer Institute, Baltimore, MD, USA); Ronald P. Williams MD (US Oncology Network–Texas Oncology, Austin, TX, USA); Rosanna Wustrack MD (University of California, San Francisco, San Francisco, CA, USA); and Juan Pablo Zumárraga MD, MSc (Hospital de Clínicas de São Paulo, Universidade de São Paulo, São Paulo, Brazil).
Footnotes
The institution of the authors has received, during the study period, funding from a McMaster Surgical Associates Innovation Grant (MG).
All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research ® editors and board members are on file with the publication and can be viewed on request.
Clinical Orthopaedics and Related Research ® neither advocates nor endorses the use of any treatment, drug, or device. Readers are encouraged to always seek additional information, including FDA-approval status, of any drug or device prior to clinical use.
Each author certifies that his or her institution approved or waived approval for the human protocol for this investigation and that all investigations were conducted in conformity with ethical principles of research.
This work was performed at McMaster University, Hamilton, Ontario, Canada.
Electronic supplementary material
The online version of this article (doi:10.1007/s11999-017-5482-7) contains supplementary material, which is available to authorized users.
A comment to this article is available at https://doi.org/10.1007/s11999-017-5501-8.
References
- 1.American Academy of Orthopaedic Surgeons. Metastatic Bone Disease. Available at: http://orthoinfo.aaos.org/topic.cfm?topic=A00093. Accessed May 12, 2017.
- 2.Bagi CM. Targeting of therapeutic agents to bone to treat metastatic cancer. Adv Drug Del Rev. 2005;57:995–1010. doi: 10.1016/j.addr.2004.12.014. [DOI] [PubMed] [Google Scholar]
- 3.Baker J, Lovell K, Harris N. How expert are the experts? An exploration of the concept of ‘expert’ within Delphi panel techniques. Nurse Res. 2006;14:59–70. doi: 10.7748/nr2006.10.14.1.59.c6010. [DOI] [PubMed] [Google Scholar]
- 4.Coppin C, Kollmannsberger C, Le L, Porzsolt F, Wilt TJ. Targeted therapy for advanced renal cell cancer (RCC): a Cochrane systematic review of published randomised trials. BJU Int. 2011;108:1556–1563. doi: 10.1111/j.1464-410X.2011.10629.x. [DOI] [PubMed] [Google Scholar]
- 5.Evaniew N, Nuttall J, Farrokhyar F, Bhandari M, Ghert M. What are the levels of evidence on which we base decisions for surgical management of lower extremity bone tumors? Clin Orthop Relat Res. 2014;471:2017–2027. doi: 10.1007/s11999-013-3311-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ghert M, Deheshi B, Holt G, Randall RL, Ferguson P, Wunder J, Turcotte R, Werier J, Clarkson P, Damron T, Benevenia J, Anderson M, Gebhardt M, Isler M, Mottard S, Healey J, Evaniew N, Racano A, Sprague S, Swinton M, Bryant D, Thabane L, Guyatt G, Bhandari M; PARITY Investigators. Prophylactic antibiotic regimens in tumour surgery (PARITY): protocol for a multicentre randomised controlled study. BMJ Open. 2012;2:e002197. [DOI] [PMC free article] [PubMed]
- 7.Goel A, Christy ME, Virgo KS, Kraybill WG, Johnson FE. Costs of follow-up after potentially curative treatment for extremity soft-tissue sarcoma. Int J Oncol. 2004;25:429–435. [PubMed] [Google Scholar]
- 8.Greatorex J, Dexter T. An accessible analytical approach for investigating what happens between the rounds of a Delphi study. J Adv Nurs. 2000;32:1016–1024. [PubMed] [Google Scholar]
- 9.Honeyman JN, Theilen TM, Knowles MA, McGlynn MM, Hameed M, Meyers P, Crago AM, La Quaglia MP. Desmoid fibromatosis in children and adolescents: a conservative approach to management. J Pediatr Surg. 2013;48:62–66. doi: 10.1016/j.jpedsurg.2012.10.017. [DOI] [PubMed] [Google Scholar]
- 10.Hopkins RB, Goeree R, Longo CJ. Estimating the national wage loss from cancer in Canada. Curr Oncol. 2010;17:40–49. doi: 10.3747/co.v17i2.477. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Hoste AEJ, Bagshaw SM, Bellomo R, Cely CM, Colman R, Cruz DN, Edipidis K, Forni LG, Gomersall CD, Govil D, Honoré PM, Joannes-Boyau O, Joannidis M, Korhonen AM, Lavrentieva A, Mehta RL, Palevsy P, Roessler E, Ronco C, Uchino S, Vazquez JA, Andrade EV, Webb S, Kellum JA. Epidemiology of acute kidney injury in critically ill patients: the multinational AKI-EPI study. Intensive Care Med. 2015;41:1411–1423. doi: 10.1007/s00134-015-3934-7. [DOI] [PubMed] [Google Scholar]
- 12.Huang K, Wang CM, Chen JG, Du CY, Zhou Y, Shi YQ, Fu H. Prognostic factors influencing event-free survival and treatments in desmoid-type fibromatosis: analysis from a large institution. Am J Surg. 2014;207:847–854. doi: 10.1016/j.amjsurg.2013.08.007. [DOI] [PubMed] [Google Scholar]
- 13.Jansen-Landheer MLEA, Krijnen P, Oostindiër MJ, Kloosterman-Boele WM, Noordijk EM, Nooij MA, Steup WH, Taminiau AHM, Vree R, Hogendoorn PCW, Tollenaar RAEM, Gelderblom H. Improved diagnosis and treatment of soft tissue sarcoma patients after implementation of national guidelines: a population-based study. Eur J Surg Oncol. 2009;35:1326–1332. doi: 10.1016/j.ejso.2009.05.002. [DOI] [PubMed] [Google Scholar]
- 14.Jeys LM, Kulkarni A, Grimer RJ, Carter SR, Tillman RM, Abudu A. Endoprosthetic reconstruction for the treatment of musculoskeletal tumors of the appendicular skeleton and pelvis. J Bone Joint Surg Am. 2008;90:1265–1271. doi: 10.2106/JBJS.F.01324. [DOI] [PubMed] [Google Scholar]
- 15.Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311:376–380. doi: 10.1136/bmj.311.7001.376. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Kashani K, Al-Khafaji A, Ardiles T, Artigas A, Bagshaw SM, Bell M, Bihorac A, Birkhahn R, Cely CM, Chawla LS, Davison DL, Feldkamp T, Forni LG, Ng Gong M, Gunnerson KJ, Haase M, Hackett J, Honore PM, Hoste EAJ, Joannes-Boyau O, Joannidis M, Kim P, Koyner JL, Laskowitz DT, Lissauer ME, Marx G, McCullough PA, Mullaney S, Ostermann M, Rimmelé T, Shapiro NI, Shaw AD, Shi J, Sprague AM, Vincent JL, Vinsonneau C, Wagner L, Walker MG, Wilkerson RG, Zacharowski K, Kellum JA. Discovery and validation of cell cycle arrest biomarkers in human acute kidney injury. Crit Care. 2013;17:R25. doi: 10.1186/cc12503. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Kellum JA, Mehta RL, Levin A, Molitoris BA, Warnock DG, Shah SV, Joannidis M. Ronco C; Acute Kidney Injury Network (AKIN). Development of a clinical research agenda for acute kidney injury using an international, interdisciplinary, three-step modified Delphi process. Clin J Am Soc Nephrol. 2008;3:887–894. doi: 10.2215/CJN.04891107. [DOI] [PubMed] [Google Scholar]
- 18.Leopold SS. Editor’s Spotlight/Take Five: What are the levels of evidence on which we base decisions for surgical management of lower extremity bone tumors? Clin Orthop Relat Res. 2014;472:3–7. doi: 10.1007/s11999-013-3332-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Longo CJ, Deber R, Fitch M, Williams AP, D’Souza D. An examination of cancer patients’ monthly ‘out-of-pocket’ costs in Ontario, Canada. Eur J Cancer Care (Engl). 2007;16:500–507. doi: 10.1111/j.1365-2354.2007.00783.x. [DOI] [PubMed] [Google Scholar]
- 20.McIntyre S, Novak I, Cusick A. Consensus research priorities for cerebral palsy: a Delphi survey of consumers, researchers, and clinicians. Dev Med Child Neurol. 2010;52:270–275. doi: 10.1111/j.1469-8749.2009.03358.x. [DOI] [PubMed] [Google Scholar]
- 21.Mehta RL, Kallum JA, Shah SV, Molitoris BA, Ronco C, Warnock DG. Levin A; Acute Kidney Injury Network. Acute Kidney Injury Network: report of an initiative to improve outcomes in acute kidney injury. Crit Care. 2007;11:R31. doi: 10.1186/cc5713. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Nipp RD, Zullig LL, Samsa G, Peppercorn JM, Schraq D, Taylor DH, Jr, Abernethy AP, Zafar SY. Identifying cancer patients who alter care or lifestyle due to treatment-related financial distress. Psychooncology. 2016;25:719–725. doi: 10.1002/pon.3911. [DOI] [PubMed] [Google Scholar]
- 23.Ota S, Cron RQ, Schanberg LE, O’Neil K, Mellins ED, Fuhlbrigge RC, Feldman BM. Research priorities in pediatric rheumatology: the Childhood Arthritis and Rheumatology Research Alliance (CARRA) consensus. Pediatr Rheumatol. 2008;6:5. doi: 10.1186/1546-0096-6-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Ottaviani G, Jaffe N. The epidemiology of osteosarcoma. Cancer Treat Res. 2009;152:3–13. doi: 10.1007/978-1-4419-0284-9_1. [DOI] [PubMed] [Google Scholar]
- 25.Prowle JR, Liu YL, Licari E, Bagshaw SM, Egi M, Haase M, Haase-Fielitz A, Kellum JA, Cruz D, Ronco C, Tsutsui K, Uchino S, Bellomo R. Oliguria as predictive biomarker of acute kidney injury in critically ill patients. Crit Care. 2011;15:R172. doi: 10.1186/cc10318. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Puri A, Gulia A, Hawalder R, Ranganathan P, Badwe RA. Does intensity of surveillance affect survival after surgery for sarcomas? Results of a randomized noninferiority trial. Clin Orthop Relat Res. 2014;472:1568–1575. doi: 10.1007/s11999-013-3385-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Renard AJ, Veth RP, Schreuder HWB, van Loon CJ, Koops HS, van Horn JR. Function and complications after ablative and limb-salvage therapy in lower extremity sarcoma of bone. J Surg Oncol. 2000;73:198–205. doi: 10.1002/(SICI)1096-9098(200004)73:4<198::AID-JSO3>3.0.CO;2-X. [DOI] [PubMed] [Google Scholar]
- 28.Rendon JS, Swinton M, Bernthal N, Boffano M, Damron T, Evaniew N, Ferguson P, Galli Serra M, Hettwer W, McKay P, Miller B, Nystrom L, Parizzia W, Schneider P, Spiguel A, Vélez R, Weiss K, Zumárraga J, Ghert M. Barriers and facilitators experienced in collaborative research in orthopaedic oncology. Bone Joint Res. 2017;6:307–314. doi: 10.1302/2046-3758.65.BJR-2016-0192.R1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Schneider P, Evaniew N, Rendon JS, McKay P, Randall RL, Turcotte R, Vélez R, Bhandari M, Ghert M. Moving forward through consensus: protocol for a modified Delphi approach to determine the top research priorities in the field of orthopaedic oncology. BMJ Open. 2016;6:e011780. doi: 10.1136/bmjopen-2016-011780. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Shehadeh A, Noveau J, Malawer M, Henshaw R. Late complications and survival of endoprosthetic reconstruction after resection of bone tumors. Clin Orthop Relat Res. 2010;468:2885–2895. doi: 10.1007/s11999-010-1454-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Skulmoski GJ, Hartman FT, Krahn J. The Delphi methods for graduate research. J Inf Technol Educ. 2007;6:1–21. [Google Scholar]
- 32.Slamon DJ, Leyland-Jones B, Shak S, Fuchs H, Paton V, Bajamonde A, Fleming T, Eiermann W, Wolter J, Pegram M, Baselga J, Norton L. Use of chemotherapy plus a monoclonal antibody against HER2 for metastatic breast cancer that overexpresses HER2. N Engl J Med. 2001;344:783–792. doi: 10.1056/NEJM200103153441101. [DOI] [PubMed] [Google Scholar]
- 33.Smith AJ, Dieppe P, Vernon K, Porter M. Blom AW; on behalf of the National Joint Registry of England and Wales. Failure rates of stemmed metal-on-metal hip replacements: analysis of data from the National Joint Registry of England and Wales. Lancet. 2012;379:1199–1204. doi: 10.1016/S0140-6736(12)60353-5. [DOI] [PubMed] [Google Scholar]
- 34.Sung NS, Crowley WF, Genel M, Salber P, Sandy L, Sherwood LM, Johnson SB, Catanese V, Tilson H, Getz K, Larson EL, Scheinberg D, Reece EA, Slavkin H, Dobs A, Grebb J, Martinez RA, Korn A, Rimoin D. Central challenges facing the national clinical research enterprise. JAMA. 2003;289:1278–1287. doi: 10.1001/jama.289.10.1278. [DOI] [PubMed] [Google Scholar]
- 35.Thompson CA, Charlson ME, Schenkein E, Wells MT, Furman RR, Elstrom R, Ruan J, Martin P, Leonard JP. Surveillance CT scans are a source of anxiety and fear of recurrence in long-term lymphoma survivors. Ann Oncol. 2010;21:2262–2266. doi: 10.1093/annonc/mdq215. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Yu HH, Tsai YY, Hoffe SE. Overview of diagnosis and management of metastatic disease to bone. Cancer Control. 2012;19:84–91. doi: 10.1177/107327481201900202. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.