Issues and Challenges With Integrating Patient-Reported Outcomes in Clinical Trials Supported by the National Cancer Institute–Sponsored Clinical Trials Networks

Abstract

Purpose

The objective of this report is to provide a historical overview of and the issues and challenges inherent in the incorporation of patient-reported outcomes (PROs) into multinational cancer clinical trials in the cancer cooperative groups.

Methods

An online survey of 12 cancer cooperative groups from the United States, Canada, and Europe was conducted between June and August of 2006. Each of the cooperative groups designated one respondent, who was a member of one of the PRO committees within the cooperative group.

Results

There was a 100% response rate, and all of the cancer clinical trial cooperative groups reported conducting PRO research. PRO research has been conducted in the cancer cooperative groups for an average of 15 years (range, 6 to 30 years), and all groups had multidisciplinary committees focused on the design of PRO end points and the choice of appropriate PRO measures for cancer clinical trials. The cooperative groups reported that 5% to 50% of cancer treatment trials and an estimated 50% to 75% of cancer control trials contained PRO primary and secondary end points. There was considerable heterogeneity among the cooperative groups with respect to the formal and informal policies and procedures or cooperative group culture towards PROs, investigator training/mentorship, and resource availability for the measurement and conduct of PRO research within the individual cooperatives.

Conclusion

The challenges faced by the cooperative groups to the incorporation of PROs into cancer clinical trials are varied. Some common opportunities for improvement include the adoption of standardized training/mentorship mechanisms for investigators for the conduct of PRO assessments and data collection and the development of minimal criteria for PRO measure acceptability. A positive cultural shift has occurred in most of the cooperative groups related to the incorporation of PROs in clinical trials; however, financial and other resource barriers remain and need to be addressed.

INTRODUCTION

In 1955, the first US clinical trials cancer cooperative group was funded by the National Cancer Institute (NCI). Currently, there are 10 national US-sponsored cancer cooperative groups and more throughout the world. The primary objective, at least initially, of these groups was to conduct randomized clinical trials of therapies to improve the survival and decrease morbidity of patients diagnosed with cancer.

It took another 30 years before additional end points, such as the measurement of the impact symptoms had on quality of life (QOL), were a recognized goal of cooperative group clinical trials. Prompted by the 1985 US Food and Drug Administration (FDA) changes in approval requirements for anticancer drugs, which included favorable effects on survival and/or QOL,¹ the NCI Cancer Therapy Evaluation Program (CTEP) revised its mission statement in 1988 stating, “Research aimed at improving survival and QOL for persons with cancer is of the highest priority.”² This was seen almost as a mandate by most of the cooperative groups, and the success of this addition to the CTEP mission can be seen in the literature where inventories of health-related QOL clinical trials have been published to enhance awareness and subsequent recruitment, encourage collaboration, and provide a reference for common instruments used to measure health-related QOL.^3-6 Again prompted by recent FDA activity in the form of the “Guidance for Industry: Patient-Reported Outcome Measures,”⁷ the cancer clinical trials cooperative groups are taking a fresh look at the measurement of symptom, behavioral, and QOL outcomes under the more encompassing term of patient-reported outcomes (PROs).

The primary objectives of this article are to provide a historical perspective of the incorporation of PROs into multinational cancer clinical trials cooperative group studies and to provide an overview of the issues and challenges faced by cooperative groups to integrating PRO measures in cancer clinical trials. This is the first comprehensive survey of the cancer cooperative groups that inventories their infrastructure, capabilities, policies, and procedures for using PROs in clinical trials. We report here on the heterogeneity among cooperative groups in terms of formal and informal policies and procedures as well as resource availability for the measurement and conduct of PRO research. For the purpose of this discussion, we have adopted the FDA definition of a PRO as “a measurement of any aspect of a patient's health status that comes directly from the patient (ie, without interpretation of the patient's responses by a physician or anyone else).”⁷

METHODS

To assess the use, process, and barriers to the conduct of PRO research, a survey of 12 cancer clinical trials cooperative groups from the United States, Canada, and Europe was conducted between June and August of 2006. A 75-item questionnaire was designed using the Internet design software, SurveyMonkey (SurveyMonkey, Portland, OR). The survey was designed by a committee representing the cooperative group leaders in PRO research, a subset of the group that completed the survey.

Questions were designed to assess the process, policies, and procedures for the conduct of PRO research in both cancer treatment and cancer control trials. Issues related to criteria for inclusion of PRO end points and the support and resources available for this line of inquiry were assessed. The culture within each group that facilitates or hinders PRO research was also evaluated.

An electronic link to the survey was emailed to the chairperson of the PRO committees at each of the 12 participating cooperative groups. Only one response was permitted per group, although the responses were a collaborative effort among PRO committee members in several of the groups. Respondents completed the survey online, and there was a 100% response rate to the survey. The survey included both categoric and open-ended items. Frequency distributions were reported for all categoric items.

RESULTS

Names and acronyms of the 12 clinical trials cooperative groups that participated in the survey are listed in Table 1, along with the date the group was formed and the date the first committee with a focus on PROs was formed. In addition to the 10 US cancer clinical trials cooperative groups asked to participate in the survey, the European Organisation for Research and Treatment of Cancer (EORTC) and the National Cancer Institute of Canada cooperative groups were asked to participate because of their long-standing and close collaboration with the US cooperative groups. On average, the cooperative groups have been conducting PRO research for approximately 15 years, with a range of 6 to 30 years (Table 1).

Table 1.

Length of Experience With PROs in Cancer Clinical Trials Cooperative Groups

Cancer Clinical Trials Cooperative Group	Acronym	Year of Group's Inception	Year of PRO Committee Inception	No. of Years the Group Has Been Conducting PRO Research
American College of Surgeons Oncology Group	ACOSOG	1998	1998	8
American College of Radiology Imaging Network	ACRIN	1999	1999	7
Cancer and Leukemia Group B	CALGB	1956	1976	30
Children's Oncology Group*	COG	2000	2000	6
Eastern Cooperative Oncology Group	ECOG	1955	1989	17
European Organisation for Research and Treatment of Cancer	EORTC	1962	1981	25
Gynecologic Oncology Group	GOG	1970	1991	15
North Central Clinical Trials Group	NCCTG	1981	1999	7
National Cancer Institute of Canada	NCIC	1980	1986	20
National Surgical Adjuvant Breast and Bowel Project	NSABP	1957	1992	14
Radiation Therapy Oncology Group	RTOG	1968	1989	17
Southwest Oncology Group	SWOG	1956	1989	17

Abbreviation: PRO, patient-related outcomes.

^*The Children's Oncology Group was formed through a merger of the Children's Cancer Study Group, Pediatric Oncology Group, Intergroup Rhabdomyosarcoma Study Group, and the National Wilms' Tumor Study Group. Although some of the legacy groups had been in existence for many years, we report here only on the currently active Children's Oncology Group.

Survey respondents chaired a variety of committees that were identified by the respondents as the committees primarily responsible for the oversight of PRO research in their respective groups (Table 2). These committees varied in scope, from focusing exclusively on QOL to covering domains of health economics, behavioral research, symptom management, and cancer control. The 12 cooperative groups also identified a total of 36 individual PRO-related committees with membership sizes ranging from one to five members to more than 25 members. The focus of the standing PRO committees varied across the cooperative groups with seven (60%) of 12 cooperative groups reporting a primary focus on cancer control, six (50%) of 12 reporting a primary focus on nursing, and five (42%) of 12 reporting a primary focus on QOL.

Table 2.

Committees Chaired by the Survey Respondents Within Their Respective Cooperative Groups

	Respondents
Committee	No.	%
Quality-of-Life Committee	7	58
Outcomes/Health Services/Economics	2	17
Behavioral and Health Outcomes	2	17
Nursing Research Committee	1	8

Committee Membership

Although the names of the committees that focus on or have primary responsibility for the conduct of PRO research in their respective cooperative groups varied, committee membership for all of the groups was reported as highly multidisciplinary (Fig 1). All of the cooperative groups reported that statisticians were part of their respective PRO committees. Psychologists were the next most highly represented and were on almost 92% of the PRO committees (11 of 12 committees). Approximately 83% of the committees (10 of 12 committees) that conduct PRO research in the cooperative groups reported having medical oncologists, nurses, and patient advocates as committee members, whereas only a quarter of the committees reported that social workers were members.

Fig 1.

Specialist participation on multidisciplinary patient-related outcome committees. RNs, nurses; PRO, patient-related outcome. (*) Other category includes radiologists, gynecologic oncologists, and clinical research associates (data field manager).

The role of most of the disciplines on these committees is the design of PRO end points and choice of appropriate PRO measures. Clinical research associates contribute to the discussion of logistical issues related to PRO measures, and patient advocates are primarily used (in nine of 10 groups that responded to this question) to promote clinical trials with PRO end points. The committees that focus on PROs all make use of liaisons to other site and modality committees within their respective groups to facilitate adding PRO end points to developing trials as appropriate. Nine (75%) of 12 groups report that the liaisons are always or usually incorporated as full members into the committees to which they are assigned.

Trials With PRO Assessments in the Cooperative Groups

Cooperative group respondents were asked to estimate the number of open treatment trials that include PRO primary or secondary end points (Table 3). The percentage of treatment trials with PRO end points ranged from 5% to 50% depending on the group. There were less data available on cancer control or prevention trials containing PRO end points; however, of the groups conducting cancer control research, respondents estimated that a higher proportion (50% to 75%) of these studies contain PRO primary or secondary end points. Sixty-six percent of the respondents (eight of 12 respondents) reported that the percentage of treatment and cancer control trials with PRO measures was either stable or increasing over the last 5 years; one third of the groups reported that this percentage was decreasing.

Table 3.

Cancer Clinical Trials Cooperative Group Open Treatment Trials With PRO Primary or Secondary End Points

	All Open Treatment (CTEP) Trials With PRO End Points
Group	No.*	%
ACOSOG	2/10	20
ACRIN†	5/11	45
CALGB	3/76	4
COG	5/101	5
ECOG	8/113	7
EORTC	32/72	26-50
GOG	5/77	7
NCCTG	18/86	21
NCIC	12/25	48
NSABP	5/15	33
RTOG	18/45	40
SWOG	6/133	5

Abbreviations: PRO, patient-related outcome; CTEP, Cancer Therapy Evaluation Program; ACOSOG, American College of Surgeons Oncology Group; ACRIN, American College of Radiology Imaging Network; CALGB, Cancer and Leukemia Group B; COG, Children's Oncology Group; ECOG, Eastern Cooperative Oncology Group; EORTC, European Organisation for Research and Treatment of Cancer; GOG, Gynecologic Oncology Group; NCCTG, North Central Clinical Trials Group; NCIC, National Cancer Institute of Canada; NSABP, National Surgical Adjuvant Breast and Bowel Project; RTOG, Radiation Therapy Oncology Group; SWOG, Southwest Oncology Group.

^*Numerator is from the cooperative group respondent, and denominator is from National Cancer Institute CTEP Web site (http://ctep.cancer.gov) as of September 2006.

^†ACRIN open trials refer to imaging trials, not necessarily treatment trials.

Use of PROs and Common PRO Measures

Most of the cancer clinical trials cooperative groups (11 of 12 groups) reported that PRO measures were primarily used to assess outcome comparisons among treatment arms including QOL outcomes, symptom assessments, and survivorship issues. Seven of the groups (58%) reported using PRO measures for behavioral assessments, and one third of the groups reported using PROs to assess translational research questions (eg, correlating PROs with biomarkers) and to assess complementary and alternative research questions.

The three most common PRO measures used are the EORTC Quality of Life Questionnaire C30⁸ (EORTC QLQ-C30) (seven groups, 58%), the Functional Assessment of Cancer Therapy (FACT) or the Functional Assessment of Current Illness Therapy⁹ (six groups, 50%), and the Medical Outcomes Short Form Health Survey¹⁰ (five groups, 42%). Fifty-eight percent of the respondents (seven of 12 respondents) reported that their cooperative group occasionally or sometimes conducted studies in new PRO development. Also, two thirds of the cooperative groups (eight of 12 groups) reported that PRO validation was included in studies with PRO measures. All of the respondents used PROs in phase III treatment trials. In contrast, fewer cooperative groups used PRO end points in phase I (one group, 8%), phase II (six groups, 50%), and cancer control (nine groups, 67%) trials.

Policies and Procedures for Conducting PRO Research in Cooperative Group Trials

The groups were queried about formal and informal training for investigators and clinical research associates in the conduct of PRO research in their respective groups. Eighty-three percent of the groups (10 of 12 groups) reported a formal training process for clinical research associates, but only two of the respondents (17%) indicated that PRO investigators had some formal training, and only four(33%) reported having formal PRO mentorship for investigators. Training methods varied across the participating cooperative groups and included the following: periodic lectures and discussions at group meetings (10 groups, 83%), Web-based materials (four groups, 33%), video/digital versatile discs (DVDs; three groups, 25%), orientation lectures (two groups, 17%), and compact discs (CDs; one group, 8%).

Approximately 60% of the respondents (n = 7) reported that they used written policies and procedures related to the incorporation of PRO end points into cooperative group clinical trials. Of those with written polices and procedures, less than half (n = 4) of the respondents indicated that they believed there was efficient or frequent utilization of the PRO policies and procedures. When asked about the recommended frequency of introducing/updating PRO policies in cancer clinical trial cooperative groups, five groups (42%) suggested annually, three (25%) suggested periodically, and three (25%) recommended only at orientation. In terms of written procedures for the conduct of PRO research (ie, a template for how PROs are collected, what to do if language translations are not available, what to do if the patient misses a study visit when a PRO measure is due, what to do if a patient requires assistance in filling out a PRO, and so on), 11 respondents (92%) reported that study-specific PRO information was written into individual protocols, and nine respondents (75%) had general standardized study procedures for the conduct of PRO measurement.

When asked how each group determined which studies would include PRO end points, 11 of the participating cooperative groups (92%) had a review process in place for PRO end points, with nine (75%) reviewing all protocols and two (17%) reviewing select protocols. Only one cooperative group indicated that there was no review process and that the study chair or protocol team amended each protocol as they determined necessary.

We attempted to determine at what point PRO investigators were included in the process of protocol development once it was determined that a protocol should have a PRO end point. This was assessed as one indicator of the culture or how the cooperative groups value the incorporation of PRO end points relative to the incorporation of other end points like survival, toxicity, or biomarkers. Half of the respondents reported that the investigators were included when PRO secondary end points were in the development process from or almost at the start of the study design, and the remaining half reported that the investigators were included in time for NCI (or appropriate international approval agency) submissions. Three groups (25%) also indicated that secondary PRO end points were added late in the study design. Most of the sites (92%) reported that the timing of the inclusion of the investigators in the PRO development process had either improved somewhat or greatly. Only in one cooperative group was the process of timely engagement of PRO investigators reported to have gotten worse over time.

Criteria for PRO Measure Inclusion in Clinical Trials

Although eight of the groups (67%) reported the use of conceptual frameworks to guide the incorporation of study-specific PRO end points, only one group reported on an overarching model for the incorporation of PROs into cooperative group clinical trials.¹¹ Two thirds of the respondents (eight of 12 respondents) used specific psychometric criteria for validity and reliability when selecting PRO measures.

Facilitators to PRO Inclusion in Cooperative Group Clinical Trials

Overall, respondents perceived positive support from the top leadership of their cooperative group for PRO research. Two thirds of the respondents (eight of 12 respondents) reported receiving good to excellent support, and one third (four of 12 respondents) indicated that they received moderate support for PRO research from their cooperative group chair.

Qualitative data from the survey in response to open-ended questions eliciting suggestions for facilitators of PRO research in the cancer clinical trials cooperative groups included the following (statements were consolidated):

• Communication between the PRO liaison and their assigned disease site committee is key. PRO committees have to actively interact with other committees within the cooperative groups.

• There is a belief that PRO end points are no different from other end points in terms of resources/funding.

• Compelling results from several phase III trials on the importance of PROs have provided more support from within the cooperative groups.

• Educate investigators on the value of PROs (perhaps standardize education methods across cooperative groups).

• Support from clinicians (eg, community oncologists) and having the physician study chair actively support PRO assessments at the presentation of the trials would be helpful.

• Additional funding is needed to pay for the use and collection of PRO measures.

Barriers to PRO Inclusion in Cooperative Group Clinical Trials

Although there was significant support cited for PRO research from the group chairs, there was wide variability in statistical support for PRO research among the groups. Four of the groups (33%) reported excellent support, three (25%) reported good support, four (33%) reported moderate support, and one (8%) reported very little statistical support. In cases where PRO end points were not included in protocols when the committee thought they were appropriate, participants were asked whether lack of statistical support was a reason. Six (50%) of 12 respondents answered yes. Additionally, seven (58%) of 12 respondents reported having some difficulty hiring statisticians experienced in PRO analysis.

Nine of the cooperative groups surveyed (75%) reported that they had either occasionally or frequently encountered financial barriers to the inclusion of PROs as study end points; and five of these respondents reported that their committee had been asked to limit the number of protocols with PRO end points. When asked whether, in cases where PRO end points were not included in protocols when the committee thought they were appropriate, financial barriers (ie, funding for PRO measures, data collection and financial issues other than lack of statistical resources) were implicated, nine respondents (75%) reported that financial resources were a frequent or occasional barrier. Additional barriers included the occasional (n = 5, 42%) or frequent (n = 1, 8%) use of a separate informed consent for the participation in PRO end points of a cooperative group protocol and lack of funding for the purchase of copyrighted PRO measures.

Qualitative data from the survey in response to barriers to the incorporation of PRO research in the cancer clinical trials cooperative groups included:

• PROs are ancillary to survival end points.

• There is little statistical support during PRO development.

• PROs were difficult to analyze statistically as a result of complications arising from missing data.

• Oncologists are frustrated with the lack of clinical meaning of results from PROs.

• Clinical site commitment (eg, resources) to quality PRO data collection is lacking.

• There is lack of enforcement for noncompliance with PRO data management practices.

• Investigator time and lack of funding are rate-limiting steps in PRO development.

Cooperative Group Culture and PRO Research

To assess the cooperative group culture related to the incorporation of PROs into clinical trials, participants were asked about how well they perceived PRO end points are valued compared with mortality, morbidity, and biomarker end points. The majority of respondents (n = 8, 66%) reported that PRO end points are moderately valued (the groups permit PRO end points, but it is somewhat difficult to get the resources needed). Three respondents (25%) reported that PROs are well valued (still not on par with other more traditional outcomes but well accepted and supported). Only one respondent chose the response option that indicated that PROs were very well valued (on par with other outcomes and well supported with resources).

From the qualitative portion of the survey soliciting additional comments, the following two quotes highlight the positive and negative aspects to the cooperative culture surrounding PROs.

• “By and large, most clinicians support the value of PROs. There are some who do not. However, over time, we have observed positive changes.”

• “First was the basic belief that these PRO end points were ancillary to our real (funded) job of obtaining survival end points for [Cancer Treatment Evaluation Program] CTEP. Secondly, it generally has been felt that these end points were too difficult to analyze statistically (ie, doing power analyses, worrying about our ability to deal with missing QOL data and so on). There was never a great deal of enthusiasm for them from the statistical sections and as a consequence there is very little statistical support for PRO end points as study concepts were being developed.”

DISCUSSION

All of the national and international cancer clinical trial cooperative groups surveyed include PROs in clinical trials with wide variation in the number and types of trials in which PROs are included. The choice of PRO instrument (eg, EORTC QLQ-C30 or FACT) used by specific groups seems largely to be based on geography and tradition. Specifically, the majority of the US-based groups seem to have a preference for FACT, whereas the National Cancer Institute of Canada and the EORTC have historically opted for EORTC QLQ-C30 in the majority of their trials.

A concern for financial and/or resource barriers to PRO research seems significant. Half of the respondents reported lack of funding for the purchase of copyrighted PRO measures. Although some PRO instruments are available on a license-free basis for academic trials, this is not the case for all instruments. Statistical support seems to be the other major resource barrier, with almost half of the respondents reporting only moderate to poor support.

Overall, respondents thought that PRO end points were valued by the cooperative groups, although most indicated that PROs are not valued to the same extent as other outcomes including mortality, morbidity, and biomarkers. Although the clinical trials cooperative group culture seems to have improved somewhat over time, the continued reticent acceptance of the value of PROs is becoming more difficult to justify given the valuable data PROs have provided from their earliest incorporation in clinical trials. Important findings have been documented with PROs, from the now classic sarcoma trial debunking the long-standing assumption that any therapy, no matter how aggressive, is better than surgical limb amputation¹² to results from a number of more recent cancer clinical treatment trials showing survival to be positively correlated with PRO measures at the initiation of cancer treatment across disease types.^13-16 A recent review by Gotay et al (unpublished data) examined randomized clinical trials that assessed PROs, biomedical factors, and survival in cancer patients. These trials reflected more than 13,000 patients with a wide range of cancer diagnoses; in 34 of 37 reports, PROs were linked with survival, even when controlling for biomedical variables in multivariate analysis.¹⁷

In addition to providing comparative treatment data and prognostic information, there is growing evidence that PROs provide significantly more toxicity and symptom data than physician or Common Terminology Criteria of Adverse Events assessments.^18,19 For specific subjective symptoms such as sexual function, studies have shown that clinician assessments using the Common Terminology Criteria of Adverse Events (or earlier versions) are no better than chance in predicting PROs of the same symptoms.^20,21

This report highlights the existing infrastructure, policies, and procedures towards PROS at cooperative groups. Each cooperative group has had to adapt its existing infrastructure to accommodate the integration of PROs to clinical trials, and they have done so with varying degrees of success. It is the commitment, as evinced by the level of activity of the PRO committees within the cooperative groups, that drives the integration process. An example of this is the increasing visibility of the PRO committees within the cooperative groups. All of the survey respondents reported that their PRO committees send liaisons to disease site or other committees within their respective cooperative groups, and 75% of the PRO committee liaisons were full members to their assigned committees. The importance of these liaisons or champions of PROs on individual disease site or other committees cannot be overstated. They are integral to overseeing the incorporation of PRO end points into study protocols and to building more uniform support and infrastructure, which will in turn enhance the cooperative groups’ ability to support the growing demand for PROs in clinical trials.

The current survey assists in documenting the issues and challenges faced by cooperative groups to integrating PRO measures in cancer clinical trials. It also helps to identify opportunities for improving the incorporation of PRO end points and measures into cancer cooperative group clinical trials. For example, standardized training DVDs or CDs for the conduct of PRO assessments and data collection in cooperative group clinical trials could be developed jointly among the groups. Two thirds of the cooperative groups reported using specific psychometric criteria for validity and reliability when selecting PRO measures. These criteria could be shared among the groups to set minimal criteria for PRO measure acceptability. The FDA “Guidance for Industry: Patient-Reported Outcome Measures”⁷ may be of assistance in this by increasing the attention to PROs. There seems to be opportunity for improving PRO training for cooperative group statisticians. Economy of scale would suggest that the development and conduct of this training would be most efficient if shared among the cooperative groups. In addition, future discussions among the NCI, the cooperative groups, and the committees responsible for PRO end points are needed to establish the necessary resources required for PRO assessments in cancer clinical trials.

A positive cultural shift has occurred in most of the cooperative groups related to the incorporation of PROs in clinical trials. However, financial and other resource barriers remain and need to be addressed.