Prioritizing comparative effectiveness research for cancer diagnostics using a regional stakeholder approach

Abstract

Aims

This paper describes our process to engage regional stakeholders for prioritizing comparative effectiveness research (CER) in cancer diagnostics. We also describe a novel methodology for incorporating stakeholder data and input to inform the objectives of selected CER studies.

Materials & methods

As an integrated component to establishing the infrastructure for community-based CER on diagnostic technologies, we have assembled a regional stakeholder group composed of local payers, clinicians and state healthcare representatives to not only identify and prioritize CER topics most important to the western Washington State region, but also to inform the study design of selected research areas. A landscape analysis process combining literature searches, expert consultations and stakeholder discussions was used to identify possible CER topics in cancer diagnostics. Stakeholders prioritized the top topics using a modified Delphi/group-nominal method and a standardized evaluation criteria framework to determine a final selected CER study area. Implementation of the selected study was immediate due to a unique American Recovery and Reinvestment Act funding structure involving the same researchers and stakeholders in both the prioritization and execution phases of the project. Stakeholder engagement was enhanced after study selection via a rapid analysis of a subset of payers’ internal claims, coordinated by the research team, to obtain summary data of imaging patterns of use. Results of this preliminary analysis, which we termed an ‘internal analysis,’ were used to determine with the stakeholders the most important and feasible study objectives.

Results

Stakeholders identified PET and MRI in cancers including breast, lung, lymphoma and colorectal as top priorities. In an internal analysis of breast cancer imaging, summary data from three payers demonstrated utilization rates of advanced imaging increased between 2002 and 2009 in the study population, with a great deal of variability in use between different health plans. Assessing whether breast MRI affects treatment decisions was the top breast cancer study objective selected by the stakeholders. There were other high-priority research areas including whether MRI use improved survival that were not deemed feasible with the length of follow-up time following MRI adoption.

Conclusion

Continuous stakeholder engagement greatly enhanced their enthusiasm for the project. We believe CER implementation will be more successful when undertaken by regional stakeholders.

The Institute of Medicine (IOM) identified stakeholder involvement as a key element of the comparative effectiveness research (CER) effort stimulated by the American Recovery and Reinvestment Act (ARRA) of 2009 [101]. While stakeholder involvement in community-based participatory research has been advocated for many years to improve the relevance of clinical research [1], this is not the norm for clinical research. In fact, academic research has traditionally been initiated by investigators who select the research topic and then define and implement specific aims [2]. This approach is successful in generating research matched to the interests and expertise of the investigators; however, the results do not always correspond to the evidence gaps most important to clinical decision-makers or patients [3,4].

To address the disconnect between research designed by individual investigators and the evidence needs of decision-makers, there has been increased efforts to engage stakeholders in all aspects of CER, particularly during the initial phases of prioritization and selection of specific research topics [5]. In addition to improving research relevance, stakeholder buy-in can aid researchers with access to data, maximizing the utility of the limited resources that are devoted to research.

This article describes a process for engaging stakeholders in a regional panel to assist researchers in defining and prioritizing topics in cancer diagnostics in western Washington State. Many aspects of the methodologies for stakeholder engagement and prioritization described in this paper are readily generalizable to CER studies conducted elsewhere. Convening the stakeholder group was part of an ARRA-funded CER effort called Advancing Innovative Comparative Effectiveness Research in Cancer Diagnostics (ADVICE), the goal of which was to develop the infrastructure to perform rapid-response community-based CER and to conduct two proof-of-principle stakeholder-informed CER studies on cancer diagnostics.

Materials & methods

ADVICE project overview

In western Washington State, leaders in medicine, research and health insurance are embracing the idea that healthcare is regional. Referral patterns, the supply of healthcare professionals, employers and even healthcare payers are largely tied to specific localities [6,7]. Solutions to problems in the healthcare system should, therefore, reflect the interests, needs and priorities of regional stakeholders. ADVICE is a CER effort that was designed to combine a multi-institutional and multidisciplinary research team from the University of Washington (WA, USA), the Fred Hutchinson Cancer Research Center (WA, USA) and the Group Health Research Institute with community stakeholders to conduct stakeholder-informed research. The research team consists of experts in epidemiology, oncology, medical imaging, health economics and health policy. Stakeholders include representatives from the major fee-for-service health plans (Premera Blue Cross and Regence Blue Shield), publicly funded systems (Medicare, Medicaid, the Washington State Health Care Authority and the Puget Sound Veterans Affairs Health Care System), and an integrated healthcare provider (Group Health Cooperative). Together, the ADVICE network represents approximately 80% of insured patients in the western Washington region, allowing us to conduct generalizable population-based research.

We hypothesized that stakeholder-informed research would lead to more engagement from the payers in the research and lead to research that has real-life clinical relevance to payers because the regional stakeholders who are empowered to make changes within their systems are participating. To help curb the use of inappropriate imaging that may be contributing to this trend, most payers have begun using third party systems that require physicians to obtain prior authorization before an advanced imaging procedure is reimbursed. While these programs have done well in general, unfortunately the underlying algorithms do not do a good job of guidance in the setting of most cancers, as there is insufficient evidence or even consensus expert opinion on the right guidelines.

In 2010, with active stakeholder involvement, ADVICE began two observational studies in cancer imaging linking Surveillance, Epidemiology and End Results (SEER) cancer registry data to administrative claims data from the aforementioned health plans. The first study was defined broadly in the original grant proposal to be in the area of advanced imaging use in breast cancer; whereas the second research topic and cancer site were exclusively determined by stakeholders. In addition to defining and prioritizing topics, refining objectives and providing data, stakeholders will disseminate results and consider whether reimbursement and payment policies should change in response to our findings.

Stakeholder group recruitment & development

Three principles guided our stakeholder recruitment process:

• ▪Broad definition: while there are differing interpretations of the term ‘stakeholder,’ our definition was relatively broad and based on a the definition suggested by O’Haire in a recent Agency for Healthcare Research and Quality (AHRQ) report on stakeholder engagement: “Stakeholders are individuals or organizations who have a personal or professional interest in the topic” [8];
• ▪Regional participation: in light of the significant regional variation that exists in healthcare access, utilization, costs and outcomes [9–11], we felt that identification of priorities and implementation of CER findings to improve system efficiency and clinical care would be best addressed by regional stakeholders;
• ▪High level of involvement: while stakeholder involvement in CER can vary considerably (from ad hoc methods of convening stakeholders with limited participation [2,12,13] to formally defined groups with significant time commitment, including multiple meetings, one-on-one interviews, and contributions to manuscripts and reports [14,15]), we felt significant stakeholder involvement, which we defined as frequent face-to-face meetings and involvement in periodic webinars and email exchanges would increase research relevance, establish lasting relationships and ensure active dissemination and implementation.

As payer claims data were to play a major role in the methods used to evaluate imaging effectiveness, it was crucial to include appropriate representation in the stakeholder group. To supplement this core group, leadership of regional healthcare delivery systems and administrators and policy makers representing Washington State healthcare agencies were also contacted. For the initial round of topic prioritization, no patients or patient advocacy groups were included as voting members. The wide range of potential CER topics, imaging techniques and therapeutic areas made it extremely difficult to identify a manageable number of patient groups that would have broad enough knowledge to contribute impartially to stakeholder group discussions. We therefore decided to first focus on a stakeholder group without direct patient representation as a foundational step in demonstrating the feasibility of stakeholder-influenced research. However, a representative from a nonprofit organization, the Puget Sound Health Alliance [102], which represents patient, provider and payer interests, was included during the initial kick-off meeting to provide some consumer perspective.

An initial kick-off meeting was convened with the research team and approximately 20 potential stakeholders representing regional payers, administrators and policy makers discussed the project goals, assessed interest and obtained a list of potential research projects that were of high importance. In order to stimulate discussion, a pre-meeting packet was emailed describing the general goals and scope of the grant, along with an initial list of possible research topics to focus on initially for two proof-of-principle demonstration projects. Subsequent to the meeting, each organization was asked to identify a representative who was invited to attend a second meeting as a voting member of the ADVICE stakeholder group. All organizations agreed to designate a voting representative and as a result of this process, a group of eight voting stakeholder members were defined, including three collaborating payers, two clinicians with specialties in cancer and three state healthcare representatives. Stakeholder meetings were held approximately every 6 months, with regular email communications and webinars between meetings. Meeting attendance was limited to just the voting stakeholder group and research team to result in a discussion group of manageable size, large enough to benefit from group dynamics, yet small enough to allow natural discourse.

The regional stakeholder group was augmented by a nonvoting National Advisory Board with members representing the areas of research, regulatory issues, medical diagnostics industry and ethics. Whereas the charge of the regional stakeholder group was primarily to influence specific research direction and implementation for the project, the National Advisory Board provided higher level assistance, providing input on national imaging guidelines, device regulatory reform, advice on methodologies for diagnostic CER, and longer term project guidance regarding the science of current and future technologies in diagnostics. Figure 1 provides an overview of the overall structure of the ADVICE project.

Figure 1. Advancing Innovative Comparative Effectiveness Research in Cancer Diagnostics overview.

Stakeholder input at all phases of the project was key to creating a motivated regional stakeholder group with vested interest in the study results.

Topic selection

Topic selection was a two-phase process, summarized in Figure 2. In the first phase, a ‘landscape analysis’ process [16,17], wherein a broad range of potential topics are identified through literature review and expert consultation, was carried out iteratively between the stakeholders and the project team. This process has also been referred to as ‘horizon-scanning’ in the literature [18–20]. During the initial kick-off meeting, stakeholders were asked to brainstorm on topics of importance at their institutions related to medical imaging or lab-based tests for cancer diagnosis, follow-up and surveillance. The ADVICE project team then spent several months gathering background information on these and related topics via a broad scan of peer-reviewed and gray literature, and consultations with local clinical experts in medical imaging and lab-based diagnostics. Grey literature searches targeted abstracts reporting recent developments in the field and included meetings relevant to cancer imaging such as the Radiological Society of North America (RSNA), Society for Nuclear Medicine (SNM) and the American Society of Clinical Oncology (ASCO). The goal of the landscape analysis was not to perform a complete systematic review of the potential topics, but to gather information necessary to support an informed roundtable discussion during the next meeting of the voting members of the stakeholder group. Based on the high initial interest in advanced imaging, especially PET, local clinical experts in PET and MRI oncologic imaging were invited to the stakeholder roundtable. A main objective of the stakeholder roundtable was to identify five to ten well-defined topics that would be the subject of a subsequent prioritization process.

Figure 2. Comparative effectiveness research topic selection.

A two-phase approach was used for comparative effectiveness research topic selection. A ‘landscape analysis’ phase spanned approximately 2 months and was the focus of the first two stakeholder meetings. Literature searches, consultations with clinical experts and early roundtable discussions with the stakeholders were used to identify potential high-priority topics in cancer diagnostics. During the next 6 months, a second ‘prioritization phase’ was carried out to rate the relative importance of the top six topics identified by the landscape analysis. Detailed technical briefs were developed for the stakeholders using a standardized evaluation criteria framework (Appendix 1), and ratings were obtained through a web-based survey followed by an in-person stakeholder meeting discussion and rating session.

Another important objective of the stakeholder roundtable was developing agreement on a set of criteria to frame formal topic evaluation. The criteria were derived from recommendations in the Federal Coordinating Council for Comparative Effectiveness Research Report to the President and the Congress [103] and modeled on criteria used in the prioritization process of health technology assessment groups [21,104]. As seen in Appendix 1, the criteria were divided into three levels: Minimum Criteria that must be satisfied for the topic to be considered; Primary Criteria that should weigh heavily in the decision process; and Secondary Criteria that should also be considered during the evaluation. These criteria could be used for prioritization of almost any topics of interest. Once an initial list of high-priority topics had been identified, we began the prioritization phase, modeled on the mixed methods, modified Delphi/group-nominal method proposed by the Center for Medical Technology Policy (CMTP) group [105].

We developed concise, one-page topic summaries for each candidate research topic that included descriptions of how well the topic fit the evaluation criteria, as well as an importance rating for each criterion to guide the stakeholder’s overall interpretation. The rating used a three-point ‘low, medium, high’ scale, and was agreed upon by the ADVICE project team based on the reviewed literature and summary of expert clinical opinion. An example of a criteria summary for the topic of ‘PET and lung cancer staging’ is included in Appendix 2. To supplement these summaries, the ADVICE project team also created a set of more extensive technical briefs with sections suggested by the CMTP group [105] including:

• ▪Description of the technology, approved indications, known information about diffusion;
• ▪Description of the patient group (with estimated patient numbers), expected utilization rates;
• ▪Current diagnostic or treatment alternatives;
• ▪Known over-, under- and misuse of technology, variations in use or outcomes;
• ▪High level summary of the current research evidence of clinical effectiveness, benefits, harms, and any known evidence gaps, patient safety issues;
• ▪Details of any ongoing or related research activities;
• ▪Direct and indirect healthcare costs (annual and/or lifetime);
• ▪Potential cost–effectiveness as compared with alternative treatments;
• ▪Known information about reimbursement (public and private payers);
• ▪Political climate surrounding the technology, particular reasons for urgency;
• ▪Social or ethical concerns related to the technology;
• ▪Methodological implications (requirements to complete research).

Through the development of these in-depth technical briefs we identified critical evidence gaps that could potentially be addressed with data from our stakeholder group.

Stakeholders were sent the criteria summaries and technical briefs approximately 1 month before the scheduled face-to-face meeting and were asked to rate each topic via a web survey as low, medium or high’ priority based upon their review of the technical briefs. Results of the survey, compiled before the face-to-face meeting, allowed the project team to identify areas of controversy or disinterest, aiding in the organization of a 2-h meeting. During the meeting, a brief overview of each topic was given by the project team, then, after a short discussion, an open voting procedure was used to obtain an overall rating for each topic. In a manner akin to the Medicare Evidence Development and Coverage Advisory Committee (MEDCAC) panel voting procedure [22,106], each stakeholder was asked to give each topic a strength of preference rating (five-point Likert scale, where one represented the lowest and five represented the highest priority). Stakeholders were allowed to give the same rating to different topics, and the voting process was transparent. Stakeholders, selected in random order, held up a card representing their vote as well as an additional card that identified which of the criterion were most influential for that vote. Votes were tabulated and sent to the stakeholders via email after the meeting.

Stakeholder-influenced study objectives

A novel aspect of our stakeholder engagement process was the two-step analysis approach. After stakeholders identified topics of interest, the research team worked with a subset of health plans to conduct a rapid-turnaround ‘internal analysis’ of utilization trends. The purpose of this initial analysis was to establish study feasibility and refine the scope and direction of the CER questions. Second, we conducted a full analysis of data from all payers.

Internal analysis

From the initial discussions with stakeholders, it became clear that the regional patterns of use for breast cancer imaging modalities was unknown to payers, clinicians and policymakers alike. While stakeholders believed that utilization of advanced imaging was increasing with high variability between clinicians, they did not have data reflecting which types of imaging tests were increasing or the time points at which the imaging occurred (e.g., diagnosis, staging, treatment monitoring, or surveillance).

To address this, the ADVICE research team and stakeholder group designed and implemented a rapid-response internal analysis. This analysis used fairly simple inclusion/exclusion criteria and Current Procedural Terminology (CPT) and International Classification of Diseases, Ninth Revision (ICD-9) codes, which were applied by the three participating payers to their own claims records and then reported as summary data in identical shell tables. ADVICE team members collected the de-identified data and presented these data to the voting members of the stakeholder group. This greatly simplified or obviated the requirement for data use agreements between the research team and payer groups, which typically can take months.

After presentation and discussion of the breast cancer internal analysis results during an in-person stakeholder meeting, the stakeholders were asked to consider a number of possible study objectives and to add or modify them as appropriate. After a list of potential study objectives was generated, the stakeholders gave a ‘strength of preference’ rating using the same five-point scale and procedures used for study topic prioritization. Voting results were compiled, sent to the stakeholders via email, and the project research team immediately proceeded to create an analysis plan for the top study objectives.

Results

Study topic prioritization

At the first stakeholder meeting, use of PET/computed tomography (CT), MRI and in vitro biomarkers were discussed as diagnostic areas with evidence gaps. Stakeholders expressed concern over potentially rising costs due to anecdotal evidence of greatly increasing use. Cancers with the greatest incidence were of highest concern, since these were presumed to have the greatest burden on the healthcare system. The stakeholders were also interested in finding the amount of variability that existed between clinicians, healthcare providers and insurance plan types. They felt that variability in imaging could be used to identify over- or under-utilized diagnostic techniques. The following four general topics were identified: determine the variation in advanced imaging practice among the four most common cancers (breast, lung, colorectal and prostate); evaluate PET or PET/CT effectiveness for measuring treatment response and for post-treatment surveillance for survivors of non-small-cell lung cancer (NSCLC); compare the effectiveness of PET/CT versus CT for staging, restaging and surveillance of lymphoma; and identify lab-based surveillance techniques that are most effective for monitoring tumor recurrence.

After the second in-person stakeholder meeting, six candidate studies reached the prioritization phase, excluding breast cancer which had already been chosen as a topic for the first ADVICE study. The six candidate studies were:

• ▪PET/CT and lung cancer staging: for patients with advanced NSCLC, does PET/CT-guided staging influence diagnostic evaluation and procedures, choice of primary therapy, cancer-specific and overall survival and short- and long-term medical care costs when compared with conventional staging?
• ▪PET/CT and lung cancer surveillance: among patients with regional and advanced stage NSCLC who have completed primary therapy, does the use of PET/CT influence post-treatment surveillance, subsequent treatment choices, patient survival and cost of care compared with CT alone?
• ▪PET/CT and head and neck cancer staging: does the use of PET/CT for staging of head and neck cancers influence diagnostic evaluation and procedures, improve cancer-specific and overall survival, and reduce short- and long-term cost of care?
• ▪PET/CT and lymphoma surveillance: does the use of PET/CT for post-treatment surveillance of lymphoma compared with CT alone influence subsequent treatment choices, improve cancer-specific and overall survival, and reduce short- and long-term cost of care?
• ▪PET/CT and thyroid cancer surveillance: does the use of PET/CT for post-treatment surveillance of thyroid cancer of follicular cell origin compared with no imaging influence treatment choices, and reduce short- and long-term cost of care?
• ▪Carcinoembryonic antigen and colorectal cancer surveillance: does the use of carcinoembryonic antigen post-treatment surveillance of colon cancer influence subsequent treatment choices, improve cancer-specific and overall survival, and reduce short- and long-term costs of care?

Final prioritization votes for the candidate topics gathered at the conclusion of the third stakeholder meeting are displayed in Table 1, and a summary of the most important criteria used by the stakeholders is seen in Table 2. PET use during lung cancer staging was the highest ranked study, primarily due to the high incidence of lung cancer, potential high costs of PET imaging for lung cancer patients and the unknown effectiveness of PET in community (as opposed to randomized control trial) settings. In general, economic impact of diagnostic testing was a main concern to the stakeholder group, and the stakeholders cited this criterion as a major factor affecting their votes for both high- and low-priority topics (Table 2). Study feasibility was also an important criterion, particularly for studies that received low-priority scores. Similarly, PET use during lung cancer surveillance was seen as an important evidence gap to fill, but since surveillance PET is not reimbursed by most payers, it was uncertain whether sufficient claims data would be available to make this study viable.

Table 1.

Results of candidate study prioritization.

Topic	Total priority score
PET/CT and lung cancer staging	28
PET/CT and lung cancer surveillance	23
PET/CT and head and neck cancer staging	14
PET/CT and lymphoma surveillance	23
PET/CT and thyroid cancer surveillance	14
CEA and colorectal cancer surveillance	25

CEA: Carcinoembryonic antigen; CT: Computed tomography.

Table 2.

Most important criteria affecting prioritization vote.

Stakeholder prioritization ratings	Most important criteria
High-priority ratings	Economic impact Pressure on payers
Low-priority ratings	Feasibility Disease burden Economic impact

Breast cancer internal analysis results & study objective prioritization

Three payers contributed to the breast cancer internal analysis to achieve a study population summarized in Table 3. The 7213 women included in the initial enrollment period represent approximately 30% of all women expected to be included in the full ADVICE breast cancer population. Since these data were gathered to establish study feasibility and guide the study design, these results should only be interpreted as general trends of imaging practices in our community. The inherent limitations of the rapidly collected data were fully disclosed and discussed, including the fact that data were limited to the claims that were easily accessible at each site, data may have been analyzed differently at each of the three sites, cancer cases were unconfirmed, results were not linked with outcomes, and there was no way to determine if the imaging was because of breast cancer or because of an unrelated medical condition.

Table 3.

Breast cancer internal analysis population.

Definition	Female Age 18+ Breast cancer diagnosis Continuously enrolled in plan at diagnosis (−3–6 months) Resides in Surveillance, Epidemiology and End Results Puget Sound region
Enrollment period (in relation to diagnosis)	Site
	A	B	C	Total
−3–6 months	2250	2964	1999	7213
7–12 months	2221	2649	1934	6804
13–24 months	1893	2094	1569	5556
25–36 months	1247	1657	900	3804
37 or more months	793	1610	383	2786
Diagnosis timeframe
	April 2002–December 2009	April 2002–December 2009	June 2005–December 2009

Figure 3 displays the utilization rates by year for PET/CT, MRI and CT for the three plans during the study period and shows the marked increase of MRI and PET/CT between 2002 and 2009. Use of mammography and ultrasound remained relatively stable during the same period [Unpublished Data]. Figure 4 displays utilization rates by imaging interval with respect to date of diagnosis. Mammography and ultrasound use were both high in the period before diagnosis, as this period also includes imaging done for conventional screening. Figure 4 also gives an indication of the variability for advanced imaging between health plans. Most advanced imaging was used in the 6-month period after diagnosis, and we observed a great deal of variation in usage between plans. Two plans showed markedly higher usage of both MRI and PET compared with the third plan in the study. The ratio of each modality’s use in the pretreatment period compared with the post-treatment period differed by plan (Figure 5). While mammography and ultrasound were used fairly consistently by all three plans, we observed more plan variability in the use of advanced imaging. For example, site B clinicians used MRI and PET/CT proportionally much more in the post-treatment setting than clinicians who provided care under site A and C insurance plans.

Figure 3. Breast cancer imaging internal analysis results.

Utilization rates for advanced imaging during the peri-diagnostic period (date of diagnosis to 3 months postdiagnosis) climbed dramatically for all payer plans between 2002 and 2009. Greatest increases and related cost increases were seen for MRI, which led the stakeholders to rate evaluations of this modality the highest priority objective for the breast cancer study.

BC: Breast cancer; CT: Computed tomography.

Figure 4. Breast cancer imaging internal analysis – variability by payer plan.

While utilization rates of CT were fairly consistent between plans, considerable variability was seen in the use of MRI and PET imaging for breast cancer patients. Stakeholders viewed high variability as a signal for high-priority studies to evaluate over- or under-utilized diagnostic techniques.

BC: Breast cancer; CT: Computed tomography.

Figure 5. Breast cancer pretreatment/post-treatment imaging variability.

Comparing the ratio of imaging procedures carried out pretreatment versus post-treatment, the internal analysis also showed great variability among plans for MRI and PET/CT, while usage of mammography, ultrasound and CT was more consistent. Pretreatment: 3 months prior to diagnosis up to (and including) date of treatment. Post-treatment: date of treatment to 12 months postdiagnosis CT: Computed tomography.

Results of the ratings of study objectives are summarized in Table 4. The first four objectives focus on advanced imaging used with early stage cancer; because PET/CT is not generally reimbursed for early stage breast cancer, these objectives reflect MRI imaging only. A fifth potential objective considered PET/CT usage in regionally advanced breast cancer. MRI utilization rates were much higher than PET/CT rates based on the internal analysis, so the stakeholders felt the MRI objectives were more important. Assessing whether MRI affects treatment decisions – potentially increasing the mastectomy rate with no survival benefit [23] – was judged as the top priority study objective, followed by costs of care.

Table 4.

Breast cancer study candidate objectives.

Stage	Potential study	Priority score
Early-stage cancer	Re-excision and time to definitive surgery Treatment differences Cost of care initial diagnostic period Cost of care during surveillance period	21 33 28 29
Regionally advanced cancer	Use of PET/CT for staging	22

CT: Computed tomography.

Discussion

The regional stakeholder-based prioritization process developed for the ADVICE project addresses several potential obstacles to effective stakeholder engagement in CER. Early involvement of stakeholders enabled strong alignment of the research objectives of greatest priority for the region. Logistically, relying on a group of regional, rather than national, stakeholders and clinical experts made scheduling of multiple face-to-face meetings feasible, and was time and cost efficient. As other researchers have found, face-to-face stakeholder meetings were crucial in establishing common goals, mutual understanding amongst all parties and overall buy-in of the project [24]. Similarly, stakeholder engagement was maintained by involving the group in data collection, study design and implementation phases. Finally, the same stakeholders will also be involved in disseminating the results to impact community clinical practice as the research is completed.

The stakeholder group and prioritization process developed as a part of this project were unique in many aspects. Stakeholders were involved not only in initial selection of research topics, but also in the research design and implementation. Many of the stakeholders directly contributed data to our studies, which made the research results readily applicable to their organizations, and also streamlined the satisfaction of legal hurdles required at each organization prior to data collection and analysis. Stakeholders also included representatives from organizations with competing financial interests and represented a diverse set of public, managed care and fee-for-service healthcare systems, which allowed us to conduct community-based studies of diagnostic technologies. Our two-step analysis approach involving a rapid-turnaround ‘internal analysis’ of utilization trends before the main analysis allowed stakeholders to consider the feasibility and scope of the CER question as a major part of the prioritization process. An early look at imaging utilization patterns and variability of use in community practice was very helpful, if not essential, to assess study feasibility and inform estimates of economic impact – two important stakeholder criteria in assessing the overall importance of research questions. These data are seldom available from published literature or public data sets and the time and effort required to obtain institutional review board approval and data use agreements to collect this information is significant. Despite the limitations of the analysis, the data were extremely useful for steering study design and addressing feasibility issues.

The sequence of and preparation for stakeholder meetings allowed stakeholders to substantively engage in a wide range of technical questions. By first identifying a fairly broad list of possible topics, allowing adequate time for topic evaluation via a formalized framework with agreed-upon standardized criteria, and then meeting again for a final prioritization session, we found the stakeholders to be well informed and not adversely affected by gaps in knowledge or biased by their particular areas of expertise. As an example, study feasibility was found to be a very influential criterion for the stakeholder decision process. However, it would have been difficult to evaluate feasibility without the information that was collected from literature reviews and clinical experts about regional disease incidence, recommended imaging practices and reimbursement policies between stakeholder meetings.

A problem common to CER prioritization efforts has been the unclear path between completion of the prioritization process and the actual implementation of the stakeholder recommendations in conducting CER studies. An important issue in this is the lack of a well-defined mechanism linking prioritization results to CER funding. Past efforts to prioritize research using a stakeholder approach have usually been distinct from implementation efforts. Prioritization has generally been carried out by groups different from CER researchers who would implement the work, and the mechanism for funding the research is usually unspecified. For ADVICE, linking the results of the prioritization process to study implementation was immediate and implicit within the structure of the funded grant, which was somewhat unique to the ARRA applications. The project was funded from the outset as a unified process including both the stakeholder-guided prioritization and implementation stages. Such a design was possible via funding from the 2009 ARRA Stimulus Bill, but whether this model is acceptable to funding agencies in the future is unknown. Funding open-ended, stakeholder-guided research has not been the norm for agencies such as the AHRQ and the NIH [25]. With the recent release of requests for proposals from the new Patient Centered Outcomes Research Institute (PCORI) [107], this trend may change in the future.

The ADVICE methodology for stakeholder engagement was not without challenges. Logistically, focus on a regional network of stakeholders made initial contact via telephone or email feasible and effective, yet scheduling compatible times for even regional stakeholders was not trivial. The limited time available for executive-level stakeholder members to participate in the project required the utmost attention to making each meeting, phone call and email communication as efficient as possible. Identifying appropriate stakeholders could also be an issue. ADVICE stakeholders were identified primarily through outreach to key opinion leaders in the community using the network of contacts known to the study team. This is a commonly used method for past research prioritization efforts [8], and may be particularly suited to identification of regional stakeholders, but this methodology may not be appropriate or effective for other research teams.

Two areas that were of high importance during the landscape analysis phase that were particularly difficult to address were the expected imaging utilization rates and direct costs of diagnostic tests. Few published or publicly available data sets were available, with the possible exception of Medicare data. In the future, establishment of data registries or distributed data networks, such as the Health Maintenance Organization Research Network Virtual Data Warehouse [26] and US FDA Sentinel project [27], may allow for rapid and practical identification of high imaging variability and areas of under- or over-utilization that warrant high-priority studies. Likewise, access to such databases for full CER studies could greatly reduce the time and effort required to obtain institutional review board approval and establish data use agreements and protocols, which otherwise are likely to be of a ‘one time use’ form.

Another challenge was how to adequately represent the interests of patient stakeholder groups in CER, particularly during the early phases of project prioritization and selection. Initially, we included a patient perspective by including a representative from the Puget Sound Health Alliance at the kick-off meeting, but this representative was the extent of patient involvement in our process to date. For the ADVICE project, it was not feasible to include patient stakeholders due to the inherent uncertainty around topic selection, and time and budgetary constraints. The range of potential CER topics at the outset of the project was very large, spanning all cancers and phases of diagnosis and treatment. Identification and selection of a representative patient group or groups that would have a broad and unbiased knowledge over the various conditions, imaging tests, outcomes and claims data may have been possible, but the investigators felt that it would be difficult to do this within the framework of the grant timeframe and budget. This is largely uncharted territory in CER research, and one of the priority areas of PCORI [28]. It would have been more straightforward to involve patient groups from the start if the initial scope of potential research projects were somewhat narrower. For example, if the scope of research was limited to a topic within breast cancer imaging, it would have been natural to consider local breast cancer advocacy groups in the prioritization process. We intend to incorporate patient stakeholders in the dissemination phase of ADVICE, and in fact, this is the topic of an upcoming stakeholder meeting, but ideally, patients would be involved through all phases of the CER project.

Conclusion

We have described a regional stakeholder engagement process that has proven to be effective for topic prioritization and study implementation for cancer diagnostics CER. Continuous involvement of payer, clinical and health policy stakeholders from all phases of project development, including project prioritization, study design and contribution of patient-level data greatly enhanced stakeholder engagement and enthusiasm for the project. A focus on regional interests and use of regional data will likely lead to improved dissemination of research results to the clinical community and ultimately a change in clinical practice to make more effective use of diagnostic technology applied to cancer therapies.

Executive summary.

• ▪Regional participation of stakeholders from the beginning of the project ensured that research objectives were appropriate to the needs of the local community.
• ▪Proximity of stakeholders made multiple in-person meetings logistically efficient.
• ▪Stakeholders included representatives from organizations with competing financial interests from a diverse set of publicly managed and fee-for-service healthcare systems.
• ▪A high level of involvement with data collection and study design helped stakeholders to establish common goals, develop a mutual understanding, and ensure long-term buy-in to the project.
• ▪Use of an internal analysis improved study efficiency. Although the analysis had limitations, early looks at pooled de-identified data across multiple plans was critical for assessing study feasibility and cross-plan variability as well as informing estimates of economic impact.
• ▪Stakeholders participated in selection and modification of study objectives. Researchers incorporated stakeholder preferences into the internal analyses.
• ▪A focus on regional interests and use of regional data will lead to improved dissemination of results to the clinical community to make more effective use of cancer diagnostic technologies.
• ▪Adequately representing the interests of patient stakeholder groups was challenging, particularly during the early phases of project prioritization and selection.

Appendix 1. ADVICE study criteria selection matrix.

Criteria for selection of study	Description of criteria
Minimum criteria
Included within statutory limits of Federal Coordinating Council’s definition of CER	Is the topic included within the definition of CER detailed in the mandate of the Recovery Act?
Stakeholder/community interest	Does the topic answer to expressed needs and preferences of patients, clinicians, and other stakeholders?
Feasibility	Are there available databases, registries, collaborations and resources to support study design? Are data from sufficient numbers of patients available for analysis? Do the data span sufficient time to evaluate desired outcomes?
Evidence needed	Is the modality currently being used without adequate evidence?
Primary criteria
Standards for modality and implementation of test results	Are guidelines in place for the various uses of this modality? Are existing guidelines evidence-based and unbiased? Are they consistently implemented by providers?
Potential patient harm/safety concerns	What is the potential degree of harm (e.g., anxiety, radiation exposure, unnecessary treatment) an individual may experience from use of the modality? What are potential downstream harms from incorrect diagnosis using this modality?
Diagnostic accuracy and efficacy potential	How accurate is the modality for disease diagnosis? Does the modality offer potential for changing patient treatment path or outcome?
Economic impact	What is the estimated total direct cost per year (increase/decrease) of the modality and downstream treatments?
Secondary criteria
Severity of condition treated/diagnosed by modality	Are significant morbidity, mortality, and/or disability associated with disease?
Pressure on payers	Is there a high degree of pressure on payers to cover this modality, even though little evidence supports its use?
Potential or observed variation	Does payer/provider use of modality vary widely?
Diverse populations	Is there potential to evaluate CER in diverse populations and patient sub-groups affected by disease and/or modality?
Overlap/redundancy	Is/has similar research being/been conducted elsewhere?
Disease burden	How many persons are affected by disease per year? What is the disease cost to society?

CER: Comparative effectiveness research.

Appendix 2. Example of criteria summary table: PET and lung cancer staging.

Criteria	Evidence for criteria	Criteria grade
Minimum criteria
Included within statutory limits of Federal Coordinating Council’s definition of CER	Yes	High
Stakeholder/community interest	Yes	High
Feasibility	High incidence and mortality rate result in relatively large data sets available for analysis through insurance claims data linked to SEER. Initially, would be informative to look at claims data to get idea of variations in use of PET/CT for staging, restaging, and post-treatment surveillance of NSCLC patients in western Washington State	High
Evidence needed	Two RCTs performed outside the USA found fewer futile surgeries in the pre-operative PET/CT group, but no difference in survival. Studies looking at use of PET/CT in community (vs academic) setting needed	Medium
Primary criteria
Standards for modality and implementation of test results	NCCN guidelines recommend that PET/CT be used for pretreatment evaluation of NSCLC, but PET/CT is still not regarded as a standard of care for NSCLC staging. Controversy exists regarding PET/CT as an add-on or replacement test	High
Potential patient harm/safety concerns	Unnecessary radiation exposure in patients who receive PET/CT scans; potential unnecessary surgeries due to PET/CT false positives, or delayed treatment for false negatives; also potential for unnecessary surgeries in patients who do not receive PET/CT scans	High
Diagnostic accuracy and efficacy potential	Considerable evidence of PET/CT offering superior accuracy in NSCLC staging. RCTs and retrospective studies generally agree that PET/CT offers potential to change patient management, but there is little evidence for survival improvements	High
Economic impact	According to SEER–Medicare data, approximately 12% of lung cancer patients filed claims for PET/CT between 2005 and 2007. Cost of PET/CT is approximately US$2500–4000, but might reduce costs by eliminating unnecessary surgeries and by allowing surgeons to better target areas of resection	High
Secondary criteria
Severity of condition treated/diagnosed by modality	High incidence and low survival rates; accounts for the most cancer-related deaths. US incidence rate: 69.0/100,000; 5-year survival: 16%; US death rate: 53.4/100,000	High
Pressure on payers	Anecdotal evidence suggests that use of PET/CT scans is increasing, with no evidence that the scans provide information that decreases morbidity, mortality, or recurrence rates	Medium
Potential or observed variation	No literature on this	No data
Diverse populations	Access to PET/CT machines is variable depending on patient location	Medium
Overlap/redundancy	Two RCTs looking at PET/CT for detecting recurrence in several cancers, including lung; none in western Washington State	Medium
Disease burden	Approximately 4000 incident cases in Washington State state per year; US prevalence: 370,617; aggregate 5-year Medicare cost (in 2004 dollars) of care for treatment of lung cancer was estimated to be US$4238 million	High

CER: Comparative effectiveness research; CT: Computed tomography; NSCLC: Non-small-cell lung cancer; RCT: Randomized controlled trial; SEER: Surveillance, Epidemiology and End Results.