It takes a (virtual) village: crowdsourcing measurement consensus to advance survivorship care planning

Abstract

We report results from the use of an innovative tool (the Grid-Enabled Measures (GEM) database) to drive consensus on the use of measures evaluating the efficacy and implementation of survivorship care plans. The goal of this initiative was to increase the use of publicly available shared measures to enable comparability across studies. Between February and August 2012, research and practice communities populated the GEM platform with constructs and measures relevant to survivorship care planning, rated the measures, and provided qualitative feedback on the quality of the measures. Fifty-one constructs and 124 measures were entered into the GEM-Care Planning workspace by participants. The greatest number of measures appeared in the domains of Health and Psychosocial Outcomes, Health Behaviors, and Coordination of Care/Transitional Care. Using technology-mediated social participation, GEM presents a novel approach to how we measure and improve the quality of survivorship care.

BACKGROUND

Development of an evidence base in survivorship care planning

In its 2006 report “From Cancer Patient to Cancer Survivor: Lost in Transition,” the Institute of Medicine (IOM) suggested that the cancer care community consider new models and approaches to coordinating transitional and follow-up care for cancer survivors [1]. Among the IOM’s specific recommendations was the suggestion that all patients completing primary treatment for cancer be provided with a survivorship care plan (or SCP), a document combining a treatment summary with a recommended course for follow-up care, and an end-of-treatment consultation. The IOM also called for care planning interventions to be evaluated for efficacy and feasibility [1]. Subsequently, the American College of Surgeons’ Commission on Cancer (CoC) called for implementation of SCPs by January 2015 in all accredited programs [2]. While the practice community strives to comply with the CoC directive and impending deadline, the scientific community is scrambling to catch up: evidence regarding the impact of survivorship care plans and care planning interventions remains scant, largely developmental, and somewhat controversial, as outlined in a series of earlier commentaries and articles [3–16]. The challenges to the development of science in this area have included the absence of a conceptual framework guiding empirical efforts and a lack of consensus and recommendations about the appropriate constructs and process and outcome measures to use in assessing survivorship care planning.

This paper reports findings from the use of a novel participative technology platform to support measurement identification and development, and ultimately to support the coordinated growth of a knowledge base in survivorship care planning that integrates the perspectives of researchers, policymakers, and clinical practitioners. As such, we present findings from a case study of an experiment aimed at vetting and promoting the use of shared measures. Specifically, we asked the scientific and clinical communities to participate in a Grid-Enabled Measures (GEM)-Care Planning Initiative (GEM-CP) and to provide us with suggestions and feedback on the best constructs and measures to evaluate the efficacy and impact of survivorship care planning. Our approach was to interact with stakeholder communities to build agreement around a framework and measures to guide evidence-based survivorship care planning as it unfolds at a national level. The purpose of this report is to describe the GEM process [17], present preliminary results, and provide an example of a technology-mediated social participation experiment.

What is GEM? Using technology-mediated social participation to drive agreement on measurement

With the goal of stimulating research on survivorship care planning, we made use of the NCI’s Grid-Enabled Measures database [17–19]. GEM is a Web 2.0 platform—an Internet-based tool that provides architecture for the participation of multiple individuals working together in a coordinated and accelerated fashion [20–22]. GEM uses a process referred to as technology-mediated social participation (TMSP) to accept feedback (data) from multiple users and to allow them to comment upon and respond to other’s inputs. This process creates a mechanism for tapping community intelligence or the collective knowledge, insight, and discoveries inherent, but perhaps unrecognized, within a community of scientists when they work in isolation [20, 23]. TMSP, as supported by the National Science Foundation, has been used to accelerate progress in other scientific domains by capitalizing on the network effect of coordinating inputs simultaneously through massively distributed communication channels [24].

A major goal of the NCI in creating GEM was to catalyze progress across multiple areas of scientific inquiry by promoting the standardized use of measures and harmonization of data [25]. Survivorship care planning research is a new area of inquiry and, as such, presents an opportunity to build consensus a priori regarding the most important constructs and associated measures to consider when working together to advance a national collaborative agenda. We sought to use GEM to engage a critical mass of survivorship care planning stakeholders (including those funded by NCI and other sources) in an agreement-building process around measurement. By doing so, we hoped to encourage researchers to use common measures in their individual studies, thereby collectively engaging in a type of prospective meta-analysis [20]. In this way, we hoped, the care planning evidence base would have improved chances of growing quickly and efficiently, and being composed of harmonized, complementary, and synergistic studies.

METHOD

The purpose of the GEM-Survivorship Care Planning Initiative was to build consensus in the survivorship community around the use of high-priority process and outcome measures in studies of survivorship care planning. The ultimate goal of this initiative was to increase the use of shared measures to enable comparability across studies and to facilitate identification of strategies for implementing care planning (or barriers to that planning) for cancer survivors.

To promote a paradigm of open-access science as outlined above (aligned with recent directives to government agencies from the White House Office of Science and Technology Policy [26, 27]), the GEM-CP team was meticulous in its construction of a carefully designed “architecture of attention” [25] to focus the community’s efforts on sharing, rating, and selecting measures. The creation of a dedicated GEM-CP workspace, where relevant constructs and measures were organized in one place, was the mainstay of this architecture and was created in direct response to feedback from GEM users. Next, we solicited involvement from stakeholders who were known to have significant investment in advancing care planning research, including scholars and clinicians involved in research, evaluation, and implementation of care delivery in cancer and cancer survivorship. These “champions” served to seed the GEM-CP community with an early level of enthusiasm and engagement critical to the success of TMSP initiatives [28]. The champions consisted of 19 stakeholders who primarily represented academia but also included clinicians and those representing advocacy/policy groups. These participants were asked to reach out to their peers and professional groups to encourage others to participate and serve as brokers recruiting others to offer feedback. We made participation as easy as possible. GEM-CP users could provide extensive input to the community (e.g., add a measure or construct to the workspace) but could also make a “microcontribution” [25] (e.g., comment on an existing measure or construct in the workspace). In Fig. 1, we provide a screenshot of the measures area of the GEM-CP workspace, showing a partial list of constructs (left column) and associated measures (right column).

Fig. 1.

Screenshot of the measures page of the GEM-Care Planning workspace, showing a partial list of constructs (left column) and associated measures (right column)

The GEM project began in February 2012 and has consisted of two phases. The first phase aimed to motivate the research and practice communities to populate GEM with care planning-related measures and relevant constructs and to rate those measures using a 5-point scale and qualitative feedback. Preliminary results from this phase were shared at the NCI’s Biennial Cancer Survivorship Research Conference in June 2012. Suggested feedback on the project included a call to categorize and organize the lengthy list of measures within an evolving conceptual framework. As a result, measures were organized by construct and constructs were grouped within larger domains that mapped to the conceptual framework. This feedback was incorporated into the second phase of the GEM initiative, which followed the biennial conference: July 2012 to August 2012. In this phase, we solicited additional targeted feedback in the form of ratings and comments, specifically with respect to each measure’s (1) psychometric properties, (2) acceptability, and (3) usability in real-world settings. The findings presented herein are from a pooled analysis of the first two phases of GEM-CP (February 2012 to August 2012, combined).

RESULTS

Between February 2012 and August 2012, a total of 51 constructs were entered into the GEM-CP workspace by participants. As noted above, these constructs were ultimately organized into higher-order categories, called domains, to explicate the hierarchical relationships between domains, constructs, and measures and to make construct and measure retrieval more organized and user-friendly. Table 1 shows the seven initial domains and the number of measures associated with each domain. Table 2 shows an example of the hierarchy of domain, construct, and measure for one domain: Quality of Care. The domains and constructs in Table 2 are congruent with those published in a companion effort to develop a conceptual framework in care planning [13].

Table 1.

Categorization of measures within domains of an evolving conceptual framework

Domain	Number of GEM-CP measures
Quality of Care	8
Health and Psychosocial Outcomes	47
Patient Knowledge and Preparation	8
Organizational, Implementation	8
Health Behaviors, Self-management, Adherence	18
Costs	3
Coordination of Care/Transitional Care	10

Table 2.

Example of constructs and measures within the domain: Quality of Care

GEM domain: quality of care
GEM construct	GEM measure
Chronic Illness Care	Assessment of Chronic Illness Care (ACIC) Patient Assessment of Chronic Illness Care (PACIC)
Patient Goals	Control Preferences Scale Goal Evaluation Tool (GET) Open-Ended Goal Setting Tool Patient’s Preferred Method for Communication
Patient Satisfaction	Patient Satisfaction with Cancer Care Picker Institute Cancer Survey Modified by Ayanian

Of the 51 constructs entered into GEM, 15 constructs (29 %) had three or more measures linked to them. The most prevalent constructs with three or more measures were Quality of Life (eight measures), Care Coordination (six measures), Functioning (eight measures), Health Behavior (five measures), Stress/Distress (five measures), and Fatigue (five measures). Thirty-one constructs (70 %) had fewer than three measures associated with them, highlighting the need to continue to populate the GEM-CP workspace with measures relevant to those constructs or to develop measurement in these areas, as needed.

A total of 124 measures were linked to the GEM-CP workspace. One hundred and two (102) of the 124 measures were ready to be rated (i.e., had the minimum required meta-data) based on the following criteria: (1) psychometric properties, (2) acceptability, and (3) usability in real-world settings. These measures were then categorized into the domains appearing in Table 2 (above) during the second phase of the project. Seventy-seven (75 %) of the 102 rate-able measures received ratings from GEM-CP users. Of the 77 measures that received ratings, 31 (40 %) measures had three or more distinct ratings.

In total, 37 unique users rated a GEM-CP measure and 42 unique users commented on a GEM-CP measure, with some users electing to both rate and comment on measures. Those who commented or rated a measure represented academia (54 %), clinical settings (19 %), advocacy/policy groups (3 %), and finally, those whose affiliation was not provided and thus is unknown (24 %). Of these stakeholders, the majority contributed 1–3 comments or ratings and a small number of participants provided 20 or more ratings. Due to the crowdsourcing process used to recruit participants, it was impossible to determine how many stakeholders were ultimately asked to participate and so traditional measures of response rates are not calculable.

Table 3 provides a list of measures with three or more four- or five-star ratings, organized by domain. The number of measures meeting this criterion in each domain was, respectively, Patient Knowledge and Preparation (five measures); Health Behaviors, Self-management, and Adherence (one measure); Organizational Factors/Implementation (two measures): Health and Psychosocial Outcomes (eight measures); and Coordination of Care/Transitional Care (four measures). The greatest number of GEM-CP measures per domain was found in Health and Psychosocial Outcomes (47 measures), Health Behaviors, Self-management, Adherence (18 measures), and Coordination of Care/Transitional Care (10 measures). Domains with the fewest measures included Quality of Care, Patient Knowledge and Preparation, Organizational/Implementation, and Cost.

Table 3.

Measures with three or more ratings that received an average of four or five stars

Domain	GEM construct	GEM measure	No. of ratings
Measures with 3 or more ratings that had an average of 5 stars
Patient Knowledge and Preparation	Confidence in Survivorship Information	Confidence in Survivorship Information (CSI)	4
Measures with 3 or more ratings that had an average of 4 stars
Patient Knowledge and Preparation	Perceived Autonomy Support	Perceived Autonomy Support	6
…	Patient Activation	Perceived Efficacy in a Patient-Physician Interactions Scale	4
…	Perceived Competence Scale	Perceived Competence Scale	3
…	Chronic Illness Care	Patient Assessment of Chronic Illness Care (PACIC)	3
Health behaviors, Self-management, Adherence	Smoking Cessation	The Contemplation Ladder	3
Organizational/Implementation	Adoption	RE-AIM Adoption Measure	5
…	Implementation	RE-AIM Implementation Measure	3
Health and Psychosocial Outcomes	Cancer and Health Worry	Assessment of Survivor Concerns	7
…	Depression	Patient Health Questionnaire (PHQ-9)	6
…	Anxiety and Depression	Patient Health Questionnaire for Depression and Anxiety (PHQ-4)	5
…	Depressed Mood	Center for Epidemiologic Studies Depression Scale (CES-D)	4
…	Fatigue	Brief Fatigue Inventory	4
…	Comorbidity	The Charlson Index	3
…	Psychological Distress	Kessler Psychological Distress Scale	3
…	Stress	Distress Thermometer	3
Coordination of Care/Transitional Care	Continuity of Care	Patient Continuity of Care Questionnaire (PCCQ)	5
…	Care Coordination	Patient Expectations for Cancer Survivorship Care	3
…	Provider Attitude and Behavior	Survey of Physician Attitudes Regarding Care of Cancer Survivors-PCP	3
…	Provider Attitude and Behavior	Survey of Physician Attitudes Regarding Care of Cancer Survivors-Oncologist	3

“…” indicates continuation of the domain from the preceding row

DISCUSSION

Concurrent with its emphasis on cancer care planning, the IOM has engaged in a series of workshops designed to explore the feasibility of utilizing common data structures and harmonized measures to accelerate discovery, facilitate attainment of national quality standards, and support patient-centered research goals in health care settings. Collectively, these workshops have explored the feasibility of utilizing innovative information systems to create a “learning health care system in America” [29]. In this paper, we have reported early results from the use of one such innovative tool for identifying common measures to use in evaluating and improving the implementation of survivorship care plans, as championed by the IOM and the American College of Surgeon’s Commission on Cancer [2]. The prospect of creating a continuously improving health care system in oncology concurs with recommendations from the IOM’s “National Cancer Policy Forum” [30] as well as recommendations from the American Society of Clinical Oncology [31].

In our efforts, we created a care planning workspace on NCI’s measures harmonization platform, GEM. The volume and rating patterns of measures within the domains and constructs of interest in the GEM-Care Planning Initiative serve as indicators of the most well-developed or familiar areas of science in survivorship research (e.g., Health Behaviors and Quality of Life), as well as some of the least developed or less familiar areas of activity (e.g., Quality of Care measures). It was then the responsibility of the GEM-CP design team to understand how the design features of the online workspace could be optimized to achieve the explicit goals of the project [32]. If the team were to spend too much time in the campaign’s “populate” phase, or if feedback cues within the online architecture placed too much attention on actions that would lead to an endless proliferation of measures rather than the careful selection of best measures, then the course of the community-based project would be expected to veer away from its “use-inspired” research goals [33]. In this case, the primary research goals had to do with identifying the right set of core measures to enable broad community acceptance of harmonized data tools. Through successive iterations on the Web, the team became keenly aware of the delicate influence of system design on community behavior [20, 25, 34].

In addition to serving as a repository of measures relevant to survivorship care planning, the GEM-CP workspace may serve as a source of stakeholder-driven quality ratings for relevant measures (Table 3 indicates which measures received consistently high ratings from participants). Anecdotal feedback suggests researchers have used the GEM-CP workspace to select and compare measures. One user’s report serves as a typical use case. This researcher was able to use GEM as a decision tool to identify a specific measure to use in her research protocol for a construct of interest. By selecting a high-quality measure that others recommended and used, the researcher increased her potential ability to pool her data with other colleagues in her field. Practitioners may be able to use GEM-CP in a similar manner to determine the best measures for use in clinical settings, resulting in use of shared quality measures to support data integration.

The development of a conceptual framework for understanding the process and potential impact of survivorship care planning was foundational to moving this research area forward. The final conceptual framework [13, 35] evolved from a combination of evidence review, expert opinion, and through interaction with the GEM-CP community. By suggesting constructs and measures relevant to the proffered model of survivorship care planning, this community of practitioners and scientists guided framework development.

CHALLENGES AND IMPLICATIONS

One might consider the modest participation in GEM to be a limitation of this experiment in using social media to drive measurement consensus. Admittedly, the virtual village we formed is a small village, composed of early adopters of a technology-mediated experiment. However, the number of survivorship care planning researchers and practitioners using such measures is a finite community, and the GEM initiative achieved its goal of representing and joining stakeholders within that joint research/practice community. Further, there are many clinicians who participated in the GEM initiative who would not normally be present in research discussions and researchers who might not be privy to clinical discussions about evaluation of care planning. Current means of measuring participation in the GEM platform are not adequately sophisticated to assess all forms of involvement and specifically to distinguish those who rate items from those who might consume information but not contribute. However, by changing the paradigm [21] for how scientists and practitioners work together socially—from processes that were at one time linear and isolated to processes that are concurrent and connected—we hope to begin stimulating the cancer care planning research community to work together toward goals of greater synchronization in pursuit of IOM objectives.

We recognize that availability of the GEM infrastructure alone is insufficient to realize the scientific collaboration and data harmonization outcomes we ultimately envision. An explicit incentive structure does not currently exist to support the use of common measures and sharing of harmonized data. The National Institutes of Health has begun to create incentives through initiatives such as Big Data to Knowledge (BD2K) [36] and the promotion of the use of common data elements through a National Library of Medicine portal: http://www.nlm.nih.gov/cde/. However, without clear structural and fiscal incentives to use shared measures, participation in these types of initiatives may continue to be modest. Barriers to data harmonization cannot be overcome from a social media perspective alone but will require changes in incentives from journals, funders, and academia and perhaps from health policy makers, in the context of health care reform.

CONCLUSION

GEM-CP has utilized innovative technology and user platforms to capitalize upon this unique moment in the evolution of survivorship care planning science and practice. Having conducted a preliminary evaluation of GEM-CP, we now pose the important next-step question: how can these results be used to promote the coordinated development and meaningful use of scientific evidence on care planning? In concert with a conceptual framework [13] and a recently released NCI funding opportunity announcement in this area, the GEM-Care Planning Initiative provides an opportunity for scientists and clinicians to leverage their respective expertise to advance the field of survivorship care planning. Using technology-mediated social participation, GEM-CP also has the potential to achieve the dual aims of data harmonization and development of an evidence base relevant to both the scientific and practice communities. GEM and other such platforms present novel approaches to informing the ways in which we measure and improve the quality of follow-up care for cancer survivors, and ultimately reduce the burden associated with cancer survivorship.