Metrics to Evaluate Treatment Summaries and Survivorship Care Plans: A Scorecard

There are now approximately 13.7 million cancer survivors in the US [1], and this number will increase due to improvements in diagnosis and treatment and the aging of the population [2, 3]. Not only are cancer survivors at risk for recurrence of their primary cancer, but complex, multimodality treatments place them at risk for long-term and late effects such as secondary malignancies, cardiovascular disease, endocrine disorders [4], and general symptom distress [5–8]. These factors have stimulated a focus on survivorship care and long term follow-up, where gaps in care and fragmentation have been identified as problems [9]. To guide remediation of these difficulties, the 2006 Institute of Medicine (IOM) report [10] identified four components of survivorship care as “essential” and outlined ten recommendations for care that describe a range of activities to improve outcomes. More than six years later, however, cancer centers and survivorship programs continue to struggle to accomplish these recommendations, and systematic evaluations have been few [11–15].

A key IOM recommendation was the provision of treatment summaries (TS) and survivorship care plans (SCPs) for patients completing primary treatment [10] to facilitate transition to post-acute care and improve coordination of services [9, 16]. Other groups, such as the President’s Cancer Panel, the Centers of Disease Control, American Society of Clinical Oncology, and the LIVESTRONG™ Foundation have also called for TSs and SCPs to be provided to patients completing primary therapy [10, 17, 18], and the American College of Surgeons Commission on Cancer (COC) has mandated that accredited institutions make these available by 2015 [19, 20]. TS and SCP documents outline disease and treatment exposures and provide education regarding long-term and late effects, resource information, and a comprehensive follow-up plan for medical and psychosocial care. Numerous TS and SCP templates are in use, including the LIVESTRONG™ Care Plan powered by Penn Medicine’s Oncolink (livestrongcareplan.org), Journey Forward (journeyforward.org), What’s Next: Life After Cancer (www.cancer.org/acs/groups/content/documents/document/acspc-025795.pdf), Prescription for Living (tiny.cc/SFA8e), ASCO treatment summaries and survivorship care plans (www.asco.org), and various institutionally created models. The number and variety of these templates is likely to expand as centers begin implementing care plans in earnest, yet there has been little attention paid to evaluating the quality of these templates or the outcomes they achieve [11, 12, 21–23].

Along with mandates to provide TSs and SCPs [19, 20], there are also recommendations concerning the content to be included. The most comprehensive of these recommendations has come from the IOM [10] (Table 1) and has been adopted by the COC [20] as a minimum standard. The IOM recommendations, however, are broadly defined and non-specific, providing a general frame for a range of treatment settings, cancer sites, treatments, follow-up plans, and long-term and late effects. This lack of specificity makes it difficult to ascertain whether recommended content is actually being addressed, compare outcomes across document templates, or develop an integrated research literature that will allow for evidence-based decisions regarding which elements are truly essential to improving outcomes.

Table 1.

IOM Recommendations for content of the Survivorship Care Plan

Upon discharge from cancer treatment, including treatment of recurrences, every patient should be given a record of all care received and important disease characteristics. This should include, at a minimum: Diagnostic tests performed and results. Tumor characteristics (e.g., site[s], stage and grade, hormone receptor status, marker information). Dates of treatment initiation and completion. Surgery, chemotherapy, radiotherapy, transplant, hormonal therapy, or gene or other therapies provided, including agents used, treatment regimen, total dosage, identifying number and title of clinical trials (if any), indicators of treatment response, and toxicities experienced during treatment. Psychosocial, nutritional, and other supportive services provided. Full contact information on treating institutions and key individual providers. Identification of a key point of contact and coordinator of continuing care.

Upon discharge from cancer treatment, every patient and his/her primary health care provider should receive a written follow-up care plan incorporating available evidence-based standards of care. This should include, at a minimum: The likely course of recovery from treatment toxicities, as well as the need for ongoing health maintenance/adjuvant therapy. A description of recommended cancer screening and other periodic testing and examinations, and the schedule on which they should be performed (and who should provide them). Information on possible late and long-term effects of treatment and symptoms of such effects. Information on possible signs of recurrence and second tumors. Information on the possible effects of cancer on marital/partner relationship, sexual functioning, work, and parenting, and the potential future need for psychosocial support. Information on the potential insurance, employment, and financial consequences of cancer and, as necessary, referral to counseling, legal aid, and financial assistance. Specific recommendations for healthy behaviors (e.g., diet, exercise, healthy weight, sunscreen use, immunizations, smoking cessation, osteoporosis prevention). When appropriate, recommendations that first-degree relatives be informed about their increased risk and the need for cancer screening (e.g., breast cancer, colorectal cancer, prostate cancer). As appropriate, information on genetic counseling and testing to identify high-risk individuals who could benefit from more comprehensive cancer surveillance, chemoprevention, or risk-reducing surgery. As appropriate, information on known effective chemoprevention strategies for secondary prevention (e.g., tamoxifen in women at high risk for breast cancer; aspirin for colorectal cancer prevention). Referrals to specific follow-up care providers (e.g., rehabilitation, fertility, psychology), support groups, and/or the patient’s primary care provider. A listing of cancer-related resources and information (e.g., Internet-based sources and telephone listings for major cancer support organizations).

SOURCE: Adapted from the President’s Cancer Panel (2004)

Reprinted with permission from Hewitt et al., (2006) by the National Academy of Sciences, Courtesy of the National Academies Press, Washington, D.C.

One approach to remedying this situation is to develop standardized metrics to quantify the information being provided across the various TSs and SCPs. The objective of such work is not to mandate what information should be in a care plan, but rather to allow clinicians and researchers a means of describing what is being presented to survivors with reference to some agreed upon set of criteria. This, in turn, should foster implementation efforts by allowing assessment of content and fidelity. Similarly, it provides a means of describing and reconciling a diverse literature in which numerous TS/SCP documents are being promulgated. The goal of the current project was to develop cost-effective metrics to assess concordance between IOM recommendations for content and the TSs and SCPs being provided to breast cancer survivors, and to demonstrate that these metrics can be scored reliably across raters as a first step in the process of construct validation. Breast cancer survivors were chosen as the initial group for metric development since they constitute the largest and most well-characterized group of cancer survivors in the US [2] at approximately 2,971,610 in 2012.

Method

Operationalizing IOM Recommendations

This project required the creation of two checklists to operationalize the IOM recommendations for TS/SCP content. The multidisciplinary study team included a medical oncologist (AD), oncology nurses and nurse practitioners (LJ, CS, AJ), and clinical health psychologists (SP, BR). As a first step, all IOM recommendations listed in Table 1 were deconstructed and mapped onto potential disease characteristics, treatment exposures, and follow-up options by study leaders (SP & CS). For example, the IOM TS item 2, “Tumor Characteristics” was deconstructed into items reflecting the reporting of definitive breast cancer diagnosis, laterality, staging, histologic or nuclear tumor grade, estrogen receptor status, progesterone receptor status, HER2/neu status, and status of any lymph nodes dissected.

Content validity refers to the degree to which the elements of an instrument are relevant to and reflective of the content of a given construct used in a specific context [24]. This includes such aspects as presentation, scoring, ordering, and instruction. In the current project, the instrument being developed possesses content validity to the extent that the items represent the IOM recommendations concerning TS/SCP to be provided to breast cancer patients. As an initial attempt to establish content validity, remaining team members, representing a range of subject matter experts, were presented with the items for comment on completeness, clarity, and comprehensiveness with respect to IOM recommendations. This exposure, comment, and revision cycle was repeated in an iterative fashion until 100% consensus that items represented all aspects of the IOM recommendations was reached was reached. Following the creation of items, a similar iterative process was undertaken regarding assignment of items to overarching domains.

As a next step in establishing content validity, decision rules were established to allow for scoring. Again, exposure, comment, and revision cycles by subject matter experts were performed until consensus was reached. Items were determined to be dichotomous and reflect the presence or absence of information in the TS or SCP documentation. Items were considered present if noted in the TS/SCP or if the TS/SCP provides an explicit field to list a relevant piece of information (e.g., a field is devoted to clinical trials information) and this was noted as being non-applicable. Items were considered absent if content was not described in the TS/SCP or if an explicit field was present but not completed. Thus, non-applicable information could be scored as present so long as it was noted that it was not applicable for a given survivor. Accuracy of noted information was not taken into account. Items were weighted equally to derive a total score.

Procedure for Reliability Assessment

Once consensus was reached concerning item construction and scoring, and following approval by all relevant Institutional Review Boards, 13 clinical sites were recruited to generate a relevant pool of TS/SCP materials for rating. These included seven LIVESTRONG™ Foundation Survivorship Centers of Excellence, each an NCI-designated Comprehensive Cancer Center engaged in providing survivorship care and TS/SCP materials to breast cancer survivors. To ensure heterogeneity in type of setting and materials received for rating, we also recruited six community-based centers associated with the Centers of Excellence. Community and public hospitals were the most common type of community site, with the remaining sites representing a community health center, a university-based cancer treatment clinic, and a multi-specialty group practice. Further details about participating sites are reported elsewhere [11].

Materials to be rated to establish reliability consisted of the last five TSs and SCPs and accompanying resources (e.g., pamphlets, referrals, resource lists) that were provided at each of the 13 sites in the preceding calendar year (n=65 TSs and n=65 SCPs total). All materials were de-identified to maintain patient confidentiality. Accompanying materials (e.g., pamphlets) as well as specific TS/SCP documents were rated in order to capture all written information received by survivors. There was heterogeneity in the materials to be rated. Most sites (62%) provided patients with institutionally-developed TSs, while 23% used the LIVESTRONG™ Care Plan, and 15% used Journey Forward to create the TS document. Institutionally-developed SCPs were used by 31% of participating sites, and 46% used the LIVESTRONG™ Care Plan alone or in conjunction with additional materials. The Journey Forward care plan was used by 15% and the ASCO care plan by 8%, also in conjunction with supplementary materials.

These TS and SCP materials were then rated by two independent raters (CS & SP) using the concordance tools described above. Interrater reliability was calculated across items within each TS and SCP using Cohen’s kappa. Total scores for each TS and SCP were calculated and intraclass correlations (ICC) using a two-way mixed method were calculated as an index of reliability for absolute agreement at the document level.

Results

The Scorecard Instruments

The approach described above resulted in two scorecards: a 60-item tool assessing Treatment Summaries and a 32-item tool assessing Survivorship Care Plans, collectively called the Survivorship Care Plan Assessment Checklist. The TS Concordance Tool covered 13 content domains, while the SCP Concordance Tool covered information across 10 domains. Items are scored as positive if: 1) information corresponding to a given item is present or 2) a dedicated space (e.g., text box) for such information is present regardless of its instantiation. Tables 2 and 3 provide the scorecard instruments themselves, domains assessed, and additional scoring instructions.

Table 2.

Scorecard - Treatment Summary Items

Information Domain	Item	Descriptor
Diagnosis	1	What diagnostic tests were performed? (e.g., MRI, CAD, FFDM)
	2	When was definitive diagnosis made by biopsy?
Staging & Tumor Characteristics	3	What is the definitive diagnosis (e.g., DCIS, Invasive/Infiltrating Ductal Carcinoma, Inflammatory Breast Cancer)
	4	Is laterality noted?
	5	Is either AJCC staging or Stage Grouping noted?
	6	Is Histologic or nuclear Tumor Grade noted?
	7	Is the tumor Estrogen Receptor positive or Estrogen Receptor negative?
	8	Is the tumor Progesterone Receptor positive or Progesterone Receptor negative?
	9	Is the tumor HER2 positive or HER2 negative?
	10	Were any nodes positive?
Surgery Details	11	Was surgery performed?
	12	What type of surgery was performed (e.g., Lumpectomy, Mastectomy)
	13	Was lymph node dissection performed?
	14	Was Axillary or Sentinel (or both) Lymph Node Dissection Performed?
	15	How many nodes were dissected?
	16	When was surgery performed?
	17	Was reconstruction performed?
	18	When was reconstruction performed?
Chemotherapy Details	19	Was chemotherapy given?
	20	Was chemotherapy adjuvant or neoadjuvant? (Note: must be stated; do not infer from dates)
	21	When did chemotherapy start and stop?
	22	Are full (generic or brand) names of all chemotherapy drugs noted?
	23	How many cycles were given?
	24	What was route of administration? (Note: must be stated; do not infer)
	25	Was a dose reduction required?
	26	What was the total dose for each agent administered?
Radiotherapy Details	27	Was XRT administered?
	28	Was XRT External Beam Radiation or Brachytherapy?
	29	Was subtype of EBR or Brachytherapy provided? (i.e., EBR: accelerated, partial, 3D conformational; Brachytherapy: intracavitary, interstitial, other.)
	30	Was radiotherapy to chest wall or breast and nodal XRT or not.
	31	To which side was radiotherapy?
	32	Are Start and Stop Dates for XRT provided?
	33	Is the total XRT dose received provided?
Targeted Therapy Details	34	Was Targeted therapy provided?
	35	What type of targeted therapy was provided?
	36	Are start-stop (or continuing) dates provided?
Hormonal Therapy Details	37	Was hormone therapy provided?
	38	What type of hormone therapy was provided?
	39	Are start-stop (or continuing) dates provided for all hormone therapies?
Treatment Toxicities	40	Were any toxicities or complications of any therapy noted and if so, what were they?
Clinical Trial Information	41	Was the patient on a clinical trial?
	42	What is the Title and Number of the clinical trial?
Genetic Testing Details	43	Was Genetic Testing Performed?
	44	When was Genetic Testing Performed?
	45	What were the results of Genetic Testing?
Supportive Therapy Details	46	Was supportive psychosocial, nutritional, or other supportive therapy provided?
	47	What supportive therapy was provided?
	48	Are start-stop (or continuing) dates provided?
Contact Information for Primary Treatment Team Members	49	Who performed the surgery?
	50	Is complete contact information for the surgeon provided? (e.g., Address, phone number)
	51	Who performed XRT?
	52	Is complete contacted information for the radiologist provided? (e.g., Address, phone number)
	53	Who performed supportive therapy?
	54	Is complete contact information for the therapist provided? (e.g., Address, phone number)
	55	Is a primary oncology treatment provider identified?
	56	Is complete contact information for this provider given? (e.g., Address, phone number)
Care Coordination Details	57	Is a key contact for oncology treatment identified?
	58	Is complete contact information for the key contact provided? (e.g., Address, phone number)
	59	Is a care coordinator identified?
	60	Is complete contact information for the care coordinator provided? (e.g., Address, phone number)

Scoring: Each item is worth 1 point, maximum. Items can receive a point in two ways: a) Score 1 point for each item if present and detailed in the TS or accompanying materials. b) Score 1 point if TS or accompanying materials provide a space for information regardless of completeness of document provided to patient. Please note additional scoring instructions for items 20, 24. Total possible = 60 points.

Table 3.

Scorecard - Survivorship Care Plan Items

Information Domain	Item	Descriptor
Potential Toxicities and Late Effects	1	Are any treatment toxicities (Note: actual or possible) identified?
	2	Is the expected course (Note: critical component = time) of recovery from toxicities identified?
	3	What possible long-term and late effects could result from treatments received?
	4	What are symptoms from these possible long-term and late effects of treatment?
Breast Cancer Surveillance	5	What breast cancer-specific surveillance testing is recommended?
	6	How frequently should recommended breast cancer-specific surveillance testing be performed? (Note: 1 point maximum given for any test recommended)
	7	What provider is responsible for ordering/providing BC-specific surveillance testing?
Non-breast Cancer Surveillance	8	What non-BC cancer surveillance testing (e.g., colonoscopy, PAP) is recommended? (Note: 1 point maximum given for any test recommended)
	9	How frequently should recommended non-BC cancer surveillance testing be performed?
	10	What provider is responsible for ordering/providing non-BC cancer surveillance testing?
Non-Cancer Surveillance	11	What non-cancer surveillance testing (e.g., lipid profiles) is recommended? (Note: 1 point maximum given for any test recommended)
	12	How frequently should recommended non-CA surveillance testing be performed?
	13	What provider is responsible for ordering/providing non-CA surveillance testing?
Signs & Symptoms of Cancer	14	What are possible signs of breast cancer recurrence?
	15	What are possible signs of second cancers? (Note: 1 point maximum given for any sign/second cancer noted)
Potential Psychosocial Effects	16	What are the possible effects of BC on the marital/partner relationship (not incl. sexual functioning)?
	17	What are the possible effects of BC on sexual functioning?
	18	What are the possible effects of BC on work/employment?
	19	What are the possible effects of BC on parenting?
	20	What are the possible effects of BC on insurance?
	21	What are the possible effects of BC on finances?
	22	What is the potential future need for psychosocial support?
Referral Information	23	What are potential referral sources for counseling?
	24	What are potential referral sources for legal aid?
	25	What are potential referral sources for financial assistance?
	26	Are referrals to specific follow-up care providers given? (e.g., PCP, rehabilitation, fertility, support groups) (Note: point awarded only for referrals to specific providers or services)
	27	Is a list of cancer-related resources and information (e.g., Internet-based, major cancer support organizations) provided?
Prevention/Health Promotion	28	What are specific recommendations for healthy behaviors (e.g., physical activity, diet, weight, immunizations, etc.)
	29	Is information on known effective chemoprevention strategies for secondary cancer prevention discussed? (i.e., tamoxifen in women post ipsilateral mastectomy; aspirin for colorectal cancer prevention)
Genetic Testing Recommendations	30	Is information provided on who should consider genetic counseling and testing?
Familial Cancer Risk	31	Are recommendations provided on what information first degree relatives should be given about their cancer risk?
	32	Are recommendations provided regarding what specific cancer screening tests first degree relatives should undergo?

Scoring: Each item is worth 1 point, maximum. Items can receive a point in two ways: a) Score 1 point for each item if present and detailed in the SCP or accompanying materials. b) Score 1 point if SCP or accompanying materials provide a space for information regardless of completeness of document provided to patient. Please note additional scoring instructions for items 1, 2, 6, 8, 11, 15, 26. Total possible = 32 points.

Reliability

Overall, raters were in agreement concerning presence or absence of TS information 82% of the time across the 65 sets of documents. The mean interrater reliability (kappa) across the 65 TS ratings was 0.76 (SD = 0.12; 95% CI = 0.73–0.79) indicating ‘substantial agreement’ [34], between raters. Kappas ranged from 0.42 to 0.97 within the sample TSs. Within the sample of 65 SCP documents, raters were in agreement concerning presence or absence of SCP information 90% of the time. Mean kappa again indicated substantial agreement (M = 0.66, SD = 0.16; 95% CI = 0.62–0.70) with a range from 0.33 to 1.00. Intraclass correlation based on total score across documents was similarly high. Within TS materials, ICC = 0.85 (CI = 0.76 – 0.91, p < .001), while within SCP materials, ICC = 0.75 (CI = 0.62–0.84, p < 0.001). These results are consistent with “strong” (SCP) to “almost perfect” (TS) agreement. [25 ].

Discussion

The 2006 IOM report [10] recommended standards for the information to be included in TS and SCP documents, and the CoC has mandated that these documents be part of the basis for accreditation by 2015 [20]. Despite this, very little work has been performed to outline means of ensuring that these standards are met [11, 12]. This is unfortunate, as cancer survivors often have low confidence in their ability to navigate survivorship [26], and SCPs may prove to be one means of improving this and other relevant outcomes in post-treatment cancer survivors. Moreover, despite mandates for implementation [19] there has been little investigation concerning the efficacy or effectiveness of these interventions [21–23], and research to date has focused more on reach, uptake, and qualitative analysis of stakeholder preferences for content and delivery than on the outcomes achieved by patients [21, 22, 27–35]. Indeed, the IOM [10] (p. 154) states that “such plans (have) not yet been formally evaluated. Despite the lack of evidence to support the use of (SCPs)… some elements of care simply make sense that is, they have strong face validity...” Others [36] have noted the similarity between care plans and hospital discharge summaries, noting that these were (and are) being implemented prior to a literature demonstrating effects on outcomes accumulated.

This project developed two instruments for assessing the degree to which TS and SCP materials are in concordance with IOM recommendations. As demonstrated, ratings can be made with a high degree of interrater reliability and agreement. These characteristics enable use of these instruments for clinical, quality improvement, and research purposes. With practice, these instruments can be used reliably to evaluate the degree to which the materials provided to breast cancer survivors mirror the recommendations of the IOM. Our hope is that this will, in turn, foster implementation research and allow for greater ease in reconciling a diverse literature to enable systematic review and meta-analysis as the literature grows. For example, we have used these instruments to examine the degree to which LIVESTRONG™ Survivorship Centers provide TS/SCP materials that are in concordance with the IOM recommendations [11]. Our findings suggested that even in settings with dedicated staff and funding for such efforts, less than half of the recommended TS content and less than two-thirds of the recommended SCP content, on average, were included in materials delivered to breast cancer survivors. Some information was commonly provided (e.g., possible toxicities and late effects). Other information, however, (e.g., familial cancer risk and documentation of which provider is responsible for routine cancer surveillance) was rarely provided and this included areas that both survivors and providers have reported to be highly desirable [27, 29, 33, 37, 38]. These results have enabled members of the LIVESTRONG™ Network to target areas for improvement, demonstrating the potential of the scorecard for use in quality improvement efforts. In addition, these findings have facilitated the development of a standardized TS and SCP for use in an ongoing study exploring outcomes of care plans delivered to breast cancer survivors.

As they currently stand, the IOM recommendations for TS/SCP content are comprehensive, broad, and likely to be over- rather than under-inclusive. Given that completion of such plans is not currently a reimbursable activity and that the resources required to complete these plans is noted as a primary barrier to implementation [11, 12, 16, 29, 39, 40], less comprehensive and detailed care plans are likely to become the norm in practice. Thus, determination of and consensus about what constitutes the ‘essential elements’ to be included in a plan of care should become a priority. Instruments such as this can play a crucial role in describing informational content, allowing comparisons across different sets of materials, and will ultimately be essential for linking particular elements of care plans to demonstrated outcomes.

Others have used similar ‘scorecard’ methods to describe the content of TS/SCPs relative to IOM recommendations across differing institutes and populations [12]. Although findings from these authors are quite similar to those we have presented elsewhere [11], the tools used [12] were neither specific to the populations of interest (i.e., both breast and colorectal cancer documents were rated using the same instruments), nor were psychometric data reported. In addition, rather than examining materials presented to survivors, blank templates and “deidentified or hypothetical completed” materials were examined. These limitations were overcome in the current project.

Limitations

The use of the IOM recommendations as the basis for concordance may lead some to assume that these are either empirically-based or the ‘gold standard’ against which all TS/SCPs must be evaluated. Rather, the IOM recommendations are consensus-based, but the most comprehensive recommendations currently available, and the standard against which accreditation will be assessed [20]. As such, they provide an excellent starting point from which to begin the discussion of which elements of care plans are essential to improving outcomes. Another limitation concerns the assessment of reliability in the current project. All documents were rated by study team members primarily responsible for the creation of the rating instruments. This familiarity with the materials may have somewhat inflated reliabilities, and we are currently working to replicate psychometric findings using naïve but trained raters to better approximate clinical reliabilities. As well, presence or absence of information concordant with IOM recommendations is what is being scored. Accuracy or depth of content cannot be assessed using these methods, and this should certainly be examined in future work. Finally, the current study focused on reliability of the scorecard instruments themselves. Future work will need to examine the degree to which given items or domains affect patient outcomes such as health behaviors, knowledge concerning treatment and late effects, and communication between patients and providers.

Conclusion

The IOM, the COC, and others are moving the provision of TS/SCP materials to cancer survivors forward and mandating the content to be addressed within them. Implementation and outcomes research are lagging behind these mandates, and tools are needed to facilitate progress. This project was conducted in direct response to this need and offers two easy to use and highly reliable tools for assessing the degree of concordance between TS/SCP materials and these content areas. We are working to develop similar metrics with analogous content domains that can be applied to other specific cancer populations. Use of tools such as these can not only foster improved implementation efforts with a high degree of fidelity, but allow for meaningful comparisons across differing documents, samples, populations, and institutions so that decisions concerning what is essential in care and care planning can move toward being empirically-based.