Harmonized Outcome Measures for Use in Non–Small Cell Lung Cancer Patient Registries and Clinical Practice

Abstract

Background:

Lung cancer is the leading cause of cancer-related death in the United States and globally, and many questions exist about treatment options. Harmonizing data across registries and other data collection efforts would yield a robust data infrastructure to help address many research questions. The purpose of this project was to develop a minimum set of patient and clinician relevant harmonized outcome measures that can be collected in non–small cell lung cancer (NSCLC) patient registries and clinical practice.

Methods:

Seventeen lung cancer registries and related efforts were identified and invited to submit outcome measures. Representatives from medical specialty societies, government agencies, health systems, health information technology groups, patient advocacy organizations, and industry formed a stakeholder panel to categorize the measures and harmonize definitions using the Agency for Healthcare Research and Quality’s supported Outcome Measures Framework (OMF).

Results:

The panel reviewed 66 outcome measures and identified a minimum set of 8 broadly relevant measures in the OMF categories of patient survival, clinical response, events of interest, and resource utilization. The panel harmonized definitions for the 8 measures through in-person and virtual meetings. The panel did not reach consensus on 1 specific validated instrument for capturing patient-reported outcomes. The minimum set of harmonized outcome measures is broadly relevant to clinicians and patients and feasible to capture across NSCLC disease stages and treatment pathways. A pilot test of these measures would be useful to document the burden and value of the measures for research and in clinical practice.

Conclusions:

By collecting the harmonized measures consistently, registries and other data collection systems could contribute to the development research infrastructure and learning health systems to support new research and improve patient outcomes.

Background

Lung cancer remains the leading cause of cancer death in the United States and worldwide is the most common cancer in both incidence and mortality, with an estimated 1.8 million deaths annually.^1,2 In 2021, lung cancer will account for an estimated 12% of new cancers in the United States.¹ Non–small cell lung cancer (NSCLC) accounts for approximately 85% of all lung cancers. Although lung cancer deaths have decreased in recent years, a lung cancer diagnosis has one of the lowest 5-year relative survival rates.³

The rapid and significant changes in NSCLC diagnosis and treatment have introduced many pressing questions about, for example, which subpopulations would most benefit from screening, how and when to combine immunotherapy with chemotherapy, and which patients are unlikely to receive clinical benefit from immunotherapy. To address these and other research questions critical to improving patient outcomes, innovative research strategies and high-quality sources of data on the outcome measures that are most important to patients and clinicians are needed.

Many registries already capture consistent, longitudinal, observational data on patients with NSCLC to meet a wide range of purposes, from disease surveillance to quality improvement to clinical research. A patient registry is defined as “an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure, and that serves one or more predetermined scientific, clinical, or policy purposes.”⁴ A cancer registry is defined as “an information system designed for the collection, storage, and management of data on persons with cancer.”⁵

Yet, it is currently difficult to link, aggregate, or compare data across existing patient and cancer registries to support new research, because registries often capture different outcome measures. Even when registries capture the same outcome (eg, progression), they often define the outcome differently, reflecting the lack of consensus on outcome measure definitions across medical specialties and across research and clinical practice. Variation in outcome measures limits the ability of registries to serve as data infrastructure for new research and reduces their value as building blocks for learning healthcare systems.⁶

To reduce variation and improve the utility of registry data, the US Department of Health & Human Services, led by the Agency for Healthcare Research and Quality and in collaboration with the FDA and the National Library of Medicine, has supported the development of the Outcome Measures Framework (OMF). The OMF is a conceptual model for classifying and defining outcomes in a standard manner for a broad range of conditions.⁷

Our goal was to use the OMF as a content model for developing a minimum set of standardized outcome measures for use in NSCLC patient registries and clinical practice; this effort was part of a broader effort to develop minimum measure sets in 5 clinical areas.^8–10 The objectives were to: (1) test the utility of the OMF for categorizing lung cancer outcomes and for supporting harmonization of outcomes across treatment modalities, (2) identify a minimum set of outcome measures that could be captured in NSCLC patient registries and clinical practice, (3) agree on harmonized definitions for each outcome in the minimum measure set, and (4) map the harmonized definitions to standardized terminologies to support consistent implementation and collection of the outcome measures within electronic health record systems (EHRs).

Methods

Outcome measures currently collected in lung cancer registries, quality improvement efforts, other observational studies, public health surveillance initiatives, and clinical practice were included in this harmonization effort. The harmonization methodology is described in a related publication⁸ and summarized here. Existing lung cancer registries were identified through systematic searches of the Registry of Patient Registries,¹¹ ClinicalTrials.gov,¹² the published medical literature using PubMed and Google Scholar, and relevant websites using Google. Identified registries meeting definitional criteria for a patient outcomes–focused registry⁴ and collecting data in the United States were invited to participate as voluntary members of the registry workgroup; the registry workgroup also included thoracic surgeons, medical oncologists, and radiation oncologists as clinical experts in NSCLC treatment and measurement of outcomes. To provide a broader perspective, a multistakeholder panel was formed to review the work of the registry workgroup.

Participating registries submitted outcome measure specifications for workgroup review; the workgroup also reviewed definitions from related harmonization efforts.¹³ Through a series of virtual and in-person meetings, the workgroup categorized the measures using the OMF categories of survival, clinical response, events of interest, patient-reported outcomes (PROs), resource utilization, and experience of care, and defined a minimum measure set. The purpose of the minimum measure set is to describe a core set of outcomes that could be collected across all NSCLC registries and in routine clinical practice. For each measure in the minimum set, the workgroup reviewed existing definitions, identified and discussed the clinical significance of measure differences, and discussed how to harmonize the definition.

The workgroup recommended harmonized definitions and then met with the stakeholder group to discuss the recommendations and reach consensus, where possible. As a final step, clinical informaticists translated the narrative definitions to standardized terminologies to support implementation of the definitions within EHRs. After a public comment period, the minimum measure set was finalized.

Results

A total of 17 registries were invited to participate, and 11 agreed to participate in the registry workgroup (Table 1). Registries that declined to participate are described in supplemental eTable 1 (available with this article at JNCCN.org). The registry workgroup also included 3 clinical experts in the treatment of NSCLC, 1 biostatistician, and representatives from NCCN who provided expertise in cancer registry design and data analysis.

Table 1.

Participating Initiatives/Registries

Initiative/Registry Name (ClinicalTrials.gov Identifier)	Sponsoring Organization	Primary Purpose
SBRT vs Surgery in High Risk Patients With Early Stage Lung Cancer (NCT02562027)	Washington University School of Medicine	To create a validated risk model for treatment selection, with the goal of enhancing the ability to counsel patients regarding their specific risks/benefit ratio for surgery or SBRT.
RSSearch Patient Registry	The Radiosurgery Society	To track SRS/SBRT utilization, treatment practices and outcomes to help determine, over time, the most effective use of these systems in management of patients with life threatening tumors and other diseases.
Cancer Experience Registry	Cancer Support Community	To better understand the psychosocial experiences and needs of people who have been impacted by cancer, including patients, survivors, and caregivers.
TAPUR Study	ASCO	To describe the safety and efficacy of commercially available, targeted anticancer drugs prescribed for treatment of patients with advanced cancer that has a potentially actionable genomic variant.
PANORAMA-Real World Molecular Testing, Treatment Patterns, and Clinical Outcomes EGFR Mutation-Positive NSCLC(NCT02777658)	AstraZeneca	To assess molecular testing, treatment patterns, and associated outcomes among patients with EGFR mutation–positive locally advanced or advanced NSCLC who have progressed on or after EGFR-TKI therapy post availability of a third-generation TKI.
Prospective Study to Determine Impact of Early Palliative Care Consult on QoL, Cancer Related Symptoms in Advanced Lung Cancer Patients: Thoracic Pilot Project (NCT02459002)	MD Anderson Cancer Center	To determine the differences in participant outcomes including QoL; symptom distress; and caregiver outcomes at week 12 in patients with advanced lung cancer receiving early palliative care consultation versus those who do not.
STS National Database	The Society of Thoracic Surgeons	To collect outcomes data on cardiothoracic surgery patients to improve quality of care and patient outcomes.
Project GENIE	American Association for Cancer Research	To support precision oncology through the development of a regulatory-grade registry aggregating and linking clinical-grade cancer genomic data with clinical outcomes from cancer patients treated at participating institutions.
National Program of Cancer Registries	CDC	To collect data on cancer occurrence (including the type, extent, and location of the cancer), the type of initial treatment, and patient outcomes.
SEER Program	NCI	To provide information on cancer statistics in an effort to reduce the cancer burden among the US population.
ORIEN Lung Cancer Study	H. Lee Moffitt Cancer Center and Research Institute	To assess the real-world patient experience by evaluating the patients’ QoL, treatment toxicities, and clinical measures over a 6-month period.

Abbreviations: GENIE, Genomics Evidence Neoplasia Information Exchange; NSCLC, non–small cell lung cancer; ORIEN, Oncology Research Information Exchange Network; QoL, quality of life; SBRT, stereotactic body radiation therapy; SRS, stereotactic radiosurgery; STS, Society of Thoracic Surgeons; TAPUR, Targeted Agent and Profiling Utilization Registry; TKI, tyrosine kinase inhibitor.

Eight stakeholder organizations participated, representing patient organizations (Lung Cancer Alliance, American Lung Association, Lung Cancer Research Foundation, LUNGevity Foundation), health information technology (Flatiron Health), and federal agencies (FDA, NCI, and National Library of Medicine).

A total of 66 outcome measures were collected and categorized using the OMF. The greatest number were categorized as PROs (n=30), followed by clinical response (n=10), resource utilization (n=9), events of interest (n=8), survival (n=6), and experience of care (n=3).

Eight measures make up the minimum set. Because the measure set is intended for broad use across registries and clinical practice, the workgroup considered feasibility, relevance, and burden of collection and reporting when recommending measures. Supplemental measures are encouraged to address specific purposes. Table 2 lists measure definitions; the rationale for selection of the measures and definitions is described in the following sections.

Table 2.

NSCLC Minimum Measure Set and Harmonized Definitions

OMF Category	Outcome Measure	Definition
Survival	Overall survival	Overall survival (collect date of diagnosis and cause of death, if available).
Survival	30-d mortality	All deaths within 30 days of treatment (surgery, radiation, or systemic treatment).
Survival	Progression-free/disease-free survival	Progression-free/disease-free survival, where feasible (see definition of progression and recurrence in “Clinical Response,” below).
Clinical response	Progression and recurrence	Progression and recurrence should be measured using RECIST (see below) OR clinician documentation of progression/recurrence OR change in therapy due to progression/recurrence.
		RECIST guideline13 definition of progressive disease:  • At least a 20% increase in the sum of diameters of target lesions, taking as reference the smallest sum on study (this includes the baseline sum if that is the smallest on study). In addition to the relative increase of 20%, the sum must also demonstrate an absolute increase of at least 5 mm.  • The appearance of one or more new lesions is also considered progression.
		Notes:  • RECIST cannot be used to measure radiated lesions.  • The sites and numbers of metastases must be captured.  • The date of progression, defined as the date of the source documentation recording progression or change in lesion, must be captured.
		The above is a brief summary of RECIST 1.1. The actual cited document governs the actual decisions regarding radiologic response or progression, including special considerations for lymph nodes as target lesions. In addition, RECIST 1.1 does not consider symptomatic deterioration or other aspects of clinical progression.
		Further work is needed to recommend a consistent approach to evaluation of radiated lesions.
		Future efforts may also consider use of iRECIST in the context of immunotherapies, although pseudo-progression is sufficiently rare that a separate measure is not recommended for the minimum measure set.
Clinical response	Change in performance status	Change in performance status, as measured using the ECOG performance status or the Karnofsky performance scale index.
Events of interest	Toxicity: major complications	Major complications:  • Surgical complications, defined as one or more of the following:   ► Pneumonia   ► Acute respiratory distress syndrome   ► Bronchopleural fistula   ► Pulmonary embolus   ► Initial ventilator support >48 h   ► Reintubation/Respiratory failure   ► Tracheotomy   ► Myocardial infarction   ► Unexpected return to operating room (any cause)
		• Radiation therapy complications:   ► CTCAE grade 3 or 4 complications due to radiotherapy
		• Systemic therapy complications:   ► CTCAE grade 3 or 4 complications due to systemic therapy
Events of interest	Toxicity: other complications	Other complications:  • Complications that do not meet the definition of major complications that resulted in change in treatment, change in dose, or schedule delay.
PRO	Collection of PROs that capture at least some of the important domains using one or more validated instruments is recommended	Important domains to consider in collecting PROs are:  • Symptoms  • Functioning (cognitive, physical)  • Role (ability to participate)  • Toxicity
		These domains should be captured, if feasible. However, patient and clinician burden is an important consideration, and registries may elect to collect a subset of these domains. Validated, publicly available PROs with strong psychometric properties should be used to capture these domains. To minimize burden, use of a tool that captures multiple domains is recommended.
		Frequency of PRO collection should depend on the stage of disease, treatment modality, and frequency of office visits/patient contact.
Resource utilization	Healthcare utilization	All resource utilization related to treatment of lung cancer.

Abbreviations: NSCLC, non–small cell lung cancer; OMF, Outcome Measures Framework; PRO, patient-reported outcome.

Survival

Three measures of survival (overall survival, 30-day mortality, and progression-free survival/disease-free survival) are included in the minimum measure set. Overall survival should be captured in all registries. Treatment-related mortality is an important concept to measure, but ascertaining cause of death can be difficult, particularly in observational data sources. Studies of surgical procedures often report all-cause mortality within 30 days of treatment. To align with registries that focus on surgical procedures, the workgroup included 30-day mortality in the minimum measure set, noting that this measure will capture deaths related to procedural complications as well as major acute toxicities related to systemic therapy. It should be noted that this measure includes all deaths whether attributed to the treatment or not and is not intended as a quality measure. Finally, progression-free survival/disease-free survival are included in the minimum measure set because these outcomes have been used for regulatory approval of new therapies in NSCLC, but there can be some limitations in real-world practice, such as loss to follow-up and inconsistent documentation of progression. Challenges related to progression and recurrence are discussed further herein. When capturing the 3 survival measures, the workgroup emphasized that registries should collect date of death and report clearly on efforts made to ascertain the outcomes of patients categorized as lost to follow-up.

A key component of measuring survival, as well as the other outcomes in the minimum measure set, is the recording of dates. Throughout the harmonization process, the workgroup emphasized the importance of recording dates for diagnosis, treatment(s), and outcomes. Dates play a critical role in determining the relationship of events to treatment and in calculating measures such as overall survival and progression/recurrence. In addition, time from diagnosis to treatment is correlated with patient outcomes.

Clinical Response

Progression and recurrence are widely used measures of clinical response and are included in the minimum measure set. In clinical trials, progression typically is measured using RECIST criteria or equivalent (eg, iRECIST), which focuses on changes in target lesions over time and development of new lesions.¹⁴ The RECIST criteria are relevant across specialties (surgeons, oncologists, radiation oncologists, radiologists) and are considered an objective standard for measuring progression. However, the RECIST criteria are difficult to apply retrospectively to existing data sources and have some limitations. For example, the RECIST criteria do not consider symptomatic deterioration or other aspects of clinical progression, and the criteria cannot be used for radiated lesions or osseous lesions without a soft tissue component. Recognizing these limitations, the harmonized definition for progression and recurrence allows for clinician documentation of progression/recurrence or a change in therapy due to progression/recurrence. In all cases, date of progression/recurrence and how progression/recurrence was documented should be recorded.

In addition, the workgroup noted that clinical response should not focus solely on change in the tumor(s). Changes in how the patient is feeling should also be captured as a measure of clinical response. Performance status, although subjective, is commonly used for this purpose, and change in performance status is included in the minimum measure set. However, the group cautioned that this information may not be recorded consistently in all care settings.

Events of Interest

Two events of interest are included in the minimum measure set. In defining major complications, the workgroup divided complications into 3 categories: surgical, radiation therapy (RT), and systemic therapy. RT and systemic therapy complications are defined using the CTCAE,¹⁵ which is widely used in cancer research. The workgroup did note that a limitation of the use of CTCAE is that it emphasizes laboratory-based markers, as opposed to patient-reported items. Registries should report the version number used, as the CTCAE is updated regularly. In contrast, surgical complications are not typically captured using the CTCAE. The workgroup recommended capturing surgical complications using the definition from the STS National Database, which was generated through expert consensus, or CTCAE when applicable.

In addition to major complications, the workgroup recommended capture of other complications that result in a change in treatment, change in dose, or schedule delays, noting that a toxicity may not be categorized as major according to CTCAE but may be sufficiently bothersome to the patient to result in treatment changes. When capturing this measure, it is important that documentation specifically link the change or delay to a specific complication, as changes or delays may occur for other reasons (eg, patient vacation).

Patient-Reported Outcomes

PROs proved to be the most challenging harmonization area for the workgroup, and the group did not reach consensus on a specific validated measure to include in the minimum measure set. The participating registries capture a variety of PROs that measure different domains. Some of the domains are broadly relevant in NSCLC treatment (eg, physical functioning), whereas others are specific to a treatment modality or patient population (eg, cough for patients receiving RT). The workgroup identified 4 important domains that are relevant across treatment modalities and should be considered when selecting PRO instruments: symptoms, functioning (cognitive, physical), role (ability to participate), and toxicity.

Many validated, publicly available instruments capture these domains, such as the Trial Outcome Index (TOI) of the Functional Assessment of Cancer Therapy-Lung (FACT-L),¹⁶ Lung Cancer Symptom Scale (LCSS),¹⁷ Patient-Reported Outcomes Measurement Information System-29 (PROMIS-29),¹⁸ and Edmonton Symptom Assessment Scale (ESAS).¹⁹ These instruments differ in number of domains that are captured, the time needed to complete, and the appropriateness for patients with differing stages of disease and undergoing different treatment modalities. Because the minimum measure set is intended to be broadly relevant, the workgroup concluded that it is feasible to recommend important domains, but the selection of the instrument is left to the researcher because there was not enough consensus to recommend a single measure.

Resource Utilization

In NSCLC, resource utilization includes hospitalizations, emergency department visits, procedures, medications, and office visits, and costs are highly variable across the stages of the disease. The workgroup recommended measuring all resource utilization related to treatment of lung cancer but noted that further work is needed in this area to ensure that resource utilization is captured and reported consistently across different registries.

Experience of Care

Although not a direct patient outcome, experience of care measures is important in NSCLC given the complex nature of the condition. The workgroup did not recommend a specific measure, but noted that, depending on the care setting and patient population of interest, collection of information on domains relevant to NSCLC, such as availability of resources to manage side effects and symptoms, financial burden of illness, timeliness of care, and goals of care, should be considered. More work is needed in this area to identify and recommend specific validated instruments and to examine the correlation of these domains with patient outcomes.

Characteristics and Treatments

In addition to harmonizing outcome measures, the workgroup identified characteristics of the patient, disease, and provider that are important to collect to support risk adjustment and relevant treatments and treatment intents (Figure 1). The workgroup did not recommend a specific approach for risk adjustment; further work is needed in this area. As noted earlier, it is critical to record dates for diagnosis, treatment(s), and outcomes.

Figure 1.

NSCLC-specific OMF. The OMF depicts the minimum set of outcome measures recommended by the workgroup (right column), as well as the characteristics of the participant, disease, and provider (left column) and treatments of interest (center column) that should be captured to support risk adjustment.

Abbreviations: CHF, congestive heart failure; CKD, chronic kidney disease; COPD, chronic obstructive pulmonary disease; CVA, cerebrovascular accident; NSCLC, non–small cell lung cancer; OMF, Outcome Measures Framework; PRO, patient-reported outcome; TIA, transient ischemic attack.

^aIncluding diabetes, liver disease, AIDS, moderate to severe CKD, CHF, myocardial infarction, COPD, peripheral vascular disease, CVA or TIA, dementia, hemiplegia, connective tissue disease, solid tumor, leukemia, malignant lymphoma, peptic ulcer disease.

Standardized Library

The narrative definitions were translated to standardized terminologies to create a common outcome measure library that could be implemented within EHRs. The following were defined for each measure: the initial population for measurement (eg, all patients with lung cancer), the outcome focused population (patients who experienced the outcome of interest), and the data criteria and value sets. Three challenges were noted in this process. First, EHRs are unlikely to be able to capture the RECIST definition for progression and recurrence using structured observations; it is more likely that an imaging report will assert a change in lesion size, possibly with measurements, but without a specific reference to a specific set of lesions that have been measured.

Regarding toxicity, >750 grade 3 or grade 4 complications are listed in the CTCAE¹⁵ for RT and systemic therapy. Rather than model each complication, observations were created for CTCAE grade 3 and 4 findings. This approach also allows for the capture grade 3 or 4 complications regardless of the version of CTCAE used.

Last, in defining toxicity, the relationship between the complication and the presumed inciting procedure/therapy is inferred by date/time stamps (as opposed to a directly asserted causal relationship as is done in prospective clinical trials).

Discussion

The minimum set of harmonized outcome measures is broadly relevant to clinicians and patients and feasible to capture across NSCLC disease stages and treatment pathways. The harmonized measures are designed to build connections across routinely captured clinical data and the data collected by research, quality improvement, and public health surveillance efforts. Consistent collection of these measures in EHRs, patient registries, and other data collection systems would create opportunities for efficient new research to describe NSCLC treatment patterns and patient outcomes across treatment modalities and understand the effectiveness of new treatment approaches. Long-term capture of these outcome measures would also provide much-needed information on 5- and 10-year outcomes of patients treated with newer therapies.

Broad participation from a diverse group of stakeholders and registries who brought experience and perspectives related to different treatment specialties, treatment outcome, measurement of PROs, and use of existing data sources for research purposes was a strength of this initiative. These perspectives enabled the workgroup to consider a wide array of potential uses for the harmonized outcome measures. Translation of the narrative definitions into standardized terminologies is also a major strength; standardization is intended to facilitate consistent capture of the measures and support harmonization of data collection across learning healthcare systems.

The minimum measure set has some limitations. Most notably, the workgroup was unable to reach consensus on a specific validated instrument for measuring PROs. This finding highlights the need for further research in several areas. First, additional research is needed to guide the selection of appropriate PROs, particularly research into which domains are important to patients and how these domains differ depending on disease stage and treatment modality and intent. Next, information is needed about what level of respondent burden, both in terms of number of questions and frequency of completion, is acceptable to patients at various stages of the disease. And last, research is needed to explore how information from PROs can be used to inform clinical decision-making.

In the area of PROs, the workgroup diverged from the standard set of outcome measures for lung cancer produced by the International Consortium for Health Outcomes Measurement (ICHOM).¹³ ICHOM recommended the use of specific instruments, namely the EORTC Quality of Life Questionnaire-Core 30 (QLQ-C30)²⁰ and the corresponding lung cancer–specific module (QLQ-LC13).²¹ These validated instruments were considered by the workgroup but were not recommended for 2 reasons. The workgroup did not find evidence of wide use of these instruments in either clinical practice or registry-based research in the United States. In fact, only one of the participating registries reported use of these instruments. The workgroup also expressed concerns about burden, because completion of both questionnaires would require patients to answer 43 items.

In addition to the challenges related to PROs, further work is needed to improve the documentation of progression in routine clinical practice. Further research is also necessary to identify the experience of care concepts that are important to patients, clinicians, and other stakeholders, to identify or develop validated instruments to capture these concepts, and to determine how these measures may be used to inform clinical care. Implementation of the minimum measure set in clinical practice will require the use of templates and unstructured text in the EHR to reduce burden on providers. A pilot implementation of the measures would be valuable for demonstrating feasibility, identifying barriers, and describing the value of the measures for research and clinical decision-making.

Finally, the minimum measure set should be reviewed and potentially revised on a regular basis to reflect the rapidly evolving nature of NSCLC treatment and any implementation challenges experienced by users of the measures, including the rapidly evolving role of immune-based therapies in both advanced and localized stages of disease. Future revisions should also seek to evaluate the evolving use of PROs in clinical practice and make specific recommendations for PRO instruments.

Conclusions

By collecting the harmonized measures consistently, registries and other data collection systems could contribute to the development research infrastructure and learning health systems to support new research and improve patient outcomes.