The National Lung Screening Trial’s Endpoint Verification Process: Determining the Cause of Death

Abstract

Background

Randomized controlled trials (RCTs) evaluating cancer screening modalities usually employ cause-specific mortality as their primary endpoint. Because death certificate cause of death can be inaccurate, RCTs frequently use review committees to assign an underlying cause of death. We describe the National Lung Screening Trial’s (NLST’s) death review approach, the Endpoint Verification Process (EVP), which strives to minimize errors in assignment of cause of death due to lung cancer.

Methods

Deaths selected for review include those with a notation of lung cancer on the death certificate and those occurring among participants ever diagnosed with lung cancer. Other criteria that trigger death review include, but are not limited to, death within six months of a screen suspicious for lung cancer and death within 60 days of certain diagnostic evaluation procedures associated with a screen suspicious for lung cancer or a lung cancer diagnosis. EVP requires concordance on whether death was due to lung cancer. Deaths are first reviewed by the EVP chair. If concordance is not achieved, the death is next reviewed by an Endpoint Verification Team (EVT) member. If concordance between the chair- and member-assigned cause of death is not achieved, the death is next reviewed by a group of at least three EVT members. Cause of death is assigned at the step in which concordance was achieved, or if necessary, at the team review.

Conclusions

NLST’s EVP is designed to produce a highly accurate count of lung cancer deaths.

INTRODUCTION

RCTs that evaluate cancer screening modalities usually employ cause-specific mortality as their primary endpoint. For example, analysis of data from an RCT of lung cancer screening would involve comparison of the lung cancer mortality rate between an intervention arm and a control arm. Analyses focus on cause-specific mortality rates because such measures are free of common screening biases (lead time, length, and overdiagnosis bias) and indicate whether screening prevents or delays death from the disease of interest. While all-cause mortality is a more rigorous endpoint than cause-specific mortality, its utility is limited due to statistical power considerations.

The public health impact of results from a major cancer screening RCT may be far-reaching. It is therefore of the utmost importance that the outcome, cause-specific mortality, be ascertained accurately. While death certificates are a readily available source of cause of death information, errors are known to exist [1, 2]. Pritt et al. [1] reported “major error” rates ranging from between 24% to 37% in five studies, with a “major error” defined as a wrong cause or manner of death. Doria-Rose et al [3] demonstrated good agreement between death certificates and death review in lung cancer screening trials of the 1970s and 80s, but errors in cause of death assignment still were present. Error is especially problematic in cancer screening RCTs due to the potential for two forms of bias: sticking diagnosis bias and slippery linkage bias [3]. Sticking diagnosis refers to the phenomenon whereby persons diagnosed with an often-fatal illness are assigned that illness as the cause of death on the death certificate, even if that illness is not the actual cause of death. Slippery linkage refers to the inability of death certificates to capture screening as the underlying cause of fatal adverse events associated with diagnostic evaluation or treatment of a screen-detected abnormality or illness. Both biases are differential with respect to trial arm assignment, as they are more likely to occur among deceased participants randomly assigned to the trial arm in which disease is more likely to be suspected or diagnosed.

To circumvent the shortcomings associated with the use of death certificate cause of death, RCT investigators rely on death review committees to determine the underlying cause of death. A death review committee usually consists of a small number of persons who review medical records of deceased study participants and assign an underlying cause of death. Members of death review committees are usually otherwise unaffiliated with the specific RCT.

The NLST was a large RCT designed to evaluate whether screening wit LDCT, relative to posteroanterior CXR, reduced lung cancer mortality. The NLST’s death review committee, the EVT, consisted of a committee chair and four other members, all of whom were medical doctors and were otherwise unaffiliated with the trial. The process whereby deaths were chosen for review and then reviewed was called the EVP. The purpose of this manuscript is to describe the EVP.

THE NLST

The NLST has been described in detail elsewhere [4]. Briefly, the trial enrolled over 53,000 persons who, at the time of randomization, were ages 55 to 74 years, had a history of long-term, heavy cigarette smoking, and no prior history of lung cancer. Each participant was randomized and invited to receive three annual screens with either LDCT or CXR. Data on vital status and cause of death as of December, 2009 were collected through September, 2010. Initial results of the NLST were reported in November, 2010 [5]. The trial demonstrated a statistically significant twenty percent reduction in lung cancer mortality with CT screening, relative to CXR.

The NLST was a joint venture of ACRIN, which is sponsored by the Cancer Imaging Program in the Division of Cancer Treatment and Diagnosis at the NCI, and the LSS, which is administered by the Division of Cancer Prevention at the NCI. While ACRIN and LSS individually administered their own trial components, the two were harmonized on major trial aspects such as screening protocols and data collection, and used the same EVP and EVT. The ACRIN and LSS components had independent screening centers and independent coordinating centers.

The NLST EVP

Summary

The primary goal of the NLST EVP was to minimize the extent of error in assignment of death due to lung cancer. The process strove to identify deaths whose underlying cause was lung cancer. For the purpose of EVP, this included deaths caused directly by lung cancer, deaths attributed indirectly to lung cancer due to diagnostic evaluation of a suspected (due either to symptoms or a positive screen, and including those instances in which lung cancer was ultimately ruled out) or confirmed lung cancer, and deaths associated with treatment of lung cancer. The EVP improved upon cause of death assignment using only death certificates, as it allowed for detailed review of medical records of deaths of potential relevance to NLST, including those deaths whose relevance was not fully discernible from causes of death listed on the death certificate.

Because LDCT and CXR can identify abnormalities suspicious for illnesses other than lung cancer, the NLST EVP also evaluated whether death was the direct or indirect result of complications relating to diagnostic evaluation of or treatment for a clinically significant abnormality not suspicious for lung cancer detected on a screening exam. The deliberations concerning this secondary goal were conducted as part of the EVP but were done independently of deliberations concerning whether the death was due to lung cancer. Deaths attributable to diagnostic evaluation of or treatment for a clinically significant abnormality not suspicious for lung cancer were not counted as lung cancer deaths, that is, they were not counted against the screening process.

The EVP

The EVP utilized a Web-based system called EVICT to track and document EVP activities. All EVP data forms completed by the EVT were part of EVICT and were completed on-line.

The EVP consisted of five tasks. Each task is described in detail below. A pictorial depiction of the process can be found in Figure 1.

Figure 1. NLST’s EVP.

Task 1: Coding of death certificates

Screening centers learned of a participant death through contact with family members or friends at the time of administration of annual (LSS) or semi-annual (ACRIN) health and vital status information forms, or through other methods such as obituaries. The screening center obtained a death certificate on each decedent and sent it to the coordinating center for coding by trained nosologists. All listed causes of death and significant conditions were coded according to rules set forth in the U.S. National Center for Health Statistics’ Instruction Manual 2b [6]. An underlying cause of death (referred to in the remainder of this paper as the “NCHS-derived underlying cause of death) was derived according to rules set forth in the U.S. National Center for Health Statistics’ Instruction Manual 2a [7]. Each death certificate was coded by two nosologists. Discrepancies between the two primary codes were adjudicated by a third nosologist. The ICD-10 coding system [8] was used.

Task 2: Selection of deaths for EVP review

An EVP selection algorithm dictated which deaths were selected for review. A death could be selected through one or more of five pathways (noted in Table 1):

1. Any cause of death (including the NCHS-derived underlying cause of death) or significant condition noted on the death certificate matched an EVP cause of death of interest. A summary of these causes is noted in Table 2; the complete listing of causes can be found in Supplemental Table 1. Deaths of interest included those due to lung cancer, those due to other cancers or illnesses that could possibly represent a true lung cancer erroneously diagnosed as another illness, and those that could be the result of diagnostic evaluation for a suspected lung cancer or treatment of a confirmed lung cancer.

2. Any death occurring in a participant who was diagnosed during the course of NLST with a cancer of interest¹. A summary of these cancers is noted in Table 2; the complete listing of cancers can be found in Supplemental Table 1. Cancers include lung cancer, cancers known to metastasize to the lung, and cancers that could have been erroneously diagnosed as primary when in fact they actually were metastases of lung cancer. Triggering of EVP review for certain cancer diagnoses was dependent upon the histology of the neoplasm.

3. A death occurred within 60 days of select diagnostic evaluation procedures that occurred as a result of a screen suspicious for lung cancer or in conjunction with a diagnosis of lung cancer². These were included to capture deaths that could be the result of diagnostic evaluation for a suspected lung cancer. The particular diagnostic evaluation procedures are listed in Table 3.

4. Chronic obstructive pulmonary disease was noted on the death certificate or as the NCHS-derived underlying cause of death, and the death occurred any time after select diagnostic evaluation procedures that occurred as a result of a screen suspicious for lung cancer or in conjunction with a diagnosis of lung cancer. This pathway served to capture deaths that could have been the result of respiratory failure exacerbated by diagnostic evaluation for a suspected lung cancer, treatment of a known lung cancer, or presence of a lung cancer not recorded on the death certificate. The particular diagnostic evaluation procedures are listed in Table 3.

5. A death occurred within six months of an NLST screen suspicious for lung cancer or a clinically significant abnormality not suspicious for lung cancer. These were selected to identify deaths that were in some way related to the screening process, but not identified through pathway 3 or 4.

Table 1. Pathways through which deaths are selected for EVT review.

A death is selected for EVT review if:

Any cause of death (including the NCHS-derived underlying cause of death) or significant condition noted on the death certificate matches an EVP cause of death of interest (Table 2 and Supplemental Table 1). Death occurs in a participant who was diagnosed during the course of NLST with a cancer of interest (Table 2 and Supplemental Table 1). Death occurs within 60 days of select diagnostic evaluation procedures (Table 3) that occur as a result of a screen suspicious for lung cancer or in conjunction with a diagnosis of lung cancer. Chronic obstructive pulmonary disease is noted on the death certificate or as the NCHS-derived underlying cause of death, and the death occurs any time after select diagnostic evaluation procedures (Table 3) that occurred as the result of a screen suspicious for lung cancer or in conjunction with a diagnosis of lung cancer. Death occurs within six months of an NLST screen suspicious for lung cancer or a clinically significant abnormality not suspicious for lung cancer.

Table 2. Summary of Cancers^a and Causes of Death that Trigger EVP (Pathways 1 and 2) (For a detailed listing, see Supplemental Table 1).

Cause of death and/or condition
Malignant neoplasms – primary diseaseb Bronchus and lung Heart, mediastinum, pleura Liver and intrahepatic bile ducts Bone and articular cartilage Meninges Brain Spinal cord, cranial nerves and other parts of central nervous system Adrenal gland
Malignant neoplasms – secondary (metastatic) malignant neoplasms Lung Mediastinum Pleura Other and unspecified respiratory organs Intrathoracic lymph nodes
Malignant neoplasms – other and ill-defined sites Respiratory organs Thorax Abdomen Other ill-defined sites Overlapping lesion of other and ill-defined sites Malignant neoplasm without specification of site Malignant neoplasms of independent (primary) multiple sites
Neoplasms of uncertain or unknown behavior Respiratory organs Liver, gallbladder and bile ducts Digestive organs Bone, articular cartilage Meninges Brain and central nervous system Other and unspecified sites
Ill-defined, unknown, and other causes of mortality Sudden cardiac death, so described Hemorrhage from other sites in respiratory passages Respiratory arrest Senility Abnormal findings on diagnostic image of lung Other sudden death, cause unknown Unattended death Other ill-defined and unspecified causes of mortality
Adverse effects of health care Misadventures during surgical and medical care Surgical operations, other surgical procedures, and other medical procedures as the cause of abnormal reaction of the patient, or of later complication, without mention of misadventure at the time of the procedure Sequelae of surgical and medical care as external cause Nosocomial conditions Streptococcal or other septicemia Bacterial infection of unspecified site Drugs, medicaments and biological substances causing adverse effects in therapeutic use Poisoning by drugs, medicaments, and biological substances

^aCancers trigger selection if diagnosed, noted on the death certificate, or determined to be the NCHS underlying cause of death

^bExclusions based on morphology exist for all noted sites in this section except bronchus, lung, and adrenal gland

Table 3. Select Diagnostic Evaluation Procedures (Pathways 3 and 4).

Procedures
Biopsies
Lung: Endobronchial, percutaneous transthoracic yielding histology or cytology, cytology - other, throracoscopic, transbbronchial
Lymph nodes: lymph node scalene (supraclavicular) nodes, lymph node - other
Other: open surgical, percutaneous adrenal, percutaneous liver, biopsy – other
Scopes
Mediastinoscopy; bronchoscopy; cytology-bronchoscopic; fluoroscopy; thoracoscopy
Scans
MRI abdomen or liver; MRI bone; MRI brain; MRI chest; MRI other; CT abdomen or liver; CT abdomen and pelvis, CT brain, CT chest plus contrast-enhanced nodule densitometry; CT chest; CT limited thin section of nodule; CT other; CT spiral; CT chest - limited thin section of entire lung; CT chest and abdomen; CT chest, abdomen and pelvis; radionuclide scan – fusion PET/CT scan
Surgical procedures
Wedge resection; multiple wedge resection; segmental resection; multiple segmental resection; chest wall resection; partial pleurectomy; median sternotomy; lobectomy; bilobectomy; pneumonectomy; thoracentesis; thoracotomy; lymphadenectomy/lymph node sampling

Deaths not selected for EVP review were classified as “certified” and received no further review. The NCHS-derived underlying cause of death from the death certificate was taken to be the cause of death.

Task 3: Preparation of the EVP folder

An EVP folder was assembled for each death selected for EVP review by the screening center that enrolled the participant. The EVP folder consisted of inpatient and outpatient medical records that provided sufficient information to determine the cause of death as well as the contributing factors that led up to death. Medical records included, but were not limited to, diagnosis and treatment documents, outpatient visit notes, hospital admission history, operative procedures reports, pathology reports, chemotherapy and radiation notes, hospital discharge summaries, diagnostic imaging reports, hospice notes, autopsy reports, and clinical laboratory data. All participant identifiers were removed from the medical records, as were references to the trial, randomization arm, and information indicating whether a cancer was detected as a result of screening or symptoms.

Task 4: Concordance Process

The goal of the concordance process was to determine whether the underlying cause of death was lung cancer. Such deaths, for the purpose of the NLST, included deaths due to confirmed lung cancer, diagnostic evaluation of a screen suspicious for lung cancer, diagnostic evaluation of a suspected, symptomatic lung cancer, or treatment for a diagnosed or suspected lung cancer.

Concordance was a potentially multistep process and involved, at the first two steps, comparison of cause of death assignment made by two independent sources. Concordance was assigned if there was agreement that the death was due to lung cancer or if there was agreement that the death was not due to lung cancer, even in the instance of disagreement on the particular cause of death. Discordance occurred when the two sources disagreed with respect to whether the death was due to lung cancer.

The principal form used in the concordance process was the CDQ. The CDQ is included as Supplemental Appendix 1. The CDQ contained 17 questions and allowed the EVP reviewer to specify whether death was directly or indirectly due to lung cancer, whether complications from diagnostic evaluation or treatment for a lung cancer contributed to death, and to indicate, if the death was not directly due to lung cancer, a cause of death. If the reviewer recorded that the death was directly due to lung cancer, lung cancer was automatically assigned to be the cause of death. The assigned cause of death was called the EVT-determined underlying cause of death.

The CDQ was completed without knowledge of screening examination results, with one exception: the screening examination result, but not the examination administered (i.e., LDCT or CXR), was reviewed and a separate section of the CDQ completed if medical records did not document any history of pathologically confirmed cancer or any history of clinically diagnosed lung cancer. In that case, questions that asked whether lung cancer was suspected and whether death was related to diagnostic evaluation for or treatment of a suspected lung cancer were answered. The rationale for reviewing screening results in this instance was to make sure that a death that represented the downstream effect of a screen suspicious for lung cancer was not missed, even if there was no mention of a suspicion of lung cancer in medical records.

A second form, the CDQ-CSA (Supplemental Appendix 2), was at times completed as part of the EVP review. The CDQ-CSA, completed after and independently of the CDQ, was used to evaluate whether death was due to the downstream effects of a clinically significant abnormality identified on an NLST screen that was suspicious for an illness other than lung cancer. The CDQ-CSA was not considered in the concordance process, as the questions were not relevant to death due to lung cancer. The CDQ-CSA was completed for participants who died within 6 months of an NLST screen suspicious for a clinically significant abnormality other than lung cancer, as well as for participants who died within 6 months of an NLST screen suspicious for lung cancer. Death review of the latter group was necessary because only one summary screening result was assigned to each screening exam, with lung cancer taking precedence over other abnormalities. Therefore, for participants who had both abnormalities suspicious for lung cancer and other significant conditions, the screening result defaulted to a result suspicious for lung cancer. Completion of the CDQ-CSA for participants who had a screen suspicious for lung cancer ensured that no individual for whom a clinically significant abnormality unrelated to lung cancer was reported six months prior to death would escape death review. The CDQ-CSA was always completed with knowledge of the screening examination result, but the specific examination administered was not provided.

The concordance process was administered by the coordinating centers. The number of steps in the process was at least one and as many as three.

Step 1: Chair-level review

This review was conducted without use of the death certificate. The EVT chair first examined the medical records in the EVP folder for completeness and confirmed that no references to screening arm were included. He then reviewed the medical records and completed a CDQ. Concordance was then evaluated: the chair’s assignment of underlying cause of death was compared with the NCHS-derived underlying cause of death coded from the death certificate. If these were concordant, the death was certified and the chair’s assignment of cause of death was recorded as the EVT-determined underlying cause of death. Otherwise, the death continued to Step 2: Member-level review.

If necessary, the CDQ-CSA was completed as part of Step 1. The chair’s answers to the CDQ-CSA questions finalized assessment of whether death was due to downstream effects of an NLST screen that identified a clinically significant abnormality not suspicious for lung cancer.

Step 2 (if necessary): Member-level review

An EVT member other than the chair was selected to conduct this review. The member was given the death certificate and medical records, but was not provided with the responses to the chair-level CDQ. The member reviewed the medical records and completed a CDQ. If concordance existed between the CDQs completed by the chair and the member, the death was certified, and an agreed-upon cause of death was recorded as the EVT-determined underlying cause of death. Otherwise, the death continued to Step 3: Team-level review.

Step 3 (if necessary): Team-level review

A minimum of three EVT members, including the chair, participated in the team-level review. The members reviewed the death certificate, medical records, and chair and member CDQs. Each death was discussed by the group on a conference call and an agreed-upon cause of death was recorded as the EVT-determined underlying cause of death.

Task 5: Assignment of final causes of death and ICD-10 codes

As previously mentioned, all deceased NLST participants had an NCHS-derived underlying cause of death. Deceased participants who were selected for EVT review also had an EVT-determined underlying cause of death, which corresponded to the cause of death assigned in the final step of the review process. The EVT-determined underlying cause of death was centrally coded by two trained nosologists using ICD-10 codes. Any discrepancies were adjudicated by a third nosologist.

DISCUSSION

The NLST EVP was modeled on the death review process used by the PLCO death review process [10]. The PLCO is evaluating four hypotheses, including whether annual screening with CXR, relative to usual medical care, can reduce lung cancer mortality. Experience accumulated during the course of managing the PLCO death review process demonstrated that a conditionally tiered concordance process was accurate and efficient, and the NLST adopted this structure. In PLCO, the EVP death review panel chair frequently concurred with the death certificate. For this reason, the NLST adopted concordance between the chair and the death certificate as the first step in the concordance process.

While the basic structure of the PLCO death review process could be adopted for the NLST, the fact that the NLST intervention arm was screening with LDCT required an additional component. That component – the evaluation of whether the downstream effects of a screen suspicious for a condition unrelated to lung cancer contributed to death – was necessary because LDCT reveals a variety of clinically significant findings unrelated to the diagnosis of lung cancer [11]. The diagnostic evaluation for and treatment of some of these abnormalities carry risks that may result in mortality. The data collected on these deaths will allow a more thorough assessment of any negative impact of LDCT screening that may exist.

One might wonder if an extensive death review process like the NLST’s EVP was necessary to accurately classify deaths due to lung cancer. Data from the Mayo Lung Project, a lung cancer screening RCT of chest x-ray conducted in the 1970s and 1980s, suggest that death certificates may be adequate: researchers observed a sensitivity of 90% and specificity of 99% in reporting lung cancer as the cause of death for death certificates, using the review committee assignment as the gold standard [12]. Most importantly, sensitivity and specificity were similar across study arms. We cannot conduct the same analysis in the NLST at this time, as those data are confidential. However, the NLST may not produce similar findings because the NLST death review process differed from that of the Mayo Lung Project: Mayo reviewed all deaths, while the NLST reviewed a subset of deaths, with the chance of selection based on the likelihood of lung cancer as cause of death. In addition, LDCT screening results in a higher detection rate of abnormalities suspicious for lung cancer than does CXR [13] and thus increases the potential for deaths attributable to the screening process (that is, deaths occurring as a result of diagnostic evaluation for or treatment of lung cancer) but not due to lung cancer per se. It is unlikely that such deaths would have lung cancer noted on the death certificate, but they would have been classified as a lung cancer death according to EVP because they were due indirectly to lung cancer. For that reason, we expect that the rate of cause of death reclassification will be higher in the LDCT arm than in the CXR arm. These differences between the NLST and the Mayo Lung Project support the use of a death review process in NLST.

The effort invested in development of NLST’s EVP makes one wonder if the process could be used in clinical trials with mortality endpoints other than lung cancer. The general structure, a multi-tiered approach in which the number of reviews is contingent, in effect, on the ease of identifying the true cause of death, is efficient, and we feel translatable to other venues. The NLST EVP could be considered for non-lung-cancer mortality outcomes relatively common to older smokers, as it selects for review deaths due to a number of causes seen frequently in that population. However, the selection criteria would need to be carefully tailored to the disease of interest. Perhaps the NLST’s EVP could provide a starting point for researchers wish to develop a thorough death review process.

Strengths of the NLST EVP include an extensive selection algorithm, making it unlikely that any deaths due either directly or indirectly to lung cancer were missed. Furthermore, inclusion of deaths among participants diagnosed with lung cancer is likely to have minimized the impact of sticking diagnosis bias; inclusion of deaths among participants who died within six months of a screen suspicious for lung cancer or within 60 days of select diagnostic evaluation procedures that occurred as a result of a positive screen or in conjunction with a diagnosis of lung cancer is likely to have minimized the impact of slippery linkage bias.

The resource-intensive nature of the NLST EVP can be viewed as a limitation. The architects of the NLST EVP felt, however, that it was more important to err on the side of caution by including deaths that had a small, though non-zero, probability of truly being due to lung cancer or the downstream effects of the screening process. Another difficulty encountered with the EVP was that it relied entirely on the ability of the screening centers to collect records that documented diagnosis, treatment, and events surrounding death. That often required multiple contacts with physicians and medical institutions. The need to call upon next-of-kin to grant permission for release of medical records if physicians or institutions had concerns about confidentiality complicated matters further, as such consents could not always be obtained. In addition, the specific records needed for proper adjudication were not always obvious to the screening center personnel who collected medical records. To circumvent that problem, the NLST included a process whereby the EVT chair or member could request additional records, and this process helped in some instances to improve the reviewer’s ability to reach an informed conclusion about the cause of death.

CONCLUSIONS

It is of the utmost importance to have accurate cause of death information in any RCT that relies on such data to evaluate the efficacy of an intervention. The NLST utilized an extensive death review process with the goal of obtaining accurate cause of death information. We believe that the wide net cast by the process allowed us to capture nearly, if not all, lung cancer deaths, thus allowing us to have highly accurate lung cancer cause of death information.

ABBREVIATIONS

ACRIN

American College of Radiology Imaging Network

CDQ

Cause of Death Questionnaire

CDQ-CSA

Cause of Death Questionnaire for Clinically Significant Abnormalities Other than Lung Cancer

COD

Cause of death

CXR

Chest x-ray

DC

Death certificate

EVICT

Endpoint Verification Internal Computerized Tracking

EVP

Endpoint Verification Process

EVT

Endpoint Verification Team

LDCT

Low radiation dose helical computed tomography

LSS

Lung Screening Study

NCHS

National Center for Health Statistics

NCI

National Cancer Institute

NLST

National Lung Screening Trial

PLCO

Prostate, Lung, Colorectal and Ovarian Cancer Screening Trial

RCT

Randomized controlled trials