Abstract
Objectives
To investigate the inter- and intra-reader agreement of immune Response Evaluation Criteria in Solid Tumors (iRECIST) and Response Evaluation Criteria in Solid Tumors version 1.1 (RECIST 1.1) in patients with lung cancer treated with immunotherapy.
Materials and Methods
This retrospective study included 85 patients with lung cancer treated with PD-1 blockade. Four radiologists evaluated computed topography (CT) scans before and after initiation of immunotherapy using iRECIST and RECIST 1.1. Weighted kappa (k) with equal weights was used to assess the intra-reader agreement between 2 repeated reads on overall response at all time points, best overall response, and the response at the time point of progression, as well as the intra-reader agreement between iRECIST and RECIST. The inter-reader agreement was calculated using Light’s kappa.
Results
Intra-reader agreement for overall response at all time points, best overall response, and time point of progression was substantial to almost perfect for both iRECIST and RECIST 1.1 (k = 0.651–0.983). Inter-reader agreement was substantial for iRECIST (κ = 0.657–0.742) while RECIST 1.1 was moderate to substantial (κ = 0.587–0.686). The level of inter-reader agreement was not higher on repeat read for iRECIST (κ = 0.677–0.709 and κ = 0.657–0.742 for first and second read, respectively) as well as for RECIST 1.1 (κ = 0.587–0.659 and κ = 0.633–0.686 for first and second read, respectively). Almost perfect agreement was observed between RECIST 1.1 and iRECIST at first (κ = 0.813–0.923) and second read (κ = 0.841–0.912).
Conclusion
The inter- and intra-reader agreement of iRECIST is high and similar to RECIST 1.1 in patients with lung cancer treated with immunotherapy.
Keywords: lung cancer, immunotherapy, agreement, RECIST 1.1, iRECIST
1.0. Introduction
Immunotherapy with checkpoint inhibition is a novel oncologic treatment strategy which aims to provoke an anti-tumor response by activating the patient’s immune system. There have been numerous trials demonstrating the clinical benefit of this treatment strategy, ultimately leading to Federal Drug Administration (FDA) approval for several checkpoint inhibitors in multiple cancer subtypes including lung cancer [1–7].
Radiologically assessed changes in tumor burden inform daily clinical oncologic practice and have been used as surrogates for survival or other clinical endpoints in clinical trials [8]. There are several formal radiological response assessment methods [9–16]; however, the most commonly used criteria are the Response Evaluation Criteria in Solid Tumors Version 1.1 (RECIST 1.1) [11]. RECIST 1.1 was developed in the era of cytotoxic chemotherapy. Tumor response in this setting is classically characterized by a prompt reduction in the size of lesions, without the development of new lesions.
The mechanism of action of checkpoint inhibitors can lead to tumor infiltration by immune cells, which can cause a transient increase in tumor size in patients who are in fact benefiting from treatment (this is frequently termed pseudo-progression). In such cases, tumors which are undergoing an anti-tumor immune response enlarge on radiological studies and new lesions may be seen to develop as previously occult metastases increase in size. Many patients with such an initial response ultimately go on to have late but durable responses [17]. This type of treatment response cannot be adequately captured by RECIST 1.1 which therefore led to the development of new response assessment criteria.
The first new criteria were the immune-related response criteria (irRC) [18]. The most important change was that, according to irRC guidelines, the development of new lesions does not automatically denote progressive disease at follow up, as opposed to automatically denoting them as progressive disease according to RECIST 1.1. In addition, irRC added the recommendation that radiological progression should be confirmed with follow-up imaging at ≥ 4 weeks to account for the possibility of pseudo-progression. A modified version of the irRC, typically referred to as immune-related Response Evaluation Criteria In Solid Tumors (irRECIST), was subsequently developed and used in clinical trials [19, 20].
In an attempt to standardize response data collected in clinical trials for immune-based therapies, a subcommittee of the RECIST working group convened in 2016 to develop detailed response assessment guidelines termed immune Response Evaluation Criteria in Solid Tumors (iRECIST) [21]. Similar to RECIST 1.1 in its scope and level of detail, but using the principles outlined in irRC and irRECIST, iRECIST has the potential to be the mainstay of response assessment in the era of checkpoint inhibitor immunotherapy.
The reproducibility of any radiological evaluation is a vital metric of its clinical utility and is particularly important in the context of response assessment guidelines which aim to produce consistency of assessment both within a given trial, and from trial to trial. The aim of this study is to evaluate the repeatability and reproducibility of tumor response assessments using iRECIST, by assessing the inter-reader and intra-reader variability in a sample of lung cancer patients undergoing treatment with checkpoint inhibition. In addition, we compared the reproducibility of assessments using iRECIST to those performed using RECIST 1.1.
2.0. Materials and Methods
This was an institutional review board-approved, Health Insurance Portability and Accountability Act compliant study performed at a tertiary oncologic referral center. The requirement for informed patient consent was waived by the institutional review board for this retrospective study.
2.1. Study Population
Between March 2015 and April 2016, 185 consecutive patients with metastatic lung cancer underwent treatment with PD-1 blockade at our institution. A total of 97 patients without baseline computed tomography (CT) imaging and at least one follow-up examination or serial on-treatment examinations were excluded from the study. An additional 3 patients were excluded who underwent adjuvant locoregional therapies during the follow-up period. After exclusions, the final study sample consisted of 85 patients. Only scheduled follow-up CT examinations and those performed due to clinical suspicion for progression of disease were included in the study.
2.2. Image acquisition
CT scans were performed on either a 16-slice (Lightspeed 16) or 64-slice multidetector CT (Lightspeed VCT or Discovery CT750 HD, GE Medical Systems, Chicago, IL). Imaging of the chest was obtained in supine positioning during full inspiration from the supraclavicular fossa through the adrenal glands and imaging of the abdomen and pelvis was obtained from the diaphragm to the symphysis pubis. Tube current modulation at 120 kVp yielded a tube current of between 120–380 mA (220–380 mA for the abdomen and pelvis) with a pitch of 0.984–1.375 and rotation time of 0.5 seconds. Axial images were obtained at a slice thickness of 1.25 mm, with lung algorithm reconstructions through the chest and a thickness of 5 mm with soft tissue algorithm reconstructions through the chest, abdomen, and pelvis. Coronal and sagittal images were reconstructed at a slice thickness of 5 mm with soft tissue algorithm reconstruction.
Intravenous and oral contrast was given unless contraindicated. 80 mL of Iohexol 300 mg/mL intravenous contrast was administered at a rate of 2.5 mL/sec with a 40-second scan delay for imaging of the chest, and 150 mL Iohexol 300 mg/mL was at a rate of 2.5 mL/sec with an 80-second scan delay for imaging of the abdomen and pelvis. 30 mL Iohexol 300 mg/mL oral contrast mixed in 1000 mL of aqueous diluent was administered one hour prior to scanning.
2.3. Image analysis
Images were analyzed by four radiologist readers with different levels of experience in RECIST evaluation (Reader 1, S.H.C., oncologic imaging fellow with two years of experience using RECIST 1.1; Reader 2, A.P., radiologist with five years of post-fellowship body imaging experience and five years of experience using RECIST 1.1; Reader 3, J.G., radiologist with ten years of post-fellowship body imaging experience and six years of experience using RECIST 1.1; and Reader 4, D.H., radiologist with four years of post-fellowship body imaging experience and four years of experience using RECIST 1.1 and irRECIST). All studies were interpreted and target lesions measured on our institution’s picture archiving and communication system (Centricity, GE Healthcare, Chicago, IL). Readers were blinded to the current clinical status of the patient at the time of image review.
Prior to analyzing study data, the readers were provided with iRECIST guidelines to review, and these guidelines were subsequently discussed in detail. Each radiologist was then given six training cases with a mean of three time points (2–4) to independently evaluate using iRECIST. A second informational meeting took place in which iRECIST guidelines were again discussed once the training cases were completed.
RECIST 1.1 and iRECIST evaluations were performed for all study cases. Resultant target lesion measurements using RECIST 1.1 were also used in subsequent iRECIST assessments for each time point. For RECIST 1.1, the overall response at each time point, the best overall response (BOR), and the time point at which progressive disease was documented were recorded. For iRECIST, the target lesion response, the non-target lesion response, the new lesion response, and the overall response at each time point were recorded, in addition to the BOR and the time point at which progressive disease occurred.
Response assessment was performed twice by each radiologist for all patients. To decrease recall bias, the radiologists completed the second assessment with more than a 6-week interval from the initial assessment.
2.4. iRECIST guidelines
Details of the iRECIST guidelines have been previously reported [21, 22]. A summary of these guidelines and their comparison to RECIST 1.1 guidelines is provided in Table 1.
Table 1.
Summary of comparison between RECIST 1.1 and iRECIST.
| RECIST 1.1 | iRECIST | |
|---|---|---|
| Type of measurement | Unidimensional (sum of diameters) | Follow the definitions from RECIST 1.1. |
| Definition of target lesions | - Non-nodal lesions: ≥ 10 mm in longest diameter - Nodal lesions: ≥ 15 mm in shortest diameter - Up to 5 lesions - Maximum of 2 per organ |
Follow the definitions from RECIST 1.1. |
| Definition of new target lesion | N/A | Follow the definitions of target lesion size and number from RECIST 1.1 |
| Definition of complete response (CR) | - Disappearance of all target and non-target lesions - Nodal lesions < 10 mm in shortest diameter - No new lesions |
iCR
Follow the definitions from RECIST 1.1. |
| Definition of partial response (PR) | - Decrease of 30% in sum of diameters of target lesions relative to baseline - No progression of non-target lesions - No new lesions |
iPR
Follow the definitions from RECIST 1.1. |
| Definition of stable disease (SD) | Neither PR nor PD |
iSD
Follow the definitions from RECIST 1.1. |
| Definition of progressive disease (PD) | - ≥ 20% increase in sum of diameters of target lesions relative to nadir - Unequivocal progression of non- target lesions - New lesions |
iUPD unconfirmed progressive disease
- ≥ 20% increase in sum of diameters of target lesions relative to nadir - Unequivocal progression of non-target lesions - New target lesion - New non-target lesion |
| Definition of confirmed progressive disease | N/A |
iCPD confirmed progressive disease
- Recommended minimum 4 weeks and no more than 8 weeks after the first iUPD - Increase in sum of diameters of target lesions by ≥ 5 mm - Any further increase in previously progressed non-target lesions - Increase in new lesion sum of diameters by ≥ 5 mm - Any further increase in new non-target lesions - New progressive disease in any category outside the one in which the initial determined of iUPD was made. |
2.5. Statistical analysis
Weighted kappa with equal weights was used to assess intra-reader agreement for iRECIST and RECIST 1.1 as well as between iRECIST and RECIST 1.1. When assessing agreement between iRECIST and RECIST 1.1, the response categories “unconfirmed progressive disease” and “confirmed progressive disease” (iUPD and iCPD, respectively, as defined by iRECIST) were considered equivalent to the category of progressive disease (PD), as defined by RECIST 1.1). The kappa statistic with equal weights was used to assess the intra-reader agreement. Light’s kappa with equal weights was estimated for inter-reader agreement between the four radiologists. Confidence intervals of kappa statistics were estimated using the biased corrected and accelerated bootstrap approach. Kappa (κ) values were interpreted as follows: 0.00–0.20, slight agreement; 0.21–0.40, fair agreement; 0.41–0.60, moderate agreement; 0.61–0.80, substantial agreement; and 0.81–1.00, almost perfect agreement [23]. The percent of concordance on the response assessments was also reported per radiologist between 2 repeated reads as well as between iRECIST and RECIST 1.1. All statistical analyses were performed in software packages R version 3.4 (The R Foundation for Statistical Computing).
3.0. Results
The baseline demographic features of the study sample (n = 85 patients) are summarized in Table 2. The mean age was 69 years (range 41–92 years). Across the entire study sample, 84 (99%) patients had non-small cell lung cancer (NSCLC) and 71 (85%) of these were adenocarcinoma; a single patient had small cell lung cancer. Median follow up time intervals between scans ranged from 36.5 days between fifth and sixth follow-up assessments for 2 patients, and 66 days between baseline CT and first follow-up assessment among all 85 patients. The median follow-up time from initial to final assessment was 127 days (range 36–384). Among the 255 CT scans analyzed, 210 (82%) were performed with intravenous contrast and 45 (18%) without intravenous contrast.
Table 2.
Patients’ baseline clinical characteristics.
| Number of patients (%), n=85 | |
|---|---|
| Age in years, mean (range) | 69 (41–92) |
| Sex | |
| Male | 42 (49%) |
| Female | 43 (51%) |
| Histology | |
| Non-small cell lung caner | 84 (99%) |
| Adenocarcinoma | 71 (84%) |
| Squamous cell carcinoma | 12 (14%) |
| Large cell neuroendocrine carcinoma | 1 (1%) |
| Small cell lung cancer | 1 (1%) |
| Number of follow-up CT scans | |
| 1 | 49 (58%) |
| 2 | 17 (20%) |
| ≥ 3 | 19 (22%) |
Abbreviations: CT = computed tomography
3.1. Summary of response types
Figure 1 shows the distribution of BOR by reader at first and second read. Across the entire study sample, PD was reported in 37–46 (44%–54%) patients, stable disease (SD) in 26–35 (31%–41%), partial response (PR) in 10–14 (12%–17%), and complete response (CR) in 0–1 (0%–1%), among the different readers according to RECIST 1.1. Unconfirmed progressive disease (iUPD) was reported in 31–45 (37%–53%) patients, stable disease (iSD) in 29–36 (34%–42%), partial response (iPR) in 9–17 (11%–20%), and complete response (iCR) in 0–2 (0%–2%), among the different readers according to iRECIST.
Figure 1.
Distribution of the readers’ best overall responses at first and second read according to RECIST 1.1 and iRECIST.
3.2. Intra and inter-reader agreement
Intra-reader agreement for BOR, time point of progression, and overall response at all time points was substantial to almost perfect for both iRECIST (κ = 0.651– 0.981) and RECIST 1.1 (κ = 0.626–0.983) (Table 3). The range of concordance percent was 78.7%–98.1% for iRECIST and 78.8%–98.7% for RECIST 1.1. There was no difference in intra-reader agreement between the most experienced radiologist (R3) and the least experienced radiologist (R1).
Table 3.
Intra-reader agreement on repeated reads for each reader separately (R1–R4).
| Best overall response (BOR) | Time point of progression | Overall response at all time points | |||||
|---|---|---|---|---|---|---|---|
| kappa (95% CI) | % Concordance | kappa (95% CI) | % Concordance | kappa (95% CI) | % Concordance | ||
| iRECIST | R1 | 0.831 (0.729, 0.917) | 87.10% | 0.651 (0.466, 0.773) | 82.35% | 0.778 (0.741, 0.809) | 80.00% |
| R2 | 0.91 (0.797, 0.972) | 94.10% | 0.968 (0.893, 1.000) | 97.65% | 0.981 (0.972, 0.989) | 98.10% | |
| R3 | 0.765 (0.619, 0.864) | 82.35% | 0.702 (0.522, 0.825) | 82.35% | 0.747 (0.690, 0.805) | 78.70% | |
| R4 | 0.91 (0.815, 0.969) | 92.94% | 0.936 (0.846, 0.983) | 95.29% | 0.914 (0.888, 0.936) | 92.20% | |
| RECIST 1.1 | R1 | 0.792 (0.652, 0.892) | 85.88% | 0.664 (0.482, 0.794) | 82.24% | 0.751 (0.69, 0.796) | 83.80% |
| R2 | 0.969 (0.898, 1.000) | 97.64% | 0.948 (0.852, 1.000) | 96.47% | 0.983 (0.973, 0.991) | 98.70% | |
| R3 | 0.760 (0.619, 0.859) | 81.20% | 0.626 (0.436, 0.762) | 78.80% | 0.764 (0.713, 0.814) | 82.30% | |
| R4 | 0.877 (0.785, 0.950) | 90.59% | 0.893 (0.732, 0.962) | 94.11% | 0.920 (0.893, 0.943) | 94.10% | |
Inter-reader agreement for BOR, time point of progression, and overall response at all time points was substantial for iRECIST (κ = 0.657–0.742) and was moderate to substantial for RECIST 1.1 (κ = 0.587–0.686) (Table 4). There was no difference in inter-reader agreement on the first read compared to the second read for iRECIST (κ = 0.677–0.709 and κ = 0.657–0.742 for first and second read, respectively) or for RECIST 1.1 (κ = 0.587–0.659 and κ = 0.633–0.686 for first and second read, respectively). 95% confidence intervals of κ-values overlapped for all categories assessed.
Table 4.
Inter-reader agreement for 4 readers
| Best overall response (BOR) | Time point of progression | Overall response at all time points | ||
|---|---|---|---|---|
| kappa (95% CI) | kappa (95% CI) | kappa (95% CI) | ||
| iRECIST | 1st Read | 0.708 (0.618, 0.787) | 0.677 (0.572, 0.769) | 0.709 (0.671, 0.744) |
| 2nd Read | 0.742 (0.653, 0.815) | 0.657 (0.538, 0.753) | 0.687 (0.554, 0.771) | |
| RECIST 1.1 | 1st Read | 0.659 (0.559, 0.756) | 0.587 (0.485, 0.680) | 0.615 (0.562, 0.661) |
| 2nd Read | 0.686 (0.596, 0.783) | 0.633 (0.517, 0.740) | 0.654 (0.554, 0.773) |
3.3. Agreement between iRECIST and RECIST 1.1
Almost perfect agreement for BOR, time point of progression, and overall response at all time points was noted between iRECIST and RECIST 1.1 for all readers at first read (κ = 0.813–0.923) and second read (κ = 0.841–0.912) (Table 5). Figure 2 shows a representative case of disagreement between iRECIST and RECIST 1.1 in a patient with pseudo-progression.
Table 5.
Agreement between RECIST 1.1 and iRECIST for each reader (R1–R4).
| Best overall response (BOR) | Time point of progression | Overall response at all time points | |||||
|---|---|---|---|---|---|---|---|
| kappa (95% CI) | % Concordance | kappa (95% CI) | % Concordance | kappa (95% CI) | % Concordance | ||
| 1st Read | R1 | 0.887 (0.719, 0.967) | 95.30% | 0.840 (0.656, 0.927) | 91.76% | 0.863 (0.804, 0.917) | 94.19% |
| R2 | 0.845 (0.668, 0.939) | 91.76% | 0.899 (0.720, 0.979) | 95.29% | 0.908 (0.864, 0.948) | 95.45% | |
| R3 | 0.849 (0.711, 0.935) | 90.59% | 0.813 (0.659, 0.923) | 90.59% | 0.865 (0.813, 0.915) | 92.16% | |
| R4 | 0.923 (0.772, 0.984) | 96.47% | 0.893 (0.638, 0.969) | 95.29% | 0.859 (0.796, 0.918) | 93.46% | |
| 2nd Read | R1 | 0.893 (0.753, 0.966) | 94.11% | 0.887 (0.749, 0.948) | 91.76% | 0.904 (0.867, 0.939) | 94.84% |
| R2 | 0.909 (0.707, 0.971) | 95.29% | 0.885 (0.729, 0.962) | 94.11% | 0.902 (0.860, 0.939) | 94.84% | |
| R3 | 0.912 (0.801, 0.972) | 94.12% | 0.884 (0.731, 0.969) | 95.29% | 0.894 (0.845, 0.940) | 93.55% | |
| R4 | 0.865 (0.726, 0.944) | 91.76% | 0.841 (0.613, 0.943) | 94.11% | 0.861 (0.799, 0.918) | 93.42% | |
Figure 2.
Axial computed tomography (CT) images from a 61-year-old female with lung adenocarcinoma. Radiologic pseudo-progression was noted for all four readers. (A) CT scan performed before the start of the immunotherapy shows the presence of target and non-target lesions in the left lower lobe. (B) At 1 month of therapy, the pulmonary lesions enlarged, corresponding to progressive disease according to RECIST 1.1 and to unconfirmed progression disease according to iRECIST. (C) At 5 months of therapy, the target lesion was significantly smaller and the non-target lesion was no longer seen, corresponding to partial response according to iRECIST.
3.4. Pseudo-progression
Five patients were documented by at least one reader to have had an initial increase in the size of lesions or the appearance of new lesions, followed by a decrease in tumor burden (Table 6). Among these patients, 36 initial and repeat iRECIST assessments by 4 readers included lung lesions (90%), 22 included thoracic nodal metastases (55%), 8 included intramuscular metastases in a patient with a psoas muscle lesion (2%), and 5–7 included adrenal, abdominal nodal, and osseous metastases (1–2%). Early pseudo-progression was observed in all five patients, with a median time to increase in tumor burden of 8.7 weeks (range 3.9 – 12.0 weeks). New lesions were observed five times by two of the readers and included new thoracic and abdominal nodes, osseous lesions, and an intramuscular lesion. In all cases, at least one other reader observed pseudo-progression while not accounting for new lesions. New lesion contribution in determining increase in tumor burden was simultaneous to an increase in sum lesion diameter (SLD). All patients classified as exhibiting radiologic pseudo-progression according to the readers were considered to have progressive disease at the first time point per RECIST 1.1.
Table 6.
Summarized response assessment according to RECIST 1.1 and iRECIST from four readers for patients (P1–P5) with initial increase in the sum of long diameters (SLD) of target lesions or appearance of new lesions followed by a decrease in tumor burden.
| Reader 1 | Reader 2 | Reader 3 | Reader 4 | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SLD (% change) | NL | OR | SLD (% change) | NL | OR | SLD (% change) | NL | OR | SLD (% change) | NL | OR | |||
| 1st Read | P1 | TP1 | 20.4 % | N | PD iUPD | 12.5% | N | SD iSD | 11.5% | N | SD iSD | 20.4% | Y | PD iUPD |
| TP2 | −14.2% | N | PD iSD | −12.5% | N | SD iSD | −19.5% | N | SD iSD | 5.8% | N | PD iSD | ||
| P2 | TP1 | 33.6% | Y | PD iUPD | 50.0% | N | PD iUPD | 11.4% | N | SD iSD | 24.3% | Y | PD iUPD | |
| TP2 | −66.7% | N | PD iPR | −100% | N | PD iCR | −63.9% | N | PR iPR | −71.9% | N | PD iPR | ||
| P3 | TP1 | 21.5% | N | PD iUPD | 7.6% | N | SD iSD | 9.8% | N | SD iSD | 36.7% | Y | PD iUPD | |
| TP2 | −32.2% | N | PD iPR | −10.9% | N | SD iSD | −45.9% | N | PR iPR | −12.2% | N | PD iSD | ||
| P4 | TP1 | 42.4% | N | PD iUPD | 45.4% | N | PD iUPD | 44.1% | N | PD iUPD | 39.3% | N | PD iUPD | |
| TP2 | −18.1% | N | PD iPR | −6.0% | N | PD iSD | −17.6% | N | PD iSD | −15.1% | N | PD iSD | ||
| P5 | TP1 | 1.94% | N | SD iSD | 5.3% | N | SD iSD | 12.8% | N | SD iSD | 15.5% | N | SD iSD | |
| TP2 | −21.3% | N | PR iPR | −20.2% | N | SD iSD | −4.2% | N | SD iSD | −14.2% | N | SD iSD | ||
| 2nd Read | P1 | TP1 | 14.8% | N | SD iSD | 16.5% | N | SD iSD | 10.3% | N | SD iSD | 16.5% | N | SD iSD |
| TP2 | −6.7% | N | SD iSD | −9.0% | N | SD iSD | −16.2% | N | SD iSD | −9.02% | N | SD iSD | ||
| P2 | TP1 | 26.5% | N | PD iUPD | 48.2% | N | PD iUPD | 23.0% | N | PD iUPD | 30.7% | Y | PD iUPD | |
| TP2 | −68.3% | N | PD iPR | −100% | N | PD iCR | −64.96% | N | PD iPR | −70.5% | N | PD iPR | ||
| P3 | TP1 | 28.0% | N | PD iUPD | 12.3% | N | SD iSD | 5.4% | N | SD iSD | 29.0% | N | PD iUPD | |
| TP2 | −24.0% | N | SD iSD | −20.0% | N | SD iSD | −25.0% | N | SD iSD | 8.0% | N | PD iSD | ||
| P4 | TP1 | 35.1% | N | PD iUPD | 35.3% | N | PD iUPD | 42.4% | N | PD iUPD | 42.2% | N | PD iUPD | |
| TP2 | −24.3% | N | PD iSD | −6.0% | N | PD iSD | −6.0% | N | PD iSD | −15.1% | N | PD iSD | ||
| P5 | TP1 | 25.2% | N | PD iUPD | −5.9% | N | SD iSD | 6.4% | N | SD iSD | 8.6% | N | SD iSD | |
| TP2 | −25.2% | N | PD iPR | −20.7% | N | SD iSD | −14.5% | N | SD iSD | −17.2% | N | SD iSD | ||
Abbreviations: iCR = immune complete response, iPD = immune confirmed progressive disease, iPR = immune partial response, iSD = immune stable disease, iUPD =immune unconfirmed progressive disease, NL = new lesion, PD = progression of disease, SD = stable disease, TP = time point
4.0. Discussion
In this retrospective study, we investigated the intra and inter-reader agreement using iRECIST and RECIST 1.1 for assessing response to therapy in a cohort of patients with lung cancer treated with Nivolumab. iRECIST was developed by a sub-committee of the RECIST working group in 2017 for use in the context clinical trials of checkpoint inhibitor immunotherapy. The intra-reader agreement in categorizing response was substantial to almost perfect for both iRECIST and RECIST 1.1, and the inter-reader agreement was substantial for iRECIST. These results demonstrate that the reproducibility of response assessment using iRECIST is comparable to RECIST 1.1 and lend weight to the use of iRECIST in clinical trials [21].
Reproducibility is an important feature of any method used for assessing treatment response in daily radiological practice or clinical trials. In previous studies which have examined the reproducibility of RECIST 1.1, intra-observer agreement tends to be substantial, while agreement between observers is more modest [24–26]. In one study, there was almost perfect agreement between RECIST 1.1 and RECIST in assessing tumor response in patients with NSCLC after targeted therapy [27].
Several studies have also evaluated the reproducibility of modified/novel response assessment criteria [28–30]. In the immunotherapy setting, response assessment by irRC and modified irRC based on RECIST 1.1 that reduces the number of measurements was shown to have almost perfect agreement in a population of melanoma patients treated with ipilimumab [31]. Similarly, Choi et al investigated the reproducibility of the modified RECIST criteria (mRECIST) based on tumor viability and found moderate to excellent inter-reader agreement in patients with hepatocellular carcinoma after transarterial chemoembolization. They also observed increased reproducibility with more experienced radiologists, a finding not seen in our study, perhaps due to shared subspecialized training in oncologic imaging among our readers despite differing levels of experience [32]. Furthermore, we found that agreement did not improve significantly between iRECIST and RECIST1.1, suggesting that there was no additional acquired expertise for new criteria use, possibly because the principles of tumor assessment are for the most part unchanged in iRECIST.
We observed almost perfect agreement between iRECIST and RECIST 1.1 for BOR, time point of progression, and overall response at all time points for all readers at first and second reads. This trend confirms recent pooled FDA analysis of patients treated with anti-PD-1/PD-L1 antibodies that demonstrated analogous objective response rates of 31.5% and 30.5% by iRECIST and RECIST 1.1, respectively [33]. Almost perfect agreement in our study was primarily due to our limited number of cases with pseudo-progression (2% to 5%), the only distinguishing feature of iRECIST. And as expected, RECIST 1.1 was unable to capture this type of response as it indiscriminately classifies initial increased tumor burden as PD. The result of this outcome is that RECIST 1.1 underestimates the effects of immunotherapy [34]. Similar to our observations, a recent study of 42 patients with NSCLC receiving immunotherapy demonstrated superiority of iRECIST to RECIST 1.1 in response prediction among the two patients with pseudo-progression [35]. In yet another study, a single patient among 43 patients with NSCLC was identified as having pseudo-progression by iRECIST criteria that had been classified as PD by RECIST 1.1 [36].
Pseudo-progression is a unique phenomenon characterized as tumor response after initial radiologic progression. The frequency of pseudo-progression in oncologic patients receiving immunotherapy is low and has been described in 0% to 5% of patients with lung cancer [37–39]. In a recent pooled analysis, 10 of 535 NSCLC patients (1.9%) treated with checkpoint inhibitors therapy had a partial response after radiologic progressive disease [40].
iRECIST was published in an attempt to provide detailed guidelines and standardize response assessment in the immunotherapy setting; however, variability in lesion measurements or target selection could impact tumor response classification [41]. Erasmus et al showed that repeat measurements of lung tumors can be significantly different and potentially affects the response assessment by the World Health Organization and RECIST criteria [42]. Another potential cause of discrepancy in tumor response assessment is the differences in the detection of new lesions between readers [43]. In our study, both SLD variability and differences in new lesion identification affected tumor response assessment.
The limitations of this study include its retrospective design and that a majority of patients had NSCLC. Although lung cancer patients represent one of the largest patient groups receiving checkpoint inhibitors at our institution, single cases of large cell neuroendocrine carcinoma and small cell lung cancer were included in this study owing to our restrictive selection criteria and limited pool of consecutive candidates. Application of our conclusions to these underrepresented lung cancer subtypes is finite, particularly with cases of small cell lung cancer where perihilar tumors are common and accurate lesion assessment may be suboptimal. Futhermore, all readers had expertise in both oncologic imaging and formal tumor response assessment which may not accurately reflect a more heterogeneous population of radiologists in clinical practice. Future analysis is warranted to better characterize the relationship between oncological imaging expertise and accuracy in iRECIST assessment. Lastly, the association between clinical outcome estimation and RECIST 1.1 or iRECIST was not evaluated.
5.0. Conclusions
The inter- and intra-reader agreement of iRECIST in patients with lung cancer treated with checkpoint inhibitor immunotherapy was moderate to excellent. These findings support the use of iRECIST as a reliable tool for treatment response assessment in clinical trials as the use of immunotherapy becomes increasingly prevalent. The substantial agreement among readers when using iRECIST suggests that trained radiologists can equally contribute to the assessment of immunotherapy treatment response in patients with lung cancer.
Highlights.
Substantial agreement among iRECIST readers for treatment response in lung cancer
Treatment response assessment using iRECIST is comparable to RECIST 1.1
RECIST 1.1 underestimates the treatment effects of immunotherapy
Radiologist experience does not affect accuracy of iRECIST assessment
Acknowledgements
The authors would like to thank Joanne Chin, MFA, ELS, for assisting with the preparation of the manuscript.
Funding
This study is funded in part through the NIH/NCI Cancer Center Support Grant P30 CA008748. The sponsor had no role in the study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.
Conflict of Interest
MDH reports research grant from BMS; personal fees from Achilles; Arcus; AstraZeneca; Blueprint; BMS; Genentech/Roche; Genzyme, Immunai; Instil Bio; Janssen; Merck; Mirati; Natera; Nektar; Pact Pharma; Regeneron; Shattuck Labs; Syndax; as well as equity options from Arcus, Factorial, Immunai, and Shattuck Labs. The remaining authors declare no conflicts of interest.
Abbreviations
- BOR
Best overall response
- CR
Complete response
- CT
Computed topography
- FDA
Food and Drug Administration
- iCR
Immune complete response
- icPD
Immune confirmed progressive disease
- iPR
Immune partial response
- iRECIST
Immune Response Evaluation Criteria in Solid Tumors
- irRC
Immune-related response criteria
- iSD
Immune stable disease
- iUPD
Immune unconfirmed progressive disease
- NSCLC
Non-small cell lung cancer
- PD
Progressive disease
- PR
Partial response
- RECIST 1.1
Response Evaluation Criteria in Solid Tumors version 1.1
- SD
Stable disease
- SLD
Sum of long diameters
- WHO
World Health Organization
Footnotes
Data Statement
The datasets used during the current study are available from the corresponding author on reasonable request.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- [1].Hodi FS, O’Day SJ, McDermott DF, Weber RW, Sosman JA, Haanen JB, Gonzalez R, Robert C, Schadendorf D, Hassel JC, Akerley W, van den Eertwegh AJ, Lutzky J, Lorigan P, Vaubel JM, Linette GP, Hogg D, Ottensmeier CH, Lebbe C, Peschel C, Quirt I, Clark JI, Wolchok JD, Weber JS, Tian J, Yellin MJ, Nichol GM, Hoos A, Urba WJ, Improved survival with ipilimumab in patients with metastatic melanoma, N Engl J Med 363(8) (2010) 711–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [2].Topalian SL, Hodi FS, Brahmer JR, Gettinger SN, Smith DC, McDermott DF, Powderly JD, Carvajal RD, Sosman JA, Atkins MB, Leming PD, Spigel DR, Antonia SJ, Horn L, Drake CG, Pardoll DM, Chen L, Sharfman WH, Anders RA, Taube JM, McMiller TL, Xu H, Korman AJ, Jure-Kunkel M, Agrawal S, McDonald D, Kollia GD, Gupta A, Wigginton JM, Sznol M, Safety, activity, and immune correlates of anti-PD-1 antibody in cancer, N Engl J Med 366(26) (2012) 2443–54. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [3].Borghaei H, Paz-Ares L, Horn L, Spigel DR, Steins M, Ready NE, Chow LQ, Vokes EE, Felip E, Holgado E, Barlesi F, Kohlhaufl M, Arrieta O, Burgio MA, Fayette J, Lena H, Poddubskaya E, Gerber DE, Gettinger SN, Rudin CM, Rizvi N, Crino L, Blumenschein GR Jr., Antonia SJ, Dorange C, Harbison CT, Graf Finckenstein F, Brahmer JR, Nivolumab versus Docetaxel in Advanced Nonsquamous Non-Small-Cell Lung Cancer, N Engl J Med 373(17) (2015) 1627–39. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [4].Garon EB, Rizvi NA, Hui R, Leighl N, Balmanoukian AS, Eder JP, Patnaik A, Aggarwal C, Gubens M, Horn L, Carcereny E, Ahn MJ, Felip E, Lee JS, Hellmann MD, Hamid O, Goldman JW, Soria JC, Dolled-Filhart M, Rutledge RZ, Zhang J, Lunceford JK, Rangwala R, Lubiniecki GM, Roach C, Emancipator K, Gandhi L, K.−. Investigators, Pembrolizumab for the treatment of non-small-cell lung cancer, N Engl J Med 372(21) (2015) 2018–28. [DOI] [PubMed] [Google Scholar]
- [5].Rosenberg JE, Hoffman-Censits J, Powles T, van der Heijden MS, Balar AV, Necchi A, Dawson N, O’Donnell PH, Balmanoukian A, Loriot Y, Srinivas S, Retz MM, Grivas P, Joseph RW, Galsky MD, Fleming MT, Petrylak DP, Perez-Gracia JL, Burris HA, Castellano D, Canil C, Bellmunt J, Bajorin D, Nickles D, Bourgon R, Frampton GM, Cui N, Mariathasan S, Abidoye O, Fine GD, Dreicer R, Atezolizumab in patients with locally advanced and metastatic urothelial carcinoma who have progressed following treatment with platinum-based chemotherapy: a single-arm, multicentre, phase 2 trial, Lancet 387(10031) (2016) 1909–1920. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [6].Motzer RJ, Escudier B, McDermott DF, George S, Hammers HJ, Srinivas S, Tykodi SS, Sosman JA, Procopio G, Plimack ER, Castellano D, Choueiri TK, Gurney H, Donskov F, Bono P, Wagstaff J, Gauler TC, Ueda T, Tomita Y, Schutz FA, Kollmannsberger C, Larkin J, Ravaud A, Simon JS, Xu LA, Waxman IM, Sharma P, CheckMate I, Nivolumab versus Everolimus in Advanced Renal-Cell Carcinoma, N Engl J Med 373(19) (2015) 1803–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [7].Nivolumab approved for lung cancer, Cancer Discov 5(5) (2015) OF1. [DOI] [PubMed] [Google Scholar]
- [8].Kim C, Prasad V, Cancer Drugs Approved on the Basis of a Surrogate End Point and Subsequent Overall Survival: An Analysis of 5 Years of US Food and Drug Administration Approvals, JAMA Intern Med 175(12) (2015) 1992–4. [DOI] [PubMed] [Google Scholar]
- [9].Miller AB, Hoogstraten B, Staquet M, Winkler A, Reporting results of cancer treatment, Cancer 47(1) (1981) 207–14. [DOI] [PubMed] [Google Scholar]
- [10].Therasse P, Arbuck SG, Eisenhauer EA, Wanders J, Kaplan RS, Rubinstein L, Verweij J, Van Glabbeke M, van Oosterom AT, Christian MC, Gwyther SG, New guidelines to evaluate the response to treatment in solid tumors. European Organization for Research and Treatment of Cancer, National Cancer Institute of the United States, National Cancer Institute of Canada, J Natl Cancer Inst 92(3) (2000) 205–16. [DOI] [PubMed] [Google Scholar]
- [11].Eisenhauer EA, Therasse P, Bogaerts J, Schwartz LH, Sargent D, Ford R, Dancey J, Arbuck S, Gwyther S, Mooney M, Rubinstein L, Shankar L, Dodd L, Kaplan R, Lacombe D, Verweij J, New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1), Eur J Cancer 45(2) (2009) 228–47. [DOI] [PubMed] [Google Scholar]
- [12].Cheson BD, Fisher RI, Barrington SF, Cavalli F, Schwartz LH, Zucca E, Lister TA, Alliance AL, G. Lymphoma, G. Eastern Cooperative Oncology, C. European Mantle Cell Lymphoma, F. Italian Lymphoma, R. European Organisation for, G. Treatment of Cancer/Dutch Hemato-Oncology, O. Grupo Espanol de Medula, G. German High-Grade Lymphoma Study, G. German Hodgkin’s Study, G. Japanese Lymphorra Study, A. Lymphoma Study, N.C.T. Group, G. Nordic Lymphoma Study, G. Southwest Oncology, I. United Kingdom National Cancer Research, Recommendations for initial evaluation, staging, and response assessment of Hodgkin and non-Hodgkin lymphoma: the Lugano classification, J Clin Oncol 32(27) (2014) 3059–68. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [13].Byrne MJ, Nowak AK, Modified RECIST criteria for assessment of response in malignant pleural mesothelioma, Ann Oncol 15(2) (2004) 257–60. [DOI] [PubMed] [Google Scholar]
- [14].Choi H, Charnsangavej C, Faria SC, Macapinlac HA, Burgess MA, Patel SR, Chen LL, Podoloff DA, Benjamin RS, Correlation of computed tomography and positron emission tomography in patients with metastatic gastrointestinal stromal tumor treated at a single institution with imatinib mesylate: proposal of new computed tomography response criteria, J Clin Oncol 25(13) (2007) 1753–9. [DOI] [PubMed] [Google Scholar]
- [15].Bruix J, Sherman M, Llovet JM, Beaugrand M, Lencioni R, Burroughs AK, Christensen E, Pagliaro L, Colombo M, Rodes J, E.P.o.E.o. HCC, Clinical management of hepatocellular carcinoma. Conclusions of the Barcelona-2000 EASL conference. European Association for the Study of the Liver, J Hepatol 35(3) (2001) 421–30. [DOI] [PubMed] [Google Scholar]
- [16].Bruix J, Sherman M, A.A.f.t.S.o.L.D. Practice Guidelines Committee, Management of hepatocellular carcinoma, Hepatology 42(5) (2005) 1208–36. [DOI] [PubMed] [Google Scholar]
- [17].Chiou VL, Burotto M, Pseudoprogression and Immune-Related Response in Solid Tumors, J Clin Oncol 33(31) (2015) 3541–3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [18].Wolchok JD, Hoos A, O’Day S, Weber JS, Hamid O, Lebbe C, Maio M, Binder M, Bohnsack O, Nichol G, Humphrey R, Hodi FS, Guidelines for the evaluation of immune therapy activity in solid tumors: immune-related response criteria, Clin Cancer Res 15(23) (2009) 7412–20. [DOI] [PubMed] [Google Scholar]
- [19].Nishino M, Giobbie-Hurder A, Gargano M, Suda M, Ramaiya NH, Hodi FS, Developing a common language for tumor response to immunotherapy: immune-related response criteria using unidimensional measurements, Clin Cancer Res 19(14) (2013) 3936–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [20].Bohnsack O, Hoos A, Ludajic K, Adaptation and modification of the immune related response criteria (IRRC): IrRECIST, Journal of Clinical Oncology 32(15) (2014). [Google Scholar]
- [21].Seymour L, Bogaerts J, Perrone A, Ford R, Schwartz LH, Mandrekar S, Lin NU, Litiere S, Dancey J, Chen A, Hodi FS, Therasse P, Hoekstra OS, Shankar LK, Wolchok JD, Ballinger M, Caramella C, de Vries EGE, Grp RW, iRECIST: guidelines for response criteria for use in trials testing immunotherapeutics, Lancet Oncology 18(3) (2017) E143–E152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [22].Persigehl T, Lennartz S, Schwartz LH, iRECIST: how to do it, Cancer Imaging 20(1) (2020) 2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [23].Landis JR, Koch GG, The measurement of observer agreement for categorical data, Biometrics 33(1) (1977) 159–74. [PubMed] [Google Scholar]
- [24].Nishino M, Cardarella S, Jackman DM, Ramaiya NH, Rabin MS, Hatabu H, Janne PA, Johnson BE, RECIST 1.1 in NSCLC patients with EGFR mutations treated with EGFR tyrosine kinase inhibitors: comparison with RECIST 1.0, AJR Am J Roentgenol 201(1) (2013) W64–71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [25].Jang HJ, Kim BC, Kim HS, Kim JH, Song HH, Kim JB, Park JJ, Yoon SN, Woo JY, Zang DY, Comparison of RECIST 1.0 and RECIST 1.1 on computed tomography in patients with metastatic colorectal cancer, Oncology 86(2) (2014) 117–21. [DOI] [PubMed] [Google Scholar]
- [26].Krajewski KM, Nishino M, Ramaiya NH, Choueiri TK, RECIST 1.1 compared with RECIST 1.0 in patients with advanced renal cell carcinoma receiving vascular endothelial growth factor-targeted therapy, AJR Am J Roentgenol 204(3) (2015) W282–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [27].Nishino M, Jackman DM, Hatabu H, Yeap BY, Cioffredi LA, Yap JT, Janne PA, Johnson BE, Van den Abbeele AD, New Response Evaluation Criteria in Solid Tumors (RECIST) guidelines for advanced non-small cell lung cancer: comparison with original RECIST and impact on assessment of tumor response to targeted therapy, AJR Am J Roentgenol 195(3) (2010) W221–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [28].Seyal AR, Gonzalez-Guindalini FD, Arslanoglu A, Harmath CB, Lewandowski RJ, Salem R, Yaghmai V, Reproducibility of mRECIST in assessing response to transarterial radioembolization therapy in hepatocellular carcinoma, Hepatology 62(4) (2015) 1111–21. [DOI] [PubMed] [Google Scholar]
- [29].Jeon MY, Lee HW, Kim BK, Park JY, Kim DY, Ahn SH, Han KH, Baek SE, Kim HS, Kim SU, Park MS, Reproducibility of European Association for the Study of the Liver criteria and modified Response Evaluation Criteria in Solid Tumors in patients treated with sorafenib, Liver Int 38(9) (2018) 1655–1663. [DOI] [PubMed] [Google Scholar]
- [30].Bellomi M, De Piano F, Ancona E, Lodigiani AF, Curigliano G, Raimondi S, Preda L, Evaluation of inter-observer variability according to RECIST 1.1 and its influence on response classification in CT measurement of liver metastases, Eur J Radiol 95 (2017) 96–101. [DOI] [PubMed] [Google Scholar]
- [31].Nishino M, Gargano M, Suda M, Ramaiya NH, Hodi FS, Optimizing immune-related tumor response assessment: does reducing the number of lesions impact response assessment in melanoma patients treated with ipilimumab?, J Immunother Cancer 2 (2014) 17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [32].Choi MH, Park GE, Oh SN, Park MY, Rha SE, Lee YJ, Jung SE, Choi JI, Reproducibility of mRECIST in Measurement and Response Assessment for Hepatocellular Carcinoma Treated by Transarterial Chemoembolization, Acad Radiol 25(11) (2018) 1363–1373. [DOI] [PubMed] [Google Scholar]
- [33].Mulkey F, Theoret MR, Keegan P, Pazdur R, Sridhara R, Comparison of iRECIST versus RECIST V.1.1 in patients treated with an anti-PD-1 or PD-L1 antibody: pooled FDA analysis, J Immunother Cancer 8(1) (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- [34].Tazdait M, Mezquita L, Lahmar J, Ferrara R, Bidault F, Ammari S, Balleyguier C, Planchard D, Gazzah A, Soria JC, Marabelle A, Besse B, Caramella C, Patterns of responses in metastatic NSCLC during PD-1 or PDL-1 inhibitor therapy: Comparison of RECIST 1.1, irRECIST and iRECIST criteria, Eur J Cancer 88 (2018) 38–47. [DOI] [PubMed] [Google Scholar]
- [35].Beer L, Hochmair M, Haug AR, Schwabel B, Kifjak D, Wadsak W, Fuereder T, Fabikan H, Fazekas A, Schwab S, Mayerhoefer ME, Herold C, Prosch H, Comparison of RECIST, iRECIST, and PERCIST for the Evaluation of Response to PD-1/PD-L1 Blockade Therapy in Patients With Non-Small Cell Lung Cancer, Clin Nucl Med 44(7) (2019) 535–543. [DOI] [PubMed] [Google Scholar]
- [36].Liang H, Xu Y, Chen M, Zhong W, Wang M, Zhao J, Patterns of response in metastatic NSCLC during PD-1 or PD-L1 inhibitor therapy: Comparison of the RECIST 1.1 and iRECIST criteria, Thorac Cancer 11(4) (2020) 1068–1075. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [37].Nishino M, Ramaiya NH, Chambers ES, Adeni AE, Hatabu H, Janne PA, Hodi FS, Awad MM, Immune-related response assessment during PD-1 inhibitor therapy in advanced non-small-cell lung cancer patients, J Immunother Cancer 4 (2016) 84. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [38].Katz SI, Hammer M, Bagley SJ, Aggarwal C, Bauml JM, Thompson JC, Nachiappan AC, Simone CB 2nd, Langer CJ, Radiologic Pseudoprogression during Anti-PD-1 Therapy for Advanced Non-Small Cell Lung Cancer, J Thorac Oncol 13(7) (2018) 978–986. [DOI] [PubMed] [Google Scholar]
- [39].Rizvi NA, Mazieres J, Planchard D, Stinchcombe TE, Dy GK, Antonia SJ, Horn L, Lena H, Minenza E, Mennecier B, Otterson GA, Campos LT, Gandara DR, Levy BP, Nair SG, Zalcman G, Wolf J, Souquet PJ, Baldini E, Cappuzzo F, Chouaid C, Dowlati A, Sanborn R, Lopez-Chavez A, Grohe C, Huber RM, Harbison CT, Baudelet C, Lestini BJ, Ramalingam SS, Activity and safety of nivolumab, an anti-PD-1 immune checkpoint inhibitor, for patients with advanced, refractory squamous non-small-cell lung cancer (CheckMate 063): a phase 2, single-arm trial, Lancet Oncol 16(3) (2015) 257–65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [40].Kazandjian D, Keegan P, Suzman DL, Pazdur R, Blumenthal GM, Characterization of outcomes in patients with metastatic non-small cell lung cancer treated with programmed cell death protein 1 inhibitors past RECIST version 1.1-defined disease progression in clinical trials, Semin Oncol 44(1) (2017) 3–7. [DOI] [PubMed] [Google Scholar]
- [41].Kuhl CK, Alparslan Y, Schmoee J, Sequeira B, Keulers A, Brummendorf TH, Keil S, Validity of RECIST Version 1.1 for Response Assessment in Metastatic Cancer: A Prospective, Multireader Study, Radiology 290(2) (2019) 349–356. [DOI] [PubMed] [Google Scholar]
- [42].Erasmus JJ, Gladish GW, Broemeling L, Sabloff BS, Truong MT, Herbst RS, Munden RF, Interobserver and intraobserver variability in measurement of non-small-cell carcinoma lung lesions: implications for assessment of tumor response, J Clin Oncol 21(13) (2003) 2574–82. [DOI] [PubMed] [Google Scholar]
- [43].Beaumont H, Evans TL, Klifa C, Guermazi A, Hong SR, Chadjaa M, Monostori Z, Discrepancies of assessments in a RECIST 1.1 phase II clinical trial - association between adjudication rate and variability in images and tumors selection, Cancer Imaging 18(1) (2018) 50. [DOI] [PMC free article] [PubMed] [Google Scholar]