Low inter-rater reliability in grading of rectal bleeding using NCI CTC and RTOG toxicity scales: a survey of radiation oncologists

Abstract

Purpose/Objective(s)

Rectal bleeding is one of the most common toxicities following prostate radiotherapy (RT), and both NCI CTC and RTOG grading scales are frequently used to report outcomes. We measured concordance among genitourinary radiation oncologists in using these scales to grade rectal bleeding.

Methods and Materials

From 6/2013–1/2014, a web-based survey was sent to 250 American and Canadian academic radiation oncologists who treat prostate cancer. Participants were provided 4 case vignettes where patients received RT and developed rectal bleeding and were asked for management plans and to rate the bleeding according to NCI CTC v.4 and RTOG late toxicity grading (scales provided). In 2 cases, participants were also asked if they would send the patient for colonoscopy. A multilevel, random intercept modeling approach was used to assess sources of variation (case, respondent) in toxicity grading to calculate the intraclass correlation coefficient (ICC). Agreement on a dichotomous grading scale (low grades 1–2 vs. high grades 3–4) was also assessed, using kappa statistic for multiple respondents.

Results

Seventy-two radiation oncologists (28%) completed the survey. Forty-seven (65%) reported having either written or been principal investigator on a study using these scales. Agreement between respondents was moderate (ICC=0.52, 95% CI 0.47–0.58) when using NCI CTC and fair using the RTOG scale (ICC=0.28, 95% CI 0.20–0.40). Respondents who chose an invasive management were more likely to select a higher toxicity grade (p<0.0001). Using the dichotomous scale, we observed moderate agreement (kappa=0.42, 95% CI 0.40–0.44) with the NCI CTC scale, but only slight agreement with the RTOG scale (kappa=0.19, 95% CI 0.17–0.21).

Conclusion

Low inter-rater reliability was observed among radiation oncologists grading rectal bleeding using two common scales. Clearer definitions of late rectal bleeding toxicity should be constructed to reduce this variability and avoid ambiguity in both reporting and interpretation.

Introduction

Radiation therapy (RT) is widely used today as a curative treatment modality for a number of cancers, including prostate cancer. Despite numerous advances in radiation technology and treatment planning methods, patients still experience acute and late sequelae of RT. These adverse effects are of equal importance as tumor control in the assessment of RT efficacy (1). Rectal bleeding is one of the most common late effects observed after prostate RT and considered a dose-limiting toxicity.

Today, radiation oncologists frequently use toxicity scales from the National Cancer Institute (NCI) and the Radiation Therapy Oncology Group (RTOG) to report adverse outcomes after prostate RT. First developed in the early 1980s, the NCI Common Toxicity Criteria for Adverse Events (CTCAE) is widely used as a classification and severity grading scale for adverse events in cancer clinical trials (2,3). Version 4 is the most up-to-date edition of the CTCAE and was first released in May 2009. The RTOG and European Organization for Research and Treatment of Cancer (EORTC) have a morbidity scoring schema which covers the majority of organs and tissues in the body which may develop late radiation effects (4,5). Variants of the RTOG grading scale have been in use since the 1970s. The NCI CTC and RTOG scales both rely on user interpretation of patients’ toxicity.

Despite the long use of these well-established scales, inter-observer differences may exist in the interpretation of the toxicity scoring, but have not been formally measured to date. In this study, we conducted a survey to elucidate how genitourinary radiation oncologists rate (using the NCI CTC and RTOG scales) and manage rectal bleeding following RT.

Methods

Participants and procedures

Approval was received from the Johns Hopkins University Institutional Review Board for this study. Potential subjects were identified from American and Canadian academic radiation oncology websites, as well from an online registry of clinical studies (www.clinicaltrials.gov), using search terms for prostate cancer radiotherapy (6). Subjects were considered eligible for this study if they were American or Canadian radiation oncologists within an academic setting who manage prostate cancer as a part of their routine clinical practice, and participants confirmed this within the questionnaire.

From 6/2013–1/2014, a web-based survey (created using www.surveymonkey.com) was sent to 250 American and Canadian academic radiation oncologists who treat prostate cancer. One reminder email was sent within 2–4 weeks if subjects did not initially respond. Subjects were offered a $50 gift card (choice of Amazon.com or Starbucks) upon their completion of the survey, and were entered into a drawing for $199 toward the purchase of a tablet e-reader/browser.

Study questionnaire

Participants were given 4 case vignettes where patients with prostate cancer developed rectal bleeding within approximately one year after completion of RT (see Appendix 1 for full survey). Three cases involved patients treated with IMRT, and 1 case was for a patient treated with brachytherapy. Vignettes were based on actual cases observed and selected to represent a spectrum of what is encountered in clinical practice. Survey format and questions were developed and revised with input from four radiation oncologists at our institution with regard to clarity and overall representativeness of clinical practice. Management options were included based on this feedback as well as a literature search in pubmed.com (terms: ‘rectal bleeding’, ‘management’ and ‘radiotherapy’). Participants were asked to rate the rectal bleeding according to NCI CTC v.4 and RTOG late toxicity grading using provided scales (Table 1). Both the NCI CTC and RTOG scales report toxicity on a 1–4 scale with Grade 1 being the mildest and Grade 4 being the most severe (7,8). In two cases, participants were asked if they would send the patient for colonoscopy; if yes, they were provided results and asked for a management plan. Management options included invasive procedures (argon plasma coagulation, formalin application, electrocautery), patient self-administered procedures (sucralfate, hydrocortisone, stool softeners, aminosalicylates, etc.), hyperbaric oxygen, and observation alone. Participants also had the option of writing in a different treatment of their choosing; the write-in treatments from respondents included: vitamin E, pentoxifylline, and referral/endoscopic management per a gastrointestinal specialist.

Table 1.

NCI CTC v.4 and RTOG late radiation morbidity schema for rectal bleeding.

Grade	NCI CTC: Rectal hemorrhage	RTOG: Small/Large Intestine
1	Mild; intervention not indicated	Mild diarrhea; mild cramping; bowel movement 5 times daily, slight rectal discharge or bleeding
2	Moderate symptoms; medical intervention or minor cauterization indicated	Moderate diarrhea and colic; bowel movement >5 times daily; excessive rectal mucus or intermittent bleeding
3	Transfusion, radiologic, endoscopic, or elective operative intervention indicated	Obstruction or bleeding requiring surgery
4	Life threatening consequences; urgent intervention indicated	Necrosis/perforation/fistula

Participants were also asked if they were or had been the principal investigator on a protocol or clinical trial that graded toxicities using either scale. If yes, participants were asked at what time point in the patient’s follow-up course did their most recent clinical trial base its toxicity scoring, and which scale was used.

Statistical analysis

Statistical analyses were performed on NCI CTC and RTOG data separately. A multilevel, random intercept modeling approach was used to assess sources of variation (case, respondent) in toxicity grading to calculate the intraclass correlation coefficient (ICC). The ICC was defined as the proportion of the between-subject variance to the sum of the between- and within-subject variances of a variable and generally ranges between 0–1. The ICC is interpreted as ‘poor’ (ICC<0.0), ‘slight’ (0.0≤ICC≤0.2), ‘fair’ (0.21≤ICC≤0.4), ‘moderate’ (0.41≤ICC≤0.6), ‘substantial’ (0.61≤ICC≤0.8), and ‘almost perfect’ (0.81≤ICC≤1.0) (9,10). The 95% bootstrap confidence interval for ICC was estimated using the percentile method based on 500 bootstrap samples. The respondents’ management plan was subsequently added to the model as a fixed effect; treatment options were dichotomized as invasive (eg. GI procedures) or non-invasive (eg. medication or observation alone); this only pertained to participants who chose to do a colonoscopy/sigmoidoscopy in the prior question. The agreement on low (grades 1–2) vs. high (grades 3–4) toxicity grading was assessed using the kappa statistic for multiple respondents. The interpretation used for the ICC applies to the kappa statistic as well (9,10). All statistical tests were two-sided and considered statistically significant at P<0.05. The analyses were carried out using SAS software (version 9.3, SAS Institute, Cary, NC).

Results

From June 2013 to January 2014, 250 radiation oncologists were invited to participate and 72 (28%) completed the survey. Forty-seven respondents (65%) had written or been principal investigator on a clinical trial or protocol that graded toxicity. Table 2 describes the reported details of the respondents’ studies. Of note, the timing of toxicity evaluation varied; 34 (72%) participants reported that toxicity was reported at each time point, 12 (26%) reported only the highest toxicity experienced by the patient at any time during the study, while 1 (2%) reported toxicity at last follow-up.

Table 2.

Clinical trial/protocol information from the 47 responders having either written or having been principal investigator on a study of prostate radiotherapy.

Question	Response	N (%)
Which toxicity grading system does your most recent or current clinical trial utilize for late toxicity?	NCI CTC v.3	7 (15%)
	NCI CTC v. 4	20 (43%)
	RTOG	9 (19%)
	Both NCI and RTOG	10 (21%)
	Expanded Prostate Cancer Index Composite (EPIC)	1 (2%)
At what time point in the patient’s follow-up course does/did your most recent clinical trial base its toxicity scoring?	Toxicity is/was reported as experienced by patient at each time point (i.e. separate grade assigned and reported at 12 months, 24 months, 36 months, etc. as necessary)	34 (72%)
	Toxicity is/was reported only as experienced by patient at time of last follow-up	1 (2%)
	Toxicity is/was reported only as highest grade experienced by patient at any time point	12 (26%)

The frequency distributions of toxicity grading across the four cases are summarized in Table 3. Overall, the agreement between respondents was ‘moderate’ (ICC=0.52, 95% CI 0.47–0.58) when using NCI CTC for toxicity grading and ‘fair’ using the RTOG scale (ICC=0.28, 95% CI 0.20–0.40). With the NCI CTC scale, 52% of the variability in toxicity grading was attributable to cases in our study, leaving 48% of the variability in toxicity grading to be accounted for by raters; with the RTOG scale, 28% of the variability in toxicity grading was attributable to cases, leaving 72% of the variability to be accounted for by raters. For low vs. high toxicity grading, we observed a ‘moderate’ agreement (kappa=0.42, 95% CI 0.40–0.44) with the NCI CTC scale, but only ‘slight’ agreement with the RTOG scale (kappa=0.19, 95% CI 0.17–0.21).

Table 3.

Responses by case and toxicity criteria.

Toxicity criteria	Toxicity grading		N (%)
			Case 1	Case 2	Case 3	Case 4
NCI CTC	Low	1	10 (14.1) 53 (74.6) 8 (11.3) 0 (0)	1 (1.4) 46 (64.8) 24 (33.8) 0 (0)	1 (14.3) 36 (51.4) 24 (34.3) 0 (0)	0 (0) 0 (0) 53 (75.7) 17 (24.3)
		2
	High	3
		4
RTOG	Low	1	16 (22.2) 52 (72.2) 4 (5.6) 0 (0)	3 (4.2) 58 (81.7) 10 (14.1) 0 (0)	22 (31.4) 38 (54.3) 10 (14.3) 0 (0)	0 (0) 32 (45.7) 37 (52.9) 1 (1.4)
		2
	High	3
		4

Management plans were assessed in two cases, with Case 1 describing a patient with bloody streaks in the stool 2–3 times a week, and Case 2 with a cupful of blood in the stool every other day. In both cases, the majority of respondents said they would send the patient with rectal bleeding for a colonoscopy/sigmoidoscopy (Table 4). Respondents’ decisions of interventional procedures vs. patient self-administered procedures or observation alone is also detailed in Table 4. Respondents who chose an invasive management plan were more likely to choose a higher toxicity grading (p<0.0001).

Table 4.

Responders’ choices on whether to send the patient for colonoscopy/sigmoidoscopy (Cases 1 and 2) and subsequent management plans chosen by participants.

	Case 1 (n, %) *	Case 2 (n, %) **
Would you send this patient for colonoscopy?
Yes, regardless of prior colonoscopy/sigmoidoscopy	33 (45%)	63 (87%)
Yes, unless he already had one (with no significant findings) within past year	18 (25%)	4 (6%)
Yes, unless he already had one (with no significant findings) within past 3 years	9 (13%)	0 (0%)
Yes, unless he already had one (with no significant findings) within past 5 years	5 (7%)	0 (0%)
No, would not send for colonoscopy/sigmoidoscopy	7 (10%)	5 (7%)
Management plans of responders who chose sigmoidoscopy/colonoscopy:
Interventional procedures (argon, endoscopic management, GI referral, formalin, electrocautery, hyperbaric oxygen)	20 (27%)	36 (50%)
Patient self-administered therapy (sulcralfate, hydrocortisone, stool softeners, aminosalicylate) and observation alone	45 (63%)	30 (42%)
No colonoscopy/sigmoidoscopy was chosen in the previous question	7 (10%)	5 (7%)

^*Case 1 described a patient with “bloody streaks on his stools every 2–3 days for the past 2 months, with no associated pain. He estimates there is about 2–3 teaspoons of blood each time it occurs, with no clots.” The colonoscopy report describes multiple ectatic blood vessels within the rectum consistent with radiation proctitis, with friable mucosa but no active bleeding.

^**Case 2 described a patient with “bloody stools approximately every other day for the past 2 weeks. He estimates there is about a cupful of blood each time it occurs, with occasional clots. Sometimes he has felt like he had to pass gas, but ended up passing blood instead and had to change his clothing.” The colonoscopy report describes within the rectum, telangiectasias and edematous mucosa overlying the area of the prostate, consistent with radiation proctitis; no active bleeding visualized. Non-bleeding internal hemorrhoids were also noted.

Discussion

One of the most common complications following RT for prostate cancer is chronic rectal bleeding. The reporting of rectal bleeding and other such toxicities are either performed objectively by looking at measures including frequency, duration, and quantity, or subjectively with validated morbidity scales. Published morbidity rates are dependent on the specific schema used to grade and report the adverse event. Indeed, it has been shown that there is variability among those who evaluate the levels of toxicity seen in patients after cancer treatments. One study examined how data managers grade gastrointestinal, neurologic, hematologic, and genitourinary toxicity in simulated patients with bladder cancer treated with concurrent cisplatin and RT (11). The inter-rater agreement (agreement between multiple raters of the same case) and intra-rater agreement (agreement within the same rater on one case rated on multiple occasions) were assessed to determine the reliability of two toxicity scales but the results showed only moderate intra- and inter-rater reliability.

In this study, we compared the NCI CTC and RTOG late effects schema for rectal bleeding by surveying a cohort of academic genitourinary American and Canadian radiation oncologists. We selected these scales because they are commonly used to report rectal bleeding in the prostate radiotherapy literature, as supported by the responses received from those involved as investigators in prostate radiotherapy studies (Table 2), as well as recent reports in the literature of prostate radiotherapy outcomes (12–16). Although other scales are also in existence, it was not deemed feasible to include all scales in our survey while maintaining acceptable survey participation rates and adequacy of statistical evaluation for each scale. We chose the four case vignettes in this study to represent the wide spectrum of rectal bleeding generally observed after radiation therapy. It is possible that a different selection of cases would have changed the rates of inter-observer agreement, and a larger number of cases in the questionnaire would have reduced any potential case-specific biases, but again a longer survey would have likely resulted in lower participation rates. We did not have specific thoughts on the ‘correct’ toxicity grade in each case, given that any attempt to define the appropriate grading would simply be our own personal interpretation. Based on the results of our survey, the NCI CTC and RTOG late effects scoring systems differ in their performance, with the NCI CTC scale having better agreement between respondents than the RTOG scale. The NCI CTC scale also had higher levels of agreement when looking at low vs. high toxicity grading. However, even for the NCI scale, the inter-observer ICC was only 0.52, which is considered ‘moderate’ agreement; the RTOG ICC inter-observer ICC of 0.28 would be rated only as ‘fair.’ There was, however, a correlation between invasive management plans chosen and a higher toxicity rating.

Interestingly, the study design of the clinical trials/protocols that our respondents had been principal investigators of also differed in terms of the timing of toxicity evaluation. Most of the clinical trials recorded toxicity at each time point, but others reported only the highest or last toxicity experienced by the patient. The responses we received demonstrate the potential discrepancies of toxicity scoring, both in our survey and in clinical research as well. Based on these the inter-observer variability observed in this study, the utility of comparing rectal bleeding toxicities across published studies utilizing these scales is questionable, even when such studies utilize the same scale. Not only are there potential heterogeneities between different investigators grading and reporting outcomes, similar differences will exist between reader interpretation of such results.

Several prior assessments of rectal complications after prostate radiotherapy have focused on RTOG and the LENT/SOMA scales. The LENT/SOMA (Late Effects in Normal Tissue/Subjective, Objective, Management, and Analytic) toxicity scale was proposed in 1995 as a replacement of the existing RTOG/EORTC scales for the recording of RT late effects (17). One study of 352 prostate cancer patients treated with conformal RT stated that 0.7% of patients had grade 3–4 complication with RTOG while 2% had grade 3–4 complications (18). However, when they modified the LENT/SOMA scales to include bleeding that required blood transfusion or coagulation as grade 3, the complication rate increased to 6%. In a separate study, Boersma et al. estimated the incidence of late bladder and rectal complications after high-dose (70–78 Gy) conformal RT for prostate cancer using the RTOG and LENT/SOMA systems (19). They found less discrepancy between the two systems in the scoring of GI symptoms, but needed to exclude urinary frequency from the GU symptoms to get good agreement between the two systems for genitourinary toxicity. These studies all showed differences in the toxicity grading of rectal bleeding when using the RTOG and LENT/SOMA scales. Other studies examining toxicity after irradiation for prostate cancer have examined other toxicity scales, including the UCLA Prostate Index. Livsey et al. surveyed patients who had received radical RT to 50 Gy using the LENT/SOMA and UCLA Prostate Index to evaluate late effects of RT (20). They found a significant correlation between the LENT/SOMA and UCLA scales in the toxicity ratings by patients.

To our knowledge, few published studies have compared the NCI CTC and RTOG systems for the reporting of adverse effects after RT. Denis et al. compared the LENT/SOMA, NCI CTC, and RTOG/EORTC toxicity scales to prospectively assess the late toxicity rates seen after chemoradiation vs. RT alone for carcinoma of the pharynx (21). They found that the transposability of symptoms was significantly higher with the LENT/SOMA or the RTOG/EORTC scaling systems than with the NCI CTC system, but ultimately noted that the late toxicity assessment of a treatment may depend on the specific scale used for reporting. Another study of radiation-induced toxicity after treatment for non-small cell lung cancer compared the NCI CTC and RTOG morbidity scales and showed that the reported toxicity differed depending on the scoring system used (22). Our study provides further support for the idea that toxicity evaluations are inherently subjective, and comparisons of toxicities rated with different scales should be made with caution.

Notably, one striking difference between the NCI CTC and RTOG morbidity scales is that the NCI CTC addresses “rectal hemorrhage” alone, while the RTOG scale is non-specific for rectal bleeding (Table 1). The RTOG scale includes a variety of “small/large intestine” toxicities such as mucus discharge, bowel obstruction, diarrhea, and cramping. Our survey yielded results concordant with this inherent difference. It is possible that the differing bowel symptoms in the RTOG scale served as distractors from the description of rectal bleeding, ultimately contributing to the lower agreement between respondents using the RTOG scale compared to the NCI CTC late effects scale. This issue of “extraneous” toxicities that are not immediately pertinent to the toxicity in question has been examined in other studies, including Hanlon et al. and Boersma et al.’s work on the LENT/SOMA scales (18,19). These two studies modified the scales and achieved better agreement on rectal toxicity.

Toxicity and late effects are increasingly important in the evaluation of cancer treatments and trials. This is particularly true for prostate cancer treatment as patients have an assortment of treatment options to choose from, all with differing adverse outcomes. More quantitative measures of toxicity and late effect severity should be created for better reporting of radiation morbidity experienced by patients, and perhaps the existing scales should be reevaluated more often than they are currently. The timing of toxicity evaluations should also be standardized across clinical trials for better and more uniform comparisons of toxicity data. There has also been contemporary interest in measuring and reporting patients’ perceptions and experiences of toxicities, as well as work performed to correlate those patient-reported outcomes with grades assigned by physicians (23,24). Patient-reported outcomes may more accurately reflect the patient’s experience; however, these are complimentary to physician-reported toxicity scales. The reporting of clinical trial outcomes continues to utilize toxicity scales such as the NCI and RTOG systems, necessitating the need for standardization of such reporting (12–16, 25). One method for such standardization would be to encourage formal training in the use of toxicity rating scales, via a self-learning module. Future work in toxicity reporting could investigate whether a training module could increase inter-rater agreement between users of these late effects scales.

It is important to note that our study involved only the assessment of one aspect of toxicity grading, specifically late rectal bleeding. Given the similarities within these systems for grading of various non-rectal toxicities, it is possible that the variability observed here may also be present when raters are assessing other toxicities; however, our study is not sufficient for making conclusions beyond the grading of rectal bleeding. Also, although the cases presented in the survey were felt to be reflective of the variety of presentations seen in clinical practice, the inclusion of different case vignettes may have resulted in somewhat different rates of agreement between raters. Future studies that increase this dimension by incorporating a variety of cases will help develop robust estimates for inter-rate agreement.

Conclusion

Wide heterogeneities were observed among radiation oncologists grading rectal bleeding using two common scales. These data highlight the subjectivity and limitations of these tools for the reporting and evaluation of rectal toxicity and strongly suggest that clearer definitions be constructed to reduce this variability.

Summary.

A survey of genitourinary radiation oncologists was conducted to assess the NCI CTC and RTOG late toxicity scales for rectal bleeding after prostate radiotherapy. Participants graded toxicity in 4 patient cases using both scales (scales were provided for reference). Wide heterogeneities were observed among the participants grading rectal bleeding using two common scales. Clearer definitions for toxicity grading should be constructed to reduce this variability for both toxicity reporting and interpretation.