Abstract
Objectives:
To investigate uniformity and pitfalls in structured radiological staging of rectal cancer.
Methods:
Twenty-one radiologists (12 countries) staged 75 rectal cancers on MRI using a structured reporting template. Interobserver agreement (IOA) was calculated as the percentage agreement between readers (categorical variables) and Krippendorff’s α (continuous variables). Agreement with an expert consensus served as a surrogate standard of reference to estimate diagnostic accuracy. Polychoric correlation coefficients were used to assess correlations between diagnostic confidence and accuracy (=agreement with expert consensus).
Results:
Uniformity to diagnose high-risk (≥cT3 ab) versus low-risk (≤cT3 cd) cT-stage, cN0 versus cN+, lateral nodes and tumour deposits, MRF and sphincter involvement, and solid versus mucinous tumours was high with IOA > 80% in the majority of cases (and >80% agreement with expert consensus). Results for assessing extramural vascular invasion, cT-stage (cT1-2/cT3/cT4a/cT4b), cN-stage (cN0/N1/N2), relation to the peritoneal reflection, extent of sphincter involvement (internal/intersphincteric/external) and morphology (solid/annular/semi-annular) were considerably poorer. IOA was high (α = 0.72–0.84) for tumour height/length and extramural invasion depth, but low for tumour-MRF distance and number of (suspicious) nodes (α = 0.05–0.55). There was a significant positive correlation between diagnostic confidence and accuracy (=agreement with expert consensus) (p < 0.001-p = 0.003).
Conclusions:
- Several staging items lacked sufficient reproducibility.
- Results for cT- and N-staging g improved when using a dichotomized stratification.
- Considering the significant correlation between diagnostic confidence and accuracy, a confidence level may be incorporated into structured reporting for specific items with low reproducibility.
Advances in knowledge:
Although structured reporting aims to achieve uniformity in reporting, several items lack sufficient reproducibility and might benefit from dichotomized assessment and incorporating confidence levels.
Introduction
MRI is the main diagnostic technique for local tumour staging in rectal cancer. Primary goals of MRI are to establish the presence of key risk factors associated with local recurrence to determine the need for neoadjuvant treatment, and to assess the local invasion of tumours into surrounding organs and structures to help guide the surgical strategy. To ensure that these key elements of staging are adequately represented in radiological reports, various organizations such as the European Society of Gastrointestinal and Abdominal Radiology (ESGAR), Society of Abdominal Radiology (SAR), Radiological Society of North America (RSNA), and different national radiological societies, have introduced standardized (structured) reporting templates that are typically largely based on the Tumour Node Metastasis (TNM) staging system proposed by the American Joint Committee on Cancer (AJCC) / Union for International Cancer Control (UICC). 1–4
Recent studies have shown that radiologists have increasingly adopted these structured reporting templates for routine clinical reporting, 3,5–7 which has led to enhanced completeness of reporting and improved satisfaction levels of referring clinicians. 5,7–10 Another goal of structured reporting is to increase the overall level of uniformity in radiological reporting. To what extent this is successfully accomplished, is not well documented. We know from previous literature that MRI has its limitations, for example when it comes to lymph node staging. 11 In addition, there are several other pitfalls and controversies that may lead to inconsistencies in reporting despite the availability of standardized reporting templates. 12,13
This multicentre study aims to investigate the level of uniformity in the radiological staging of rectal cancer using a structured reporting template, and establish its main limitations and areas for improvement on a large scale, by testing the reproducibility among a group of more than 20 radiologists from different nationalities and with different clinical expertise levels.
Methods
Study design
This study concerns a retrospective multicentre diagnostic study approved by the local institutional review board of the principal investigating centre. Informed consent was waived.
Patient and study reader accrual
Twenty-one radiologists from 12 countries participated as study readers. Readers were accrued via an open call to the ESGAR membership (in particular members with a known interest in rectal cancer imaging). The study included n = 75 patients who were treated for newly diagnosed rectal cancer in 1 out of 10 centres in the Netherlands (one university hospital, eight teaching hospitals and one comprehensive cancer centre). Patients were selected from an existing and previously published multicentre study database 7,14,15 based on the following inclusion criteria; [1] biopsy-proven rectal carcinoma, and [2] availability of diagnostic quality primary staging MRI including at least T 2-weighted sequences in three planes (sagittal, coronal, transversal). To ensure a clinically representative sample of cases, patients were selected semi-randomly so that data from all 10 study centres were represented in the study cohort with sufficient variety in terms of clinical tumour stage.
MR imaging
MRI exams were carried out following the local protocols of the participating study centres at the time of inclusion. From the full-scan protocols, we selected 2D T 2-weighted sequences in sagittal, oblique-axial (perpendicular to the tumour axis), and oblique-coronal (parallel to the tumour axis) planes as these sequences are the minimum requirement recommended for primary rectal cancer staging as outlined in recent guidelines. 1 Slice thickness ranged between 3 and 5 mm and in-plane resolution ranged between 0.35 × 0.35 and 0.94 × 0.94 mm.
Image evaluation
Images were evaluated using a web-based platform (iScore) designed by one of the authors (NeK) that combines the Open Health Imaging Foundation (OHIF) DICOM viewing platform 16 with customizable electronic case report forms (eCRFs). For the current study, the structured reporting template for primary rectal cancer staging published by ESGAR was converted into an eCRF, comprised of 20 staging items, including 14 categorical staging variables and 6 continuous variables (listed in detail in Table 1). Readers were asked to complete the staging eCRF for each study case. For the variables cT-stage, cN-stage, MRF involvement, EMVI, and the presence of sphincter invasion, a confidence level was included in the staging (Table 1). Links to relevant background information, including TNM staging definitions and the ESGAR consensus guidelines on MRI for rectal cancer, were provided in iScore. 1,4,17 Readers were blinded to each other’s results and outcomes of surgery and histopathology.
Table 1.
Staging variables derived from the ESGAR structured reporting template that were included in the electronic case report form in iScore
| Main staging variables used for risk stratification | |||
|---|---|---|---|
| Variable | Answer options | ||
| Categorical | 1 | cT-stage |
|
| 2a | In case of cT3 tumour, cT3 sub-stage |
|
|
| 3 | Dichotomous cT-stage classification |
|
|
| 4 | cN-stage |
|
|
| 5 | Dichotomous cN-stage classification |
|
|
| 6 | Lateral nodes |
|
|
| 7 | Tumour deposits |
|
|
| 8 | MRF involvement |
|
|
| 9 | EMVI |
|
|
| Continuous | 10 | Total number of visible mesorectal nodes | Number |
| 11 | Total number of suspicious mesorectal nodes | Number | |
| 12 | Tumour-MRF distance | …. mm | |
| Other staging variables | |||
| Variable | Answer options | ||
| Categorical | 13 | Morphology (shape) |
|
| 14 | Morphology (composition) |
|
|
| 15 | Sphincter invasion |
|
|
| 16b | In case of sphincter involvement, level of involvement |
|
|
| 17 | Relation to peritoneal reflection |
|
|
| Continuous | 18 | Tumour length | …. mm |
| 19 | Tumour height | …. mm | |
| 20 | Extramural invasion depth (in cT3-4 tumours) | …. mm | |
T3 substage was not evaluated for all patient cases, but only when readers assigned a T3 stage
Level of sphincter involvement was not evaluated for all patient cases, but only when readers indicated a suspicion for sphincter involvement
Statistical analysis and standard of reference
Statistical analyses were performed using R statistics version 4.1.0 (2021) and IBM SPSS version 27 (2020). Group interobserver agreement (IOA) for the continuous variables was calculated using Krippendorff’s α. For the categorical variables, the percentage agreement between study readers was calculated and grouped into items with suboptimal agreement (<60%), moderate agreement (60%–80%) and good agreement (>80%). Correlation with histopathology was only available for a minority of patients (and for only a few of the studied staging variables) since most patients underwent neoadjuvant treatment prior to surgery. As such, two rectal MRI experts (DL and LC-S; each with >10 years of dedicated experience in rectal MRI and with a background in rectal cancer research and teaching) staged all study cases to establish a surrogate standard of reference. The two experts staged the cases independently. In case of any disagreement, a third independent expert (ML, with similar >10 years of dedicated experience in rectal MRI) was consulted to reach final expert consensus. IOA and diagnostic accuracy (= agreement with the expert reference) were calculated for the 21 study readers. For staging items 3, 5, 8, 9 and 15 (Table 1), confidence scores were dichotomized to calculate diagnostic accuracy with the cutoff between equivocal and probably involved/positive. Polychoric correlation coefficients were calculated between the various confidence level scores and diagnostic accuracy. Pearson’s chi-squared test was used to calculate p-values. The level of significance was set at p < 0.05.
Results
Patient and study reader characteristics
Forty-six patients (61%) were male, median age was 64 (range 40–82) years. The study readers had a median of 11 years of experience as a radiologist after completion of residency training (range 2–28 years). Further patient and study reader characteristics are provided in Table 2.
Table 2.
Patient and study reader characteristics
| N= | % | ||
|---|---|---|---|
| Patients | Total | 75 | 100% |
| Median age | 64 years (range 40–82) | ||
| Sex | Male | 46 | 61% |
| Female | 29 | 39% | |
| Clinical tumour stage and treatment | Surgery only (early stage) |
10 | 13% |
| Short course radiotherapy (intermediate stage) | 7 | 9% | |
| Long course (chemo)radiotherapya (advanced stage) | 58 | 77% | |
| Study readers | Total | 21 | 100% |
| Years after completion of residency training | <5 years | 5 | 24% |
| 5–10 years | 6 | 29% | |
| >10 years | 10 | 48% | |
| Workplace | Comprehensive cancer centre | 8 | 35% |
| University hospital | 4 | 19% | |
| General hospital | 7 | 33% | |
| Other | 2 | 10% | |
| Number of rectal cancer cases reported on a yearly basis | < 50 | 1 | 5% |
| 50–100 | 5 | 24% | |
| > 100 | 15 | 71% | |
| Country of origin | United Kingdom | 5 | 24% |
| Brazil | 2 | 10% | |
| India | 2 | 10% | |
| Italy | 2 | 10% | |
| Romania | 2 | 10% | |
| Switzerland | 2 | 10% | |
| The Netherlands | 2 | 10% | |
| Chile | 1 | 5% | |
| Georgia | 1 | 5% | |
| Germany | 1 | 5% | |
| Israël | 1 | 5% | |
| Sweden | 1 | 5% | |
Including 57 cases undergoing long course chemoradiation and 1 case undergoing 5 × 5 Gy radiotherapy with a prolonged waiting interval to surgery
Interobserver agreement
Table 3 shows the mean IOA for the categorical staging variables (as well as the diagnostic accuracy estimates based on the expert standard of reference). For most of the categorical staging items, good (>80%) agreement was reached for the majority of patient cases. For three items – cTstage, cN-stage, and morphology/shape – IOA was <80% in the majority of cases with poorest results for cN-staging where good IOA was achieved in only 28% of the cases. For the six continuous variables, IOA (Krippendorff’s α) was α = 0.72 for tumour length, α = 0.84 for tumour height, α = 0.33 for the tumour-MRF distance, α = 0.73 for extramural invasion depth, α = 0.55 for the total number of visible mesorectal nodes, and α = 0.05 for the total number of suspicious mesorectal nodes. Good IOA was reached for the majority (71%) of cases to detect the presence of any high risk feature (i.e. ≥T3 cd stage, N+, tumour deposits, or EMVI).
Table 3.
Mean diagnostic accuracy and interobserver agreement for the main categorical staging variables
| Staging variables a | Interobserver agreement | Agreement with expert reference | Reference standard (i.e., results of expert consensus reading; total n = 75) | ||
|---|---|---|---|---|---|
| Suboptimal (<60%) | Moderate (60–80%) | Good (>80%) | |||
| I – Main staging variables used for risk stratification | |||||
| cTstage (cT0/12/3/4a/4b) | 16% | 40% | 44% | 69% | 13 cT1-2, 39 cT3, 10 cT4a, 13 cT4b |
| dichotomized cT-stage (≤cT3 ab vs. ≥cT3 cd) | 17% | 23% | 60% | 81% | 33 low risk, 42 high risk |
| cN-stage (cN0/1/2) | 40% | 32% | 28% | 60% | 21 cN0, 26 cN1, 28 cN2 |
| dichotomized cN-stage (cN0/cN+) | 9% | 33% | 57% | 80% | 21 cN0, 54 cN+ |
| lateral N + nodes (yes/no) | 5% | 19% | 76% | 88% | 16 Yes, 59 No |
| tumour deposits (yes/no) | 7% | 19% | 75% | 86% | 13 Yes, 62 No |
| MRF involvement (yes/no) | 13% | 23% | 64% | 82% | 28 Yes, 47 No |
| EMVI (yes/no) | 9% | 36% | 55% | 77% | 30 Yes, 45 No |
| Ia – Assessing the presence of high risk disease | |||||
| Presence of any (≥1) high risk feature (≥T3 cd, N+, tumour deposits, or EMVI) | 9% | 20% | 71% | 84% | 60 Yes, 15 No |
| II – Other staging variables | |||||
| morphology – shape (annular/semi-annular/polypoid) | 29% | 31% | 40% | 67% | 29 annular, 36 semi-annular, 10 polypoid |
| morphology – composition (solid/mucinous/mixed) | 8% | 27% | 65% | 79% | 58 solid, five mucinous, 12 mixed |
| relation to peritoneal reflection (above/below/straddling) | 12% | 23% | 65% | 77% | ten above, 41 below, 24 straddling |
| sphincter invasion (yes/no) | 5% | 16% | 79% | 91% | 16 Yes, 59 No |
Note, diagnostic accuracies represent the averages for all cases and readers combined.
The categorical staging variables cT3 substage (cT3ab vs cT3cd) and level of sphincter involvement (internal, intersphincteric plane, external sphincter) are not included in this table as these were only available for varying subsets of patients depending on assigned cT-stage and presence of sphincter invasion.
Agreement with expert reference
Good (>80%) agreement with the expert standard of reference (range 80–91%) was achieved for the dichotomized assessment of low- versus high-risk cT-stage and cN0 vs N + disease, for assessing the presence of lateral nodal metastases, tumour deposits, MRF involvement, and the presence of sphincter invasion. Results for multicategorical cT-staging and cN-staging were considerably lower (69 and 60% agreement with expert reference), as were results for EMVI (77%), tumour morphology (67–79%) and relation to the anterior peritoneal reflection (77%). In patients with suspected sphincter involvement, mean agreement with the expert reference to assess the level of involvement (internal sphincter, intersphincteric plane, external sphincter) was 51%. In patients staged as cT3, accuracy for cT3 subclassification into low risk (cT3ab) vs high risk (cT3cd) was 73%.
Correlation with diagnostic confidence
A significant positive correlation between diagnostic confidence and agreement with the expert reference was found for all staging items for which diagnostic confidence scores were available (Table 1) with polychoric correlation coefficients ranging from 0.18 to 0.66 (p < 0.001-p = 0.003). Strongest effects (correlation coefficient>0.50) were found for the presence of sphincter involvement, low versus high-risk cT-stage, cN0 vs N + stage, EMVI and MRF involvement. Confidence scores were lowest for assessing the level of sphincter involvement.
Main problem areas
Morphology (shape)
As shown in Table 3, readers showed relatively poor IOA and agreement with an expert reference to differentiate between annular, semi-annular and polypoid tumours. Readers were mainly inconsistent in discerning semi-annular from polypoid tumours (accuracy 62–64%) (Figure 1).
Figure 1.
Example of two cases that were scored as semi-annular by the two expert readers. There was large variation among the study readers. Case a was scored by 29% as annular, by 48% as polypoid, and by 24% as semi-annular. Case b was scored by 10% as annular, by 43% as polypoid, and by 48% as semi-annular.
cT-staging
Agreement with the expert reference for cT-staging was 69%, which was considerably lower than for dichotomous evaluation of low-risk (cT1-2-3ab) versus high-risk (cT3cd-4) cT-stage (agreement 81%). IOA was also better for the dichotomous evaluation. Results were poorest for assessment of cT4a (54% agreement with expert reference) and cT4b tumours (59% agreement), with understaging rates of on average 30–34% (example shown in Figure 2).
Figure 2.
Example of a case that was staged as cT4a by expert consensus because of involvement of the peritoneum on the left anterior side, above the level of the anterior peritoneal reflection (arrowheads in a). There is simultaneous focal involvement of the MRF below the level of the anterior peritoneal reflection (arrow in b). The study readers unanimously scored this case as MRF+ (100%), but 81% of readers scored it as cT3 and only 19% as T4a.
cN-staging
Agreement with the expert reference for dichotomized assessment of cN0/N + stage was 80%, which was considerably higher than for multicategorical cN-staging as cN0/1/2 (agreement 60%). The number of detected positive mesorectal lymph nodes varied widely, as reflected by the very low group agreement in defining the number of suspicious mesorectal nodes (α = 0.05).
EMVI
IOA to assess EMVI was relatively low and >80% consensus between readers was only reached in 55% of the study cases, with a mean agreement with the expert reference of 77%. Agreement with the expert reference to diagnose EMVI+tumours (69%) was poorer than for the diagnosis of EMVI- tumours (82%), see Figure 3.
Figure 3.
Sagittal image (a) showing a stenosing tumour in the mid-rectum with corresponding cross-section (b) at the mid-tumour level. Adjacent to the tumour, there is some desmoplastic stranding as well as several small vessels that radiate outward from the edge of the muscularis propria into the perirectal fat (arrows in B). Because no tumour signal extends into, interrupts or expands these vessels, this case was scored as EMVI-negative by expert consensus. There was however, considerable variation among the study readers; 57% considered this case as EMVI+, 33% as EMVI- and 10% assigned an equivocal score.
Level of sphincter involvement
IOA to assess the presence of sphincter involvement (yes/ no) was high but results to assess the level of sphincter involvement were considerably poorer with an agreement of only 51% compared to the expert reference. This was probably (partly) related to suboptimal image angulation: when re-reviewing all the cases with suspected sphincter involvement, a dedicated coronal sequence angled parallel to the anal canal was only available in 38% of these cases (Figure 4).
Figure 4.
Example of a case with suspect sphincter involvement. The coronal sequence is angled parallel to the distal rectum (rather than to the anal canal), making it more difficult to assess the level of sphincter involvement. The two experts assessed this case as suspicious for intersphincteric involvement, mainly based on the transverse sequence (arrow). Results of the study readers were highly variable; 19% considered it as no sphincter invasion, 19% as internal sphincter invasion only, 19% as extending into the intersphincteric space, and 43% as extending into the external sphincter.
Tumour-MRF distance
IOA (and agreement with the expert reference) to assess an involved MRF were high, but there was large variation in measuring the tumour-MRF distance with an IOA of only α = 0.33 (two examples are shown in Figure 5).
Figure 5.
Examples of two cases where there was substantial variation in tumour-MRF distance among readers. The upper row shows the sagittal (a) and transverse (b) images of a male patient with an upper rectal tumour, situated above the level of the anterior peritoneal reflection. Six out of 21 readers (29%) erroneously interpreted the anterior invasion of the peritoneum as MRF invasion and measured a tumour-MRF distance of 0 mm, while other readers (and the expert reference) measured the distance from the tumour to dorsal MRF as >10 mm. The bottom row (sagittal c, transverse d) shows a male patient with a mid-rectal tumour. Some readers interpreted the anterior rectal wall as involved, while other readers (and the expert reference) believed this tumour was confined to the left dorsolateral wall, resulting in a tumour-MRF distance of >10 mm.
Discussion
In this study, we evaluated uniformity in structured reporting of rectal cancer among an international group of 21 radiologists. When looking at the main staging variables used for risk and treatment stratification, good results were achieved for dichotomized assessment of high- versus low-risk cT-stage and cN0 vs N + stage, lateral nodes and tumour deposits, and MRF involvement. These variables resulted in good (>80%) interobserver agreement in the majority of patient cases. Less favourable results were found for multicategorical cT- and cN-staging, and for assessing EMVI.
We observed a clear significant positive correlation between diagnostic confidence and the agreement of study readers with an expert standard of reference (which was used as a surrogate standard to estimate diagnostic accuracy, considering the lack of pathologic correlation for most study cases and variables). This correlation is an interesting finding considering that template reports tend to force radiologists to assign a specific stage or category, even if they are unsure about their diagnosis, which is a potential drawback of structured reporting. Our results show that this uncertainty can lead to substantial variations in staging. Given the potential impact on treatment, it might be better to incorporate the level of confidence into structured reporting templates to better inform risk-based clinical decision-making during multidisciplinary team discussions, at least for those items that are poorly reproducible and strongly influenced by variations in diagnostic confidence.
Interestingly, the dichotomized classification of cT-stage into high risk (≥cT3 cd) versus low risk (≤cT3 ab) resulted in better IOA than the multicategorical cT-stage classification as defined the TNM staging manual. 4 Some guidelines, like those from the National Comprehensive Cancer Network (NCCN), 18 use cT3 disease as a key risk factor indicating a need for neoadjuvant treatment. However, several studies have shown that MRI often overstages T2 tumours as T3 resulting in potential overtreatment. 19–21 Other guidelines consider early-stage T3 disease (T3ab with ≤5 mm extramural invasion) as low risk, with comparable prognostic outcomes and treatment implications as T2 tumours. 22,23 Our results show that this dichotomous classification results in better agreement and is therefore perhaps better suited to guide risk and treatment stratification. Assessment of T4a and T4b disease remains a problem area with poor IOA. As illustrated in Figure 3, there were several cT4a cases with simultaneous MRF involvement that were understaged as T3 MRF+. Apparently readers tend to choose either MRF or peritoneal (cT4a) invasion, rather than acknowledging that the two may co-occur. Moreover, some readers may mistake anterior invasion of the peritoneum above the level of the peritoneal reflection as MRF invasion (Figure 5a and b). These issues were also demonstrated as pitfalls that require further teaching in a recent international survey study. 12 It could be helpful to include the anatomical boundaries between MRF and peritoneum, and the distinction between cT3 MRF+and cT4a MRF+disease as specific staging options in template reports, aiming to increase radiologists’ awareness and ultimately uniformity in staging. Future studies should focus on establishing the benefit of further training and teaching in reducing interobserver variability in such pitfall cases.
Similar to cT-staging, results for cN-staging were also better when dichotomized (into cN0 vs cN+). Considering the known limitations of MRI, radiologists should perhaps refrain from detailed cN-staging and limit themselves to estimating the risk for cN0 or cN+disease, including a level of confidence to support their findings. Results for assessment of lateral nodes and tumour deposits were remarkably good, especially when considering the concerns voiced in recent reports on the lack of validated imaging criteria. 12 These good results can likely be partly explained by the low prevalence of positive lateral nodes and tumour deposits in our cohort and the fact that agreement with our expert standard or reference was particularly high (88%–93%) to diagnose these negative cases.
Results for assessment of EMVI were lower than expected (>80% interobserver agreement reached in only half of the study cases). Although image-detected EMVI has been acknowledged as an important prognostic factor for some time, it was introduced into reporting guides and templates more recently and is still less routinely reported than other risk factors such as TN-stage and MRF. 1,7 Some readers may, therefore, still be going through a learning curve and experience difficulties with less straightforward cases such as Figure 3. In line with a previous report that showed relatively low sensitivity and PPV (62–67%) and high specificity and NPV (88–89%), 17 radiologists in our study were also better in assessing EMVI- than EMVI+tumours. Further teaching, and inclusion of specific published grading systems to diagnose EMVI+disease into reporting templates, could be beneficial to further improve uniformity in the staging of EMVI.
When looking at the other staging items, tumour morphology (in particular the distinction between annular, semi-annular and polypoid tumours) and level of sphincter involvement showed the poorest results. Describing the tumour morphology is mainly relevant as polypoid tumours typically have a better prognosis compared to (semi-)annular tumours with a more extensive invasive margin. 24 Disease-focused panel recommendations from the Society of Abdominal Radiology (SAR) define a polypoid tumour as a tumour with a pedicle or stalk. 25 A later report by Golia Pernicka et al suggested to redefine the polypoid definition to ≤¼ wall circumference attachment and a visible pedicle. 24 Future guidelines and reporting templates should perhaps adopt such more specific definitions to improve uniformity in the radiological assessment of tumour morphology. Moreover, radiologists should be properly informed about the morphology (but also size and location) of tumour lesions from the endoscopy reports to allow proper clinically informed image interpretation and reporting.
Concerning the level of sphincter involvement, recent guidelines recommend that in distal tumours, a coronal sequence parallel to the anal canal should be included to properly assess the relation between tumour and anal sphincter. 1 In our current cohort, such a sequence was only available in 38% of the cases with suspected sphincter involvement. Our cohort dates back as far as 2010, which means that some scans were obtained using outdated study protocols, including suboptimal sequence angulation. Although a more in-depth analysis of the impact of image quality was beyond the scope of this paper, we acknowledge this as a limitation that may have introduced bias.
There are some other limitations to our study design. First and foremost, as aforementioned, accuracy figures could only be estimated and were calculated using expert consensus as a standard of reference, considering the lack of histological confirmation for the majority of patients. Our study cohort was skewed towards advanced disease (77%) and direct correlation with pathology was only available for 17 cases. Moreover, not all staging variables were routinely reported in the pathology reports. Diagnostic confidence scores were furthermore missing for some of the staging variables. Image evaluation was performed using only T 2-weighted sequences. Although DWI sequences are not routinely recommended for primary staging (mainly for restaging), 1 they are also commonly included in primary staging protocols and might have been of benefit to aid in tumour detection and delineation. MRI scans in our cohort originate from one single country. Still, we believe that the sample derived from 10 different centres is representative for general clinical routine with representation of different vendors and common variations in clinical protocols. Finally, the number of patient cases assessed in this study was relatively small (n = 75). This number was chosen as a minimum requirement to allow meaningful statistical analyses 26 while at the same time ensuring the feasibility that a large number of radiologists would complete all the scans within an acceptable timeframe.
In conclusion, this study shows that – although structured reporting aims to accomplish uniformity in staging – there are still some pitfalls that need to be acknowledged as they may result in insufficient staging reproducibility. Suggestions for improvement include more simplified, dichotomized risk stratification of cT- and cN-stage, adoption of confidence scores for items with low reproducibility, embedding more specific definitions for image interpretation into staging templates, and ensuring state-of-the-art image protocols.
Footnotes
Acknowledgements: On behalf of the rectal MRI study group: Frans C.H. Bakers, M.D.; Perla Barros, M.D.; Ferdinand Bauer, M.D.; Shira H. de Bie, M.D.; Stuart Ballantyne, M.D.; Joanna Brayner Dutra, M.D.; Nino Bogveradze, M.D.; Gerlof P.T. Bosma, MA MD PhD.; Adriana Mirela Calin - Vainak, M.D.; Vincent C. Cappendijk, M.D., Ph.D.; Francesca Castagnoli, M.D.; Anuradha Chandramohan, M.D.; Sotiriadis Charalampos, M.D.; Andrea Delli Pizzi, M.D.; Sarah Evans, MBChB; Remy W.F. Geenen, M.D.; Joost J.M. van Griethuysen, M.D., Ph.D; Jody Maclachlan, M.D.; Vandana Mahajan, M.D.; Sonaz Malekzadeh, M.D.; Peter A. Neijenhuis, M.D.; Mateus de Oliveira Taveira, M.D.; Gerald M. Peterson, M.D.; Indra Pieters, M.D.; Raluca Popita, M.D.; Niels W. Schurink, M.Sc., Ph.D.; Carmelo Sofia, M.D.; Signe Swerkersson, M.D.; Cornelis J. Veeken, M.D.; Roy F.A. Vliegen, M.D., Ph.D.; Abdel-Rauf Zeina, M.D.
Contributor Information
Najim el Khababi, Email: n.e.khababi@nki.nl.
Regina G.H. Beets-Tan, Email: r.beetstan@nki.nl.
Luís Curvo-Semedo, Email: curvosemedo@gmail.com.
Renaud Tissier, Email: r.tissier@nki.nl.
Joost Nederend, Email: joost.nederend@catharinaziekenhuis.nl.
Max J. Lahaye, Email: m.lahaye@nki.nl.
Monique Maas, Email: m.maas@nki.nl.
Doenja M.J. Lambregts, Email: d.lambregts@nki.nl.
Collaborators: on behalf of the rectal MRI study group
REFERENCES
- 1. Beets-Tan RGH, Lambregts DMJ, Maas M, Bipat S, Barbaro B, Curvo-Semedo L, et al. Magnetic resonance imaging for clinical management of rectal cancer: updated recommendations from the 2016 European Society of Gastrointestinal and Abdominal Radiology (ESGAR) consensus meeting. Eur Radiol 2018; 28(): 2711. doi: 10.1007/s00330-017-5204-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Kassam Z, Lang R, Arya S, Bates DDB, Chang KJ, Fraum TJ, et al. Update to the structured MRI report for primary staging of rectal cancer : perspective from the SAR disease focused panel on rectal and anal cancer. Abdom Radiol (NY) 2022; 47: 3364–74. doi: 10.1007/s00261-022-03612-3 [DOI] [PubMed] [Google Scholar]
- 3. Nougaret S, Rousset P, Gormly K, Lucidarme O, Brunelle S, Milot L, et al. Structured and shared MRI staging lexicon and report of rectal cancer: a consensus proposal by the French radiology group (GRERCAR) and surgical group (GRECCAR) for rectal cancer. Diagn Interv Imaging 2022; 103: 127–41. doi: 10.1016/j.diii.2021.08.003 [DOI] [PubMed] [Google Scholar]
- 4. Jessup MJ, Goldberg RM, Asare EA, et al. Colon and rectum. In: Amin MB, Edge S, Greene F, eds. AJCC Cancer Staging Manual. 8th edition. Springer; 2017, pp. 251–73. [Google Scholar]
- 5. Brown PJ, Rossington H, Taylor J, Lambregts DMJ, Morris E, West NP, et al. Standardised reports with a template format are superior to free text reports: the case for rectal cancer reporting in clinical practice. Eur Radiol 2019; 29: 5121–28. doi: 10.1007/s00330-019-06028-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Zhao AH, Matalon SA, Shinagare AB, Lee LK, Boland GW, Khorasani R. Improving the completeness of structured MRI reports for rectal cancer staging. Abdom Radiol (NY) 2021; 46: 885–93. doi: 10.1007/s00261-020-02754-6 [DOI] [PubMed] [Google Scholar]
- 7. Bogveradze N, El Khababi N, Schurink NW, van Griethuysen JJM, de Bie S, Bosma G, et al. Evolutions in Rectal cancer MRI staging and risk stratification in the Netherlands. Abdom Radiol (NY) 2022; 47: 38–47. doi: 10.1007/s00261-021-03281-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Alvfeldt G, Aspelin P, Blomqvist L, Sellberg N. Radiology reporting in rectal cancer using MRI: adherence to national template for structured reporting. Acta Radiol 2022; 63: 1603–12. doi: 10.1177/02841851211057276 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Nörenberg D, Sommer WH, Thasler W, DʼHaese J, Rentsch M, Kolben T, et al. Structured reporting of rectal magnetic resonance imaging in suspected primary rectal cancer: potential benefits for surgical planning and interdisciplinary communication. Invest Radiol 2017; 52: 232–39. doi: 10.1097/RLI.0000000000000336 [DOI] [PubMed] [Google Scholar]
- 10. Tersteeg JJC, Gobardhan PD, Crolla RMPH, Kint PAM, Niers-Stobbe I, Boonman-de Winter LJM, et al. Improving the quality of MRI reports of preoperative patients with rectal cancer: effect of national guidelines and structured reporting. AJR Am J Roentgenol 2018; 210: 1240–44. doi: 10.2214/AJR.17.19054 [DOI] [PubMed] [Google Scholar]
- 11. Brouwer NPM, Stijns RCH, Lemmens V, Nagtegaal ID, Beets-Tan RGH, Fütterer JJ, et al. Clinical lymph node staging in colorectal cancer; a flip of the coin? Eur J Surg Oncol 2018; 44: 1241–46. doi: 10.1016/j.ejso.2018.04.008 [DOI] [PubMed] [Google Scholar]
- 12. Lambregts DMJ, Bogveradze N, Blomqvist LK, Fokas E, Garcia-Aguilar J, Glimelius B, et al. Current controversies in TNM for the radiological staging of rectal cancer and how to deal with them: results of a global online survey and multidisciplinary expert consensus. Eur Radiol 2022; 32: 4991–5003. doi: 10.1007/s00330-022-08591-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Gollub MJ, Lall C, Lalwani N, Rosenthal MH. Current controversy, confusion and imprecision in the use and interpretation of rectal MRI. Abdom Radiol (NY) 2019; 44: 3549–58. doi: 10.1007/s00261-019-01996-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Schurink NW, van Kranen SR, Roberti S, van Griethuysen JJM, Bogveradze N, Castagnoli F, et al. Sources of variation in multicenter Rectal MRI data and their effect on Radiomics feature reproducibility. Eur Radiol 2022; 32: 1506–16. doi: 10.1007/s00330-021-08251-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. El Khababi N, Beets-Tan RGH, Tissier R, Lahaye MJ, Maas M, Curvo-Semedo L, et al. Comparison of MRI response evaluation methods in Rectal cancer: a Multicentre and Multireader validation study. Eur Radiol 2023; 33: 4367–77. doi: 10.1007/s00330-022-09342-w [DOI] [PubMed] [Google Scholar]
- 16. Ziegler E, Urban T, Brown D, Petts J, Pieper SD, Lewis R, et al. Open health imaging foundation viewer: an extensible open-source framework for building web-based imaging applications to support cancer research. JCO Clin Cancer Inform 2020; 4: 336–45. doi: 10.1200/CCI.19.00131 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Smith NJ, Barbachano Y, Norman AR, Swift RI, Abulafi AM, Brown G. Prognostic significance of magnetic resonance imaging-detected Extramural vascular invasion in Rectal cancer. Br J Surg 2008; 95: 229–36. doi: 10.1002/bjs.5917 [DOI] [PubMed] [Google Scholar]
- 18. Benson AB, Venook AP, Al-Hawary MM, Azad N, Chen Y-J, Ciombor KK, et al. Rectal cancer, version 2.2022, NCCN clinical practice guidelines in oncology. J Natl Compr Canc Netw 2022; 20: 1139–67. doi: 10.6004/jnccn.2022.0051 [DOI] [PubMed] [Google Scholar]
- 19. Wetzel A, Viswanath S, Gorgun E, Ozgur I, Allende D, Liska D, et al. Staging and restaging of rectal cancer with MRI: a pictorial review. Semin Ultrasound CT MR 2022; 43: 441–54. doi: 10.1053/j.sult.2022.06.003 [DOI] [PubMed] [Google Scholar]
- 20. Beets-Tan R, Beets G, Vliegen R, Kessels A, Boven HV, Bruine AD, et al. Accuracy of magnetic resonance imaging in prediction of tumour-free resection margin in rectal cancer surgery. The Lancet 2001; 357: 497–504. doi: 10.1016/S0140-6736(00)04040-X [DOI] [PubMed] [Google Scholar]
- 21. Taylor FGM, Swift RI, Blomqvist L, Brown G. A systematic approach to the interpretation of preoperative staging MRI for rectal cancer. AJR Am J Roentgenol 2008; 191: 1827–35. doi: 10.2214/AJR.08.1004 [DOI] [PubMed] [Google Scholar]
- 22. Taylor FGM, Quirke P, Heald RJ, Moran B, Blomqvist L, Swift I, et al. Preoperative high-resolution magnetic resonance imaging can identify good prognosis stage I, II, and III rectal cancer best managed by surgery alone: a prospective, multicenter, European study. Ann Surg 2011; 253: 711–19. doi: 10.1097/SLA.0b013e31820b8d52 [DOI] [PubMed] [Google Scholar]
- 23. Glynne-Jones R, Wyrwicz L, Tiret E, Brown G, Rödel C, Cervantes A, et al. Rectal cancer: ESMO clinical practice guidelines for diagnosis, treatment and follow-up. Ann Oncol 2017; 28: iv22–40. doi: 10.1093/annonc/mdx224 [DOI] [PubMed] [Google Scholar]
- 24. Golia Pernicka JS, Bates DDB, Fuqua JL, Knezevic A, Yoon J, Nardo L, et al. Meaningful words in rectal MRI synoptic reports: how "Polypoid" may be prognostic. Clin Imaging 2021; 80: 371–76. doi: 10.1016/j.clinimag.2021.08.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Hope TA, Gollub MJ, Arya S, Bates DDB, Ganeshan D, Harisinghani M, et al. Rectal cancer lexicon: consensus statement from the society of abdominal radiology rectal & anal cancer disease-focused panel. Abdom Radiol (NY) 2019; 44: 3508–17. doi: 10.1007/s00261-019-02170-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Cicchetti DV. Testing the normal approximation and minimal sample size requirements of weighted kappa when the number of categories is large. Appl Psychol Meas 1981; 5: 101–4. doi: 10.1177/014662168100500114 [DOI] [Google Scholar]