High Interobserver Agreement on PSMA PET/CT Even in the Absence of Clinical Data

Background

Recommended by current guidelines, prostate-specific membrane antigen (PSMA)–directed PET/CT is increasingly used in men with prostate cancer (PC). We aimed to provide concordance rates using the PSMA reporting and data system (RADS) for scan interpretation and also determine whether such agreement rates are affected by available patient characteristics at time of scan.

Patients and Methods

Sixty men with PC, who all underwent ⁶⁸Ga-PSMA-11 PET/CT, were included. Three independent, experienced readers indicated general scan parameters (including overall scan result, organ or lymph node [LN] involvement, and appropriateness of radioligand therapy). Applying PSMA-RADS 1.0, observers also had to conduct RADS scoring on a target lesion (TL) and overall scan level. During the first read, observers were masked to all relevant clinical information, whereas on a second read, relevant patient characteristics were displayed, thereby allowing for determination of impact of available clinical information for scan interpretation. We used intraclass correlation coefficients (ICCs; with 95% confidence intervals [CIs]), which were then rated according to Cicchetti (0.4–0.59 fair, 0.6–0.74 good, and 0.75–1 excellent agreement).

Results

For general parameters, agreement rates were excellent, including an overall scan result (ICC, 0.85; 95% CI, 0.76–0.90), LN metastases (ICC, 0.89; 95% CI, 0.83–0.93), organ involvement (ICC, 0.82; 95% CI, 0.72–0.89), and indication for radioligand therapy (ICC, 0.94; 95% CI, 0.90–0.96). Overall RADS scoring was also excellent with an ICC of 0.91 (95% CI, 0.96–09.4). On a TL-based level, 251 different lesions were selected by the 3 observers (with 73 chosen by all 3 readers). RADS-based concordance rates were fair to excellent: all lesions, ICC of 0.78 (95% CI, 0.67–0.85); LN, ICC of 0.81 (95% CI, 0.63–0.92); skeleton, ICC of 0.55 (95% CI, 0–0.84); and prostate, ICC of 0.48 (95% CI, 0.17–0.78). When performing a second read displaying patient’s characteristics, there were only minor modifications to the previously applied RADS scoring on a TL-based level (overall, n = 8): each reader 1 and 2 in 3/60 (5%) instances, and reader 3 in 2/60 (3.3%) instances. The main reason for recategorization (mainly upstaging) was provided information on PSA levels (4/8, 50%).

Conclusions

Applying PSMA-RADS, concordance rates were fair to excellent, whereas relevant modifications were rarely observed after providing clinical data. As such, even in the absence of patient information, standardized frameworks still provide guidance for reading PSMA PETs. Those findings may have implications for a high throughput in a busy PET practice, where patient details cannot always be retrieved at time of scan interpretation or in the context of clinical trials or central reviews in which readers may be blinded to clinical data.

As one of the most common malignancies in men, various novel and innovative treatment options for prostate cancer (PC) have been introduced in recent years, including immunotherapy,¹ poly(ADP-ribose) polymerase inhibitors,² or bone-targeting therapies.³ Before initiation of such targeted therapies, an exact readout of the current status quo is needed. In this regard, prostate-specific membrane antigen (PSMA)–directed molecular imaging is used for staging, restaging, and therapeutic monitoring.⁴ Despite knowledge on biodistribution and radiotracer accumulation in sites of disease, there is an increasing body of evidence on various pitfalls of scan interpretation,^5,6 which may trigger false-positive or false-negative findings.

As such, various standardized frameworks for scan interpretation have been introduced, which have aimed to provide guidance when reading PSMA-targeted PET/CT.⁷ Those systems have also been further validated, supporting the notion that high interobserver agreement rates can be achieved, even for readers with less experience in reading scans.⁸ For instance, the PSMA reporting and data system (RADS) enables the classification of any lesion using a 5-point scale, thereby allowing the determination of whether a given finding is anywhere on the scale of benign (RADS-1) to highly likely to be PC (RADS-5).⁹

Of note, that system has already demonstrated high concordance rates, regardless if first-generation ⁶⁸Ga-labeled or second-generation ¹⁸F-labeled radiotracers are used.^8,10 In clinical practice, however, patient characteristics are not always available, in particular, in a busy PET practice with high patient throughput (eg, up to 2000 scans/year),¹¹ where relevant information may be missed or cannot be retrieved in a timely manner. Further, many clinical trials and central reviews use blinding of readers to clinical data.^8,12 These considerations of interpreting scans in the absence of clinical information are fueled by the increasing installation of long-bore scanners, which can reduce scan time from 20 to 2 minutes, thereby allowing a further increase in the number of PSMA PET/CTs.¹³

As such, mimicking the clinical workflow at a high-volume imaging center, we aimed to determine whether concordance rates are affected by available clinical information at time of scan. For this purpose, observers investigated general parameters and target lesions (TLs) on PSMA PET/CT to determine interobserver agreement rates in a first read masked to the clinical status, followed by a second read displaying relevant patient characteristics, thereby allowing a determination of the impact of clinical information on scan interpretation.

PATIENTS AND METHODS

For the present retrospective analysis, 60 subjects with histologically proven PC were included. Patients had signed written informed consent and agreed to diagnostic PET/CTs. Approval was waived by the local ethics committee given the retrospective nature of this study.

Scan Assessment and Interpretation

After injection of ⁶⁸Ga-PSMA-11, PET/CTs were generated using a Biograph 2 scanner (Siemens Healthineers, Erlangen, Germany). Low-dose CTs were acquired from the skull base to the proximal thighs (512 × 512 matrices), along with concomitant PET (128 × 128; slice thickness, 5 mm each). Attenuation and scatter correction were included in the iterative reconstruction algorithm as implemented by the manufacturer. Further details can be found in Thomas et al.¹⁴ Image acquisition and processing have not been changed among patients.

Three independent observers with minimum of 3 years reading PSMA-targeted PET/CT investigated all scans using dedicated workstations. Readers were all familiar with the published version of PSMA-RADS.⁹ Every reader could identify a maximum of 5 TLs, which were defined as largest in size and/or with most intense radiotracer accumulation. For every TL, a score following PSMA-RADS had to be indicated (with a maximum of 3 lesions per organ compartment, defined as prostate, lymph nodes [LNs], skeleton, liver, lung, and soft tissue). Although PSMA-RADS is based on a 5-point scoring system, with PSMA-RADS 1 including definitely benign findings and 5 indicating sites of disease highly likely attributable to PC, PSMA-RADS 3 category is the most complex within the system, including indeterminate findings that may require further workup. PSMA-RADS 4, however, has relevant uptake in a manifestation not atypical for PC, but without respective finding on CT, still rendering this lesion most likely associated with PC.⁹ Of note, with PSMA-RADS 1B lesions, previous conventional imaging or histologic assessment would be needed; as such, PSMA-RADS 1A and PSMA-RADS 1B are given as PSMA-RADS 1 in the present analysis, as previously described in Werner et al.⁸ Moreover, an overall PSMA-RADS score (defined as the highest scoring among all investigated TLs) also had to be indicated. There were no other modifications to the current version of PSMA-RADS.⁹

Beyond RADS, the readers also had to rate the following conventional parameters in a binary fashion: overall scan result, LN and organ involvement, and indication for radioligand therapy using ¹⁷⁷Lu-labeled PSMA-directed therapeutic compounds. On a 5-point scale, further categories included number of organs affected, number of organ metastases, number of LN areas affected, and number of LN metastases. Further details on scan interpretation can be found in Werner et al.⁸ The readers had to indicate all items in a first read in which they were blinded to all clinical data, followed by a second read then revealing the following patient characteristics: age; staging; restaging; biochemical persistence after surgery; biochemical recurrence; Gleason score; PSA levels at time of scan; whether tumor marker was rising, stable, or declining; and prior therapies or other diseases (Table 1). If recategorizing of TL was performed, readers had to indicate respective reasons, defined as 1 or more of the provided patient details.

TABLE 1.

Overview of Clinical Data Provided Upon Second Read (Following the First, Blinded Scan Interpretation)

Parameter
Age	Median ± SD, y	67 ± 12.8
Indication for scan	Staging	36/60 (60%)
	Biochemical persistence after primary surgery	7/60 (11.7%)
	Biochemical recurrence	4/60 (6.7%)
	Primary diagnosis	4/60 (6.7%)
	Response assessment	8/60 (13.3%)
Gleason score	Median, available in n = 40	8
PSA	Overall level in ng/mL, median (range)	2.53 (0.1–578.8)
	PSA decline at time of PET	3/60 (5%)
	PSA rise at time of PET	37/60 (61.7%)
	PSA equal at time of PET	0/60 (0%)
	No information on PSA fluctuations at time of PET	20/60 (33.3%)
Prior therapies	In total	56/60 (93.3%)
	Surgery	35/60 (58.3%)
	Hormonal therapy	32/60 (53.3%)
	External beam radiation therapy	34/60 (56.7%)
	High field ultrasound	1/60 (1.7%)
	Prior PSMA-directed radioligand therapy	4/60 (6.7%)
	233Ra-dichloride	7/60 (11.6%)
	Transarterial chemoembolization	1/60 (1.7%)

Further relevant information included that one subject had a bladder tumor as a secondary tumor diagnosis, which was removed via transurethral resection. One subject had a nephrectomy on the right, and another patient had bronchitis shortly before the PSMA PET/CT.

Statistical Analysis

Information on statistical analysis is also provided in Werner et al.⁸ In brief, agreement rates were determined using intraclass correlation coefficients (ICCs; with 95% confidence intervals [CIs]) based on an average measure. Cicchetti provided an ICC-based interpretation, which was also applied to the present investigation (ICC <0.4 poor, 0.4–0.59 fair, 0.6–0.74 good, and 0.75–1 excellent agreement).¹⁵ We used MedCalc statistical software (version 18.2.1; Med-Calc Software Ltd, Ostend, Belgium). Significance level was reached when a P value was less than 0.05.

RESULTS

General Parameters Achieved Excellent Concordance Rates

Among general parameters rated in a binary setting, agreement rates were excellent, including overall scan result (ICC, 0.85; 95% CI, 0.76–0.90), LN (ICC, 0.89; 95% CI, 0.83–0.93), and organ involvement (ICC, 0.82; 95% CI, 0.72–0.89), as well as indication for radioligand therapy (ICC, 0.94; 95% CI, 0.90–0.96). Those high concordance rates were also achieved for categories rated on a 5-point scale, including number of organs affected (ICC, 0.88; 95% CI, 0.82–0.93), number of organ metastases (ICC, 0.98; 95% CI, 0.96–0.98), number of LN areas affected (ICC, 0.93; 95% CI, 0.89–0.95), and number of LN metastases (ICC, 0.93; 95% CI, 0.89–0.96).

Agreement Rates Using PSMA-RADS on a TL and Overall Scan Level Are Fair to Excellent

Overall RADS scoring was also excellent with an ICC of 0.91 (95% CI, 0.86–09.4; Fig. 1). On a TL-based level, 251 different lesions were selected by the 3 observers. Seventy-three of those lesions were chosen by all 3 readers, whereas 54 TLs were identified by 2 observers. In regards to TL seen by all 3 readers, concordance rates based on RADS were as follows: all lesions, ICC of 0.78 (95% CI, 0.67–0.85); LN, ICC of 0.81 (95% CI, 0.63–0.92); skeleton, ICC of 0.55 (95% CI, 0–0.84); and prostate, ICC of 0.48 (95% CI, 0.17–0.78; Fig. 2), thereby indicating fair or excellent agreement rates for RADS on a TL-based level. For lesions chosen by 2 observers, ICC for all lesions was 0.75 (95% CI, 0.56–0.85), also indicating excellent concordance rate. Further organ-based analyses of lesions chosen by at least 2 readers were omitted due to the low number of lesions allocated to the different organ compartments.

FIGURE 1.

Distribution of overall PSMA-RADS scoring for all readers (R). PSMA-RADS-1A and PSMA-RADS-1B are displayed as PSMA-RADS-1, as described in the text.

FIGURE 2.

Distribution of TL-based PSMA RADS scoring for all readers (R). PSMA-RADS-1A and PSMA-RADS-1B are displayed as PSMA-RADS-1, as described in the text.

Agreement Rates Remained Comparable After Providing Clinical Information

All observers conducted a second read, where all relevant clinical information was revealed before the repeated interpretation. There were only minimal adjustments to the RADS scoring on a TL-based level (overall, n = 8): reader 1 in 3/60 (5%), reader 2 in 3/60 (5%), and reader 3, in 2/60 (3.3%) instances. Reader 1 performed RADS-based upstaging from 3C lesions to 5 due to knowledge on previous therapy. Further changes in a TL located in the prostate fossa included upstaging from RADS-3A to RADS-4 due to rising PSA levels after hormonal therapy. In another patient, additional organ compartments were identified (LN involvement rated as 3D due to PSA levels). For reader 2, PSMA-RADS–based changes on a TL level included an additional site of disease in the prostate rated as PSMA-RADS 4 (Fig. 3), whereas in another subject, an iliac LN was also upgraded from RADS-3A to RADS-4 (indicated reason in both cases, PSA level). In 1 more subject, downstaging from RADS-3D was performed, as an enlarged mediastinal LN was classified as inflammatory-related given the provided information on recent bronchitis. Reader 3 upgraded RADS scoring in 2 TLs of 2 different patients (both from RADS-3B to RADS-5), whereas PSA levels and previous therapy triggered those modifications. As such, among all 8 instances with modifications after providing clinical data, PSA levels led to those changes in 4/8 (50%).

FIGURE 3.

PSMA PET/CT of a patient with recategorized RADS scoring due to provided information on serum PSA of 1.6 ng/mL upon second read (after revealing clinical data). The small hotspot in the left prostate fossa (arrow) was initially categorized as 3A of one reader due to a rather inhomogeneous uptake in this area, which is also visible on transaxial PET (middle right). PSMA RADS 3A is characterized by equivocal uptake where further workup can be considered (eg, by follow-up imaging or biopsy).⁹ Knowledge on clinical information of rising PSA, however, then led to recategorization and upstaging to RADS 4 (ie, malignant finding highly likely), as no morphologic correlate was found on CT.

DISCUSSION

In 60 men with PC who were imaged with PSMA PET/CT, we observed excellent agreement rates when investigating general parameters, such as overall scan result or indication for radioligand therapy. Overall, PSMA-RADS score also achieved excellent concordance, whereas on a TL-based level, agreement rates ranged from fair to excellent. Last, when comparing the first blinded read with the second interpretation (with available clinical information), changes were minimal (up to 5%). Those modifications on PSMA-RADS were mainly triggered by provided PSA levels. As such, mimicking the clinical workflow in a busy PET practice where clinical information may be missing at time of scan interpretation, PSMA-RADS still showed high concordance rates. In addition, the herein presented high agreement rate in the absence of clinical information may be also of importance for clinical trials with blinded reads. Taken together, PSMA-RADS may have potential to facilitate scan interpretation in a busy PET practice, which may have a positive impact on clinical workflow. Second, the high interobserver agreement rate using PSMA-RADS may be also useful for larger therapeutic trials, for example, to assess molecular image-based response after initiation of novel targeted therapies.

PSMA-targeted molecular imaging has become the diagnostic standard for men with PC in recent years, and both ⁶⁸Ga- and ¹⁸F-labeled radiotracers have received approval of the Food and Drug Administration in the United States. As such, a steady increase of PSMA-targeted scans can be anticipated, which will be compounded by recent technological advances including whole-body PETs.¹⁶ In that context, all relevant information of patients is not always available. Those considerations are further fueled by the fact that a substantial portion of patients are referred to a PET center from an external clinician.

However, even for men treated within the medical center where the PET scan is performed, the interpreting radiologist or nuclear medicine physician may still need to invest a significant amount of time to collect all clinical information.¹⁷ Of note, patient’s characteristics provided in electronic health records are not structured in an orderly fashion, and abstracting information may require the use of external programs.¹⁷ In addition, such electronically derived information is not superior when compared with paper-derived medical records,¹⁸ which are also frequently provided in an outpatient setting.

Novel graphic-based patient-overview PC systems, which are currently used in Sweden within the National Prostate Cancer Register, may overcome such limitations of currently used electronical or paper-based medical health records^17,19 but are limited to only a few centers worldwide.

To address the increasing demand for PSMA PET scans along with the problem of not having all clinical information on hand, we aimed to investigate whether PSMA PET/CT interpretation is affected by such patient characteristics, in particular focusing on standardized frameworks in the context of scan reading. In a first read conducted by 3 experienced observers, we achieved similar concordance rates as seen in previous studies applying the identical system to ⁶⁸Ga-labeled radiotracers. For instance, a recent study composed of 133 cases and reported on comparable high ICCs when PSMA-RADS has been applied.²⁰ Derwael et al²¹ compared different structured reporting systems and reported on substantial agreement rates with respective Krippendorff’s coefficient of minimum of 0.61, which is comparable to our findings ranging from 0.48 to 0.81 for ICCs on a TL-based level.

Letang and coworkers²² used the identical radiotracer as used in the present study and showed that PSMA-RADS led to a higher area under the receiver operating characteristics curve when compared with a standard scan assessment. In addition, that study also indicated that clinical factors such as age or Gleason score were independent predictors for identifying PC lesions,²² thereby indicating that patient characteristics are of importance for scan interpretation. After revealing all patients’ characteristics as provided in Table 1, however, we only noted minor modifications on the respective RADS scoring, ranging from 3.3% to 5% for all observers. Those changes, however, were mostly triggered by information on PSA levels. In this regard, at least PSA values should be provided at time of scan. In clinical practice, this biomarker could be retrieved relatively quickly, and thus, even in a high-volume imaging center with multiple scans on a daily basis, PSMA-RADS could be a useful tool to provide the most important findings even when other clinical data are still missing. Nevertheless, to ensure a high-quality scan interpretation, it should be taken due care to gather those patient’s details before requesting or reading PSMA PET/CTs.²³

Our findings on a high concordance rate even in the absence of clinical data may favor a more widespread adoption of this system. Nonetheless, future investigations should also focus on other standardized frameworks, such as PROMISE or the recently introduced E-PSMA.^24,25 The herein provided study design may then be used as a template to define whether such systems are also robust toward missing clinical data. In addition, such investigations may then also apply PROMISE, E-PSMA, or PSMA-RADS to novel, ¹⁸F-labeled PSMA radiotracers, as those agents have also demonstrated substantial high concordance rates in the context of standardized imaging interpretation,⁸ most likely due to inherent advantages of ¹⁸F when compared with the herein used ⁶⁸Ga-PSMA-11.²⁶ Another nuance of the present study is that we also demonstrate that readers achieve excellent agreement rates for identifying patients eligible for PSMA-targeted radioligand therapy. This may be of importance, as recent randomized trials will most likely trigger a more widespread clinical adoption of PSMA-directed theranostics in the near-term future.^27,28

The results of this study also suggest the suitability of PSMA-RADS for use in clinical trials, where central readers may be specifically blinded to clinical information.^8,12 As PSMA PET is increasingly incorporated as the standard imaging modality in PC therapeutic trials, the robustness of an interpretive system to a lack of available clinical information is a key trait. Moreover, further unfolding areas for structured reporting in the next few years may be the use of artificial intelligence applications. For instance, the herein observed high concordance rate may trigger future studies, for example, to characterize a larger set of PSMA PET/CTs based on RADS in a consensus read. The PC lesions labeled by expert readers could then be applied to machine learning algorithm, preferably to establish a software tool that can assist the interpreting physician to correctly code sites of disease.²⁹

Limitations of the described study include its retrospective nature, the many nonoverlapping TLs that were selected by different readers, and the heterogeneous clinical states of the imaged patients. In addition, the overall number of investigated subjects was rather low. However, these limitations do not detract from the applicability of the primary finding that PSMA-RADS classifications are robust to differences in availability of clinical data. In addition, management changes triggered by available clinical data could also be subject to future investigations. Moreover, further studies should also evaluate the interobserver agreement rate to conduct PSMA-targeted therapy (with and without clinical data), for example, based on the eligibility criteria defined by the VISION trial.²⁷

CONCLUSIONS

We observed high concordance rates among multiple observers when applying a structured framework for interpreting ⁶⁸Ga-labeled PSMA PET. Of note, scan findings remained nearly identical when clinical data were not presented, supporting the notion that even in the absence of relevant patient characteristics, expert readers still achieve a high concordance rate when interpreting PSMA-targeted scans. Those observations may be of relevance in a busy PET practice with a high volume of men scanned with PSMA PET/CT or in the context of central imaging review in clinical trials with readers who are blinded to clinical data. Nonetheless, in clinical routine, readers should ensure that patient’s data are available at time of scan interpretation.