Quality assuring “Plan of the day” selection in a multicentre adaptive bladder trial: Implementation of a pre-accrual IGRT guidance and assessment module

Highlights

• •A multi-centre trial QA programme including adaptive RT plan selection was developed.
• •This novel trial QA approach has been validated by 71 RTTs from 10 UK centres.
• •This process increased RTTs’ confidence; resulted in appropriate on-trial adaptive RT plan selection.

Abstract

Background and purpose

Hypofractionated bladder RT with or without image guided adaptive planning (HYBRID) is a multicentre clinical trial investigating “Plan of the Day” (PoD) adaptive radiotherapy for bladder cancer. To ensure correct PoD selection a pre-accrual guidance and assessment module was developed as part of an image guided radiotherapy quality assurance (IGRT QA) credentialing programme. This study aimed to evaluate its feasibility and effectiveness across multiple recruiting centres.

Materials and methods

Individuals from participating centres remotely accessed an image database in order to complete the PoD module. An assessment score of ≥83% was required in order to receive QA approval. A questionnaire was used to gather user feedback on the module. PoD decisions for the first patient at each recruiting centre were retrospectively reviewed for protocol adherence.

Results

71 radiation therapists (RTTs) from 10 centres completed the PoD module. The median assessment score was 92% (Range: 58–100%) with 79% of RTTs passing the assessment on first attempt. All questionnaire respondents reported that the PoD module prepared them for plan selection. In 51/60 of on-trial treatments reviewed, the PoD selected by the centre agreed with QA reviewers.

Conclusions

The PoD QA module was successfully implemented in a multicentre trial and enabled pre-accrual assessment of protocol understanding. This increased operator confidence and resulted in appropriate PoD selection on-trial.

1. Introduction

Image guided radiotherapy (IGRT) and adaptive RT are increasingly employed with modern external beam radiotherapy (RT) to ensure the accurate delivery of treatment. When implementing new IGRT techniques it is critical to ensure that this is undertaken is a safe and effective manner. This is especially important in the clinical trial setting when the trial involves the introduction of novel image guidance techniques to a recruiting centre, for which there may not be a standard departmental procedure. Delivery of a rigorous quality assurance (QA) program of the IGRT component of these adaptive RT trials is essential to maintain patient safety and ensure integrity of the trial data [1].

An initial step is to collect sufficient documentation relating to the imaging technique and tolerances for setup correction used by each centre and ensure these meet both national recommendations and minimum trial specifications. Locally developed programmes, including anatomical site specific competency assessments, are reviewed as part of the QA process to ensure that Radiation Therapists/Therapeutic Radiographers (RTTs) correctly implement IGRT within the trial [1]. Ensuring the quality of these processes and training at multiple sites remains a challenge for trials quality assurance teams.

Hypofractionated bladder RT with or without image guided adaptive planning (HYBRID; CRUK/12/055), is a randomised clinical trial in bladder RT and is the first multicentre adaptive plan-of-the-day (PoD) trial undertaken in the United Kingdom [2]. The POD approach utilises volumetric image guidance with cone-beam computed tomography (CBCT) to select the most appropriate prepared plan for that patient on a daily basis and this is suggested to be one of the commonest adaptive methods in bladder radiotherapy [3], [4]. Adaptive RT trials utilising a PoD concept present a significant challenge when designing a trial QA programme. The role of IGRT in these trials is extended beyond confirming accurate patient set-up to active treatment decision making through soft tissue evaluation, in order to choose an optimal plan which encompasses the treatment volume and minimises dose to organs at risk (OAR) [1]. Such trials are likely to have endpoints, which directly link to accuracy of plan selection, for example toxicity, hence prospective assessment of protocol compliance and individual staff member’s competency is a critical element of the trial QA. A suitable QA programme must be applicable for all participating centres and allow for multi-vendor delivery equipment.

Several single centre studies have reported on the training and assessment of RT staff for adaptive bladder PoD selection [5], [6], [7], [8]. Whilst workshop based training has been utilised effectively for these studies it is not considered practical in a larger multi-centre trial setting. A multi-centre adaptive bladder trial undertaken by the Trans-Tasman Radiation Oncology Group, utilised a series of web-based e-learning modules to improve knowledge of anatomy, treatment imaging and protocol requirements [9]. This web based platform was well received and utilised by many RT staff however a need for pre-accrual practical IGRT experience was expressed by a number of centres [10].

Against this background, a pre-accrual guidance and assessment module for adaptive plan selection in bladder RT was developed as part of an IGRT QA credentialing programme for the HYBRID trial. This module utilised a novel approach by providing RTTs practical hands-on experience selecting the PoD with bladder CBCT images prior to trial recruitment. The primary study aim was to evaluate the feasibility of implementing a pre-accrual PoD guidance and assessment module in a multi-vendor, multi-centre trial setting. A secondary aim was to analyse the effectiveness of the module with regard to improving operator knowledge and confidence when selecting the PoD for HYBRID.

2. Materials and methods

HYBRID is a phase II multicentre randomised trial which followed a single centre pilot study (APPLY). HYBRID recruited 65 patients across 10 participating RT centres and completed recruitment in August 2016 [2]. Trial participants were randomised between standard and adaptive RT treatment and received a dose of 36 Gy delivered in six fractions over six weeks. Participants randomised to the standard arm had a single RT plan covering the PTV, which was expanded from the empty whole bladder CTV using a standard isotropic margin. For those randomised to the adaptive arm, three PTVs (small, medium and large) were generated from a single empty bladder CTV using anisotropic population based expansions [11]. Three plans were then produced to cover the varying PTV sizes. For all patients a CBCT was acquired before each RT treatment and for the adaptive arm the most appropriate plan chosen to cover the entire bladder volume and minimise dose to surrounding OAR.

2.1. HYBRID QA programme

The National Radiotherapy Trials Quality Assurance (RTTQA) group worked in collaboration with the HYBRID Trial Management Group to define the expectations of the treatment plan selection and develop an appropriate QA programme consisting of pre-accrual and during accrual components. Centres wishing to participate in HYBRID completed a facility questionnaire which captured their experience with volumetric bladder IGRT and a process document to detail the adaptive patient pathways within their own department. Pre-accrual outlining and planning benchmark cases were also undertaken, which is standard practice for most RT trial QA programmes.

An IGRT credentialing programme was designed in order to prospectively assess RTT competency with PoD selection for HYBRID. This included a pre-accrual PoD guidance and assessment module, a centre visit for the first adaptive patient recruited from each centre and retrospective review of PoD decisions for all recruited patients.

2.2. PoD guidance and assessment module

The objective was to provide a vendor specific PoD guidance and assessment package to be accessed remotely by RTTs geographically spread across the UK. Therefore, two vendor specific image databases were setup with five sets of anonymised patient data each consisting of one planning CT and 6 CBCT scans from the pilot study, APPLY [7]. A web based approach to viewing and registration of image data enabled individual rather than centre credentialing and gave RTTs pre-accrual hands on experience with selecting the appropriate PoD. A multi-disciplinary working party consisting of clinicians and RTTs was established to choose the “expert consensus answers” for the guidance and assessment cases.

Three sets of data in the image databases were used as guidance cases with the appropriate PoD selection, rationale and screenshots provided in an accompanying guidance document (Fig. 1). RTTs were able to register and manipulate the planning CT with each of the CBCTs as they would do for treatment verifications, to build confidence in both the plan selection and protocol requirements. The remaining two sets of data in the image databases, one female and one male case were used for the PoD assessment.

Fig. 1.

Screen shots illustrating female (top) and male (bottom) assessment cases.

For the PoD assessment process, RTTs were asked to independently register the 12 CBCT images using the trial protocol, and adjust the image registration as necessary to include the bladder volume within the smallest possible PTV contour from the three available plans. The protocol specified the requirement for a 3 mm internal margin to allow for intra-fraction bladder filling. PoD selections for each case were recorded and submitted to RTTQA for review.

A minimum pass mark of ≥10/12 (83%) with the “expert consensus answers” was necessary in order to pass the pre-accrual PoD assessment. The minimum pass mark was selected based on the 80% pass mark suggested by McNair et al following the PoD training and assessment of staff members at a single centre [6]. Only those staff members who had successfully completed the assessment were approved to undertake plan selection for HYBRID. Recruiting centres were responsible for ensuring that individuals maintained their competency through involvement with, or retrospective review of any recruited patients.

2.3. Data analysis

The individual assessment scores were reviewed and analysed with descriptive statistics. Median assessment scores for the female and male case was compared using Wilcoxon signed rank test. Assessment results were also analysed according to staff “Agenda for change” (AFC) UK banding systems with staff ranging from Band 5 (junior) to Band 8 (management). The Kruskal-Wallis test was used to compare the assessment scores for each AFC band.

2.4. Online questionnaire

An online questionnaire was circulated to the IGRT RTTQA lead at each trial centres to obtain information regarding their experiences with the assessment and PoD selection. This questionnaire included questions on database accessibility, image quality, timings and PoD confidence (Table 1).

Table 1.

Hybrid questionnaire completed by 10 recruiting centres.

Question	Answer Options	Replies
Database Access
Did you find that the instructions provided for accessing the database were easy to follow?	Yes	10
	No	0
Did you encounter any difficulties in accessing the database?	Yes	6
	No	3
	Skipped Answer	1
Were any issues raised regarding access to the database resolved efficiently and effectively?	Yes	10
	No	0
Image Quality
How did the image quality compare to what you are used to?	Better	0
	Similar	7
	Worse	3
Did you think the image quality was sufficient to complete the plan of the day assessment?	Yes	9
	No	1
Timings
Approximately how long did you spend working through the guidance example cases in total?	0–30 Minutes	1
	30–60 Minutes	5
	>60 Minutes	3
	Skipped Answer	1
Approximately how long did it take you to complete the assessment?	0–30 Minutes	0
	30–60 Minutes	7
	>60 Minutes	2
	Skipped Answer	1
PoD Confidence
Do you feel that the guidance cases provided good examples of how to implement the protocol?	Yes	5
	No	3
	Skipped Answer	2
Do you think that completing the guidance cases increased your confidence in plan selection?	Yes	6
	No	2
	Skipped Answer	2
Do you feel that the guidance and assessment has prepared you for plan selection within the trial?	Yes	9
	No	0
	Skipped Answer	1

2.5. Retrospective review for first patient recruited at each centre

In order to ensure accuracy of PoD selection on-trial, the planning CT data and all six weekly CBCT were collected from each centre for their first HYBRID patient. The CBCTs acquired for each patient were evaluated by three QA reviewers and a consensus PoD decided for each treatment fraction. The reviewers PoD decision was compared with the PoD selected by the centre and any discrepancies reported.

3. Results

3.1. Pre-accrual PoD assessment results

The PoD guidance and assessment module was completed by 71 RTTs from 10 recruiting centres. As specified in the trial protocol, the PoD selection required the smallest possible PTV contour from the three available plans to cover the bladder volume with a 3 mm internal margin at each assessment case. The median assessment score for all individuals was 92% (Range: 58–100%) with 56/71 (79%) RTTs achieving the required pass mark on first attempt at the assessment (Fig. 2). The 15 (21%) individuals who did not pass on their first attempt were asked to repeat the same assessment after QA feedback was provided. The feedback was customised to each of those 15 individuals with detailed guidance focusing on the knowledge of pelvic anatomy, PTV margin concepts used in the trial, and the trial protocol requirements on PoD plan selections. The required concordance was achieved by 12/15 RTTs on their second attempt, with a median pass rate of 92%. The assessment was discussed via a webinar with the remaining individuals and they were subsequently QA approved during a RTTQA centre visit for their first patient recruited to HYBRID.

Fig. 2.

A bar chart illustrating RTTs scores for first attempt at assessment.

Fig. 3 shows the agreement with the expert consensus answers for each of the 12 assessment CBCT images (1–6 for the female assessment CBCT and 7–12 for the male assessment CBCT). Of the 852 total PoD selections, 751 (88%) answers agreed with the expert answer. For the PoD selections which disagreed with the expert consensus answer, no more than one size away in PTV was chosen eg medium PTV was chosen when the small PTV was the correct PoD answer. For CBCT where the expert consensus answer was medium, smaller PTVs were selected for 3% and 10% and larger PTVs were selected for 14% and <1% of female and male CBCTs respectively. Significantly different median scores (p < 0.05) were attained for the male PoD assessment (median = 100%, range 50–100%) than for the female PoD assessment (median = 83%, range 33–100%). As shown in the Fig. 4 box plot graph, when separating RTTs by staff grading, no significant difference in the median scores (p = 0.51) were seen between junior (AFC Band 5 & 6), senior (AFC Band 7) and managerial (AFC Band 8) staff.

Fig. 3.

A bar chart summarising the agreement with expert answers for each assessment CBCT. CBCT 1–6 represent the female CBCT cases and 7–12 represent the male CBCT cases.

Fig. 4.

A boxplot graph describing the assessment scores per AFC band.

3.2. Questionnaire results

The online questionnaire was completed by all 10 recruiting centres (Table 1). Users found the documentation provided to access the database was clear and easy to follow. Due to the level of IT security at recruiting centres, 6/9 centres encountered initial difficulties with remotely accessing the imaging databases, however all centres reported issues with access were resolved effectively and efficiently. Image quality was reported to be of sufficient quality to confidently complete the PoD assessment by 9/10 centres, with 7/10 centres reporting the images to be a similar quality to the CBCT images generated locally. The POD guidance and assessment module was completed by the majority of centres is less than 2 h.

All respondents reported the PoD guidance and assessment module prepared them for PoD selection within the trial. 5/8 centres felt the guidance cases provided good examples of PoD selection with 3/8 stating either there was not enough variation in PTV sizes across the guidance cases or that not enough information was provided regarding why certain PTVs were chosen. Despite this, 6/8 respondents thought the guidance cases increased their PoD selection confidence with the remaining 2/8 citing that they were already experienced in reviewing bladder patients and the guidance cases provided a good reminder of the protocol requirements.

3.3. First recruited patient concordance results

The PoD decisions for the first HYBRID patient recruited at each of the 10 participating centres, giving a total of 60 treatment fractions, were retrospectively reviewed. For 51/60 (85%) treatment fractions the PoD selected by the recruiting centre agreed with the consensus PoD decided by the QA reviewers. Of the remaining 9 treatment fractions, the PoD chosen by the recruiting centre was always larger than that chosen by the reviewers. It is suggested that there was a tendency for participating centres to be more cautious in their approach to PoD selection. There were no treatment fractions where the PoD decision varied by more than one PTV size between the QA reviewers and recruiting centre.

4. Discussion

This study describes the implementation of remote access image databases for individual staff assessment in adaptive bladder RT plan selection as part of a multi-centre randomised trial QA programme. Whilst QA exercises such as the facility questionnaire, external reference dosimetry checks, and benchmark delineation and planning cases have been widely adopted by most international trials groups, QA of the IGRT and treatment verification component of a trial remains uncommon [12], [13]. IGRT QA across multiple recruiting centres as part of a clinical trial presents several challenges including accommodating different vendors’ equipment, different levels of staff experience and departmental protocols including features such as action thresholds [15].

For single centre clinical trials workshop based training has proved successful however this is likely to be difficult to orchestrate in a multi-centre clinical trial setting [5], [6]. Multicentre pre-accrual IGRT credentialing has been attempted by other international QA groups [14], [15]. Middleton et al undertook a series of IGRT credentialing site visits using a laptop with Varian offline review package to prospectively review the image matching of fiducial markers in prostate patients. Each centre had a junior and senior RTT complete 39 image registrations and then the match data was exported for analysis. The authors noted that the ultimate goal would be to achieve a vendor independent web based system allowing credentialing of all RTTs rather than only a subset in a larger clinical trial setting [15].

As part of the BOLART adaptive bladder trial IGRT credentialing was implemented using a web based e-learning programme and on-line plan selection using phantoms to assess adherence to protocol. The BOLART programme, whilst being easy to use and significantly improving protocol knowledge provided limited practical experience for appropriate PoD selection [9]. The phantom studies do provide practical experience but do not easily demonstrate the image artefacts and bladder deformation caused by physiological changes in real patient cases, and the challenges of 3D image registration within vendor-specific systems [16].

A novel process for pre-accrual QA assessment of RTT competency in PoD selection has been developed and implemented in HYBRID. The process considers vendor specific user requirements, incorporates the complexity of real patient CBCT images and allows geographically diverse remote access across the UK providing a time efficient approach for IGRT training and QA assessment. The PoD assessment results from multiple RTTs across several recruiting centres demonstrated similar results to locally implemented programmes [6]. A median pass mark of 92% was attained, with 79% of staff achieving the required pass mark on the first attempt. The questionnaire feedback confirmed the guidance and assessment increased confidence and prepared staff for PoD selection within the HYBRID trial. There was a good agreement reported between the PoD selected by centres and PoD chosen by QA reviewers for the first patient recruited from each site.

The grade and number of staff undertaking the plan selection assessment was determined by each individual centre. In our study the grade of a staff member did not affect the pass mark achieved in the assessment. This shows that the PoD guidance and assessment module provided good comprehensive instructions and allowed each staff member, regardless of experience, to understand and implement the trial protocol. This finding is consistent with Middleton et al who reported uniform matching of fiducial markers across recruiting centres regardless of experience of the observer (junior vs senior RTT) for QA accreditation for the 08.01 PROFIT trial [15].

Our study suggested that assessment scores were lower on average for the female PoD assessment CBCT than the male PoD assessment CBCT. Also of those answers that did not agree with the expert answer, the majority of incorrect answers were larger for the female assessment CBCT and smaller for the male assessment CBCT. It is acknowledged that as only a single male and female case was used within this assessment it is difficult to draw any conclusions from this. Female and male cases were selected for the assessment to reflect the difference in patient’s anatomy that would be encountered in the trial.

The web-based PoD guidance and assessment module was not without limitations. In the online questionnaire staff commented that the guidance cases required a more in-depth explanation of isocentre corrections applied and why certain PoD decisions were made. The PoD guidance document has been updated accordingly for future RTT training. As this was the first web based assessment module of its kind undertaken by the RTTQA group, there were some initial issues with accessing the remote databases due to security restriction employed by individual recruiting centres’ IT departments. These issues were dealt with in a timely manner and future web based systems will need to take hospital IT security issues into consideration in the early conception phase.

5. Conclusion

The PoD pre-accrual guidance and assessment module was successfully implemented which enabled RTTs from HYBRID participating centres to access and utilise CBCT data using a vendor specific system. It is suggested that the module increases operator knowledge and confidence when selecting the PoD within the clinical trial. This novel approach can be utilised in future clinical trials which require QA of PoD selection.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.