Evaluating the Impact of Transitioning from Weekly to Daily Peer Review in Radiation Oncology

Abstract

Purpose:

Peer review is essential for quality assurance (QA) in radiation therapy, addressing the variability among clinicians in defining treatment target volumes. At our institution, weekly peer review meetings (PRMs) were initially conducted on Fridays, but the need for more robust review led to the implementation of PRMs on a daily basis. These daily sessions aim to enhance quality by routinely discussing each radiation therapy plan with a team of radiation oncologists.

Methods and Materials:

In this study, the impact of daily PRMs on radiation oncology planning was assessed by analyzing changes suggested during these meetings. Treatment plans for 249 patients were reviewed from April to September 2022. During PRMs, each plan was reviewed and modifications were categorized as no change, minor change, major change, or missing contour, with further stratification by target volumes, treatment field, radiation doses and treatment decisions.

Results:

A total of 249 cases were reviewed. The largest proportion of discussed plans comprised head & neck cancers (n= 79, 31.7%), thorax (n=55, 22.1%) followed by brain and pelvis (each n=39, 15.7%). Most of the plans i.e., 190 (76.3%) were based on IMRT/VMAT. PRM changes were suggested in 114 (45.7%) cases, where 78 (31.3%) were minor and 36 (14.5%) were major changes. In most cases the changes were suggested in CTV i.e., 62 (24.9%) while PTV modifications were suggested in 21 (8.4%) of the cases during PRM discussion. Among all, head and neck was the region with most frequent PRM changes, suggestions 53 (57.6%) followed by pelvis 18 (19.5%).

Conclusion:

Peer review is a vital component of quality assurance in radiation oncology, helping to identify and correct potential errors, standardize care, and enhance patient safety. It also promotes multidisciplinary collaboration and serves as a valuable educational tool, ultimately supporting more effective and individualized treatment.

Introduction

Audits and feedback are considered effective in improving professional practice [1]. Peer review (PR), defined as “the evaluation of creative work or performance by other people in the same field to enhance the quality of work or the performance of colleagues”, is considered as a means of feedback regarded as a critical element in optimal radiation oncology run-through [2]. It is a valuable tool central to quality management or Quality Assurance (QA) programs, having been proven to be beneficial across literature. Due to the sensitive nature of radiation treatment planning, major organizations such as the American Society for Radiation Oncology (ASTRO) have also recommended including peer review in departmental policies. Additionally, bodies such as ASTRO and the European Organization for Research and Treatment of Cancer (ESTRO) have also recognized peer review as a valuable of indicator of QA in radiation planning, due to considerable evidence of inter observer variability in the field [3, 4].

To address the need for peer review, many institutions have adopted implementation policies, focused on efficient and effective review of treatment planning [5, 6]. These policies typically involve a general assessment of the therapeutic rationale, target, and organ-at-risk (OAR) volumes; prescription dose and fractionation; treatment dosimetric plans; and patient setup [7]. Despite these policies, it remains unclear, however, the ideal criterion and frequency of such peer review meetings (PRMs) [7]. Though previous research has suggested that a frequency of PRM every week may be sufficient, recent literature has indicated that adopting a more frequent and thorough approach may enable accurate plan delineation, and treatment delivery [5]. These implementations may not be as feasible in low-income settings, however, where the number of practicing radiation oncologists and dosimetrists may preclude effective discussion, and frequent plan re-contouring. As such, it remains to be determined whether such techniques have a similar benefit in these settings.

The purpose of this study was to evaluate the efficacy of daily PRM in radiation therapy planning at a tertiary-care medical facility in a low-middle income setting (LMIC). The present study aimed to observe the impact of daily PRM in radiation oncology, and to determine the impact of frequency towards resultant changes in treatment plan volumes. Additionally, factors associated with changes in treatment planning were evaluated, to determine whether any region or PRM strategy enabled greater frequency of changes.

Materials and Methods

At our institution, PRMs were initially held weekly, typically at the end of the week (e.g., Friday). However, the need for a more robust and timely review process led to the implementation of daily PRMs, conducted Monday through Friday. This study was conducted following exemption from the ethical review committee (ERC) at the Aga Khan University Hospital. Data was collected prospectively from April 2022 to September 2022 from the department of Radiation Oncology at the Aga Khan University Hospital, Karachi, Pakistan. All the patients who were booked for external beam radiation therapy (EBRT) and underwent simulation followed by planning CT scan were discussed routinely in the daily PRM of the radiation oncology section at our institution and were included in this study. Patients receiving brachytherapy and urgent palliative radiation on the day of simulation were excluded.

Contouring of target volumes was completed and approved by the primary consultant radiation oncologist prior to discussion in daily PRM. During PRM, each plan was reviewed by at least two or three radiation oncologists and suggested changes were noted in the data collection sheet for study purposes as no change, minor change, major change or missing contour (Table 1). Each change was further classified as changes in target volumes including gross tumor volume (GTV), clinical target volume (CTV) and planning target volume (PTV), organs at risk (OARs), radiation doses, treatment fields or treatment decisions. The frequency of changes suggested during PRM was analyzed by patient demographics, tumor site, treatment intent, setting and radiation technique. The intent of treatment was recorded as either curative or palliative. Tumor sites included the brain, head and neck, thorax, abdomen, pelvis, extremity, and others. Radiation techniques included two-dimensional, three-dimensional conformal radiation therapy (3D-CRT), and intensity modulated radiation therapy (IMRT) or volumetric arc therapy (VMAT). Radiation planning was done on Aria-15 planning system using Eclipse workstation.

Table 1.

Operational Definitions

PRM Changes Category	Definition
1. No change	No change made in treatment volumes, fields, or doses
2. Major change	Treatment volumes, fields, or doses not acceptable
3. Minor change	Changes made within treatment volumes, fields
4. Missing contour	Treatment volume or organ at risk contour found to be missing

Data analysis was conducted using Stata Version 15 (Stata Corp LLC). Descriptive statistics were reported for quantitative variables as mean ± standard deviation, while categorical variables were described as frequencies and percentages. For categorical variables, the χ² test of independence and the fisher exact test were used for univariate analysis based on the number of observations. To assess the association of different factors with treatment changes, two binary logistic regression models were generated with Model 1 comparing no treatment changes to any treatment changes, and Model 2 comparing major planning changes to minor and no planning changes. Univariate regression was performed for relevant covariates including gender, setting, intent, region, technique, change category, patient population (pediatrics versus adults), the number of residents attending the PRM, and the number of board-certified consultants attending the PRM. Due to the small number of observations, multivariate regression was omitted. Statistical significance was assessed at two-sided p value < 0.05.

Results

Demographics and Participant Characteristics

From April to September 2022, treatment plans of 249 patients were reviewed during 69 PRMs. There were 139 (55.8%) males and 110 (44.2%) were females. The setting of treatment was definitive in 101 (40.6%) patients, adjuvant in 144 (57.8%) and neoadjuvant in 4 (1.6%) patients. The majority of our patients (n = 212, 85.1%) were treated with radical curative intent. The most common regions amongst discussed plans included head & neck cancers (31.7%), thoracic malignancies (22.1%), and brain and pelvis tumors each (15.7%). Most of the plans (n = 190, 76.3%) were based on IMRT/VMAT. The overall characteristics of patients have been summarized in Table 2.

Table 2.

Clinical and Demographic Characteristics of Patients Discussed during PRM (n = 249)

Variable	N (%)
Degreea
No change	132 (53.0)
Minor changes	78(31.3)
Major changes	36 (14.5)
Category
CTV	75 (30.1)
PTV	28 (11.2)
Doses	18 (7.23)
Others (OAR delineation, treatment decisions, RT fields etc)	8 (3.21)
Changes due to RO Residents	3.49 (0.73)
Changes due to RO Consultants	4.25 (1.08)

PRM, Peer review meeting; CTV, Clinical target volume; PTV, Planning tumor volume; RO, Radiation Oncology; ^a, Contour data missing for three participants.

Characteristics of Daily PRMs

No changes were found to be suggested during 132 (53.1%) of plans discussed during PRM (Table 3). When changes were suggested (n = 114, 45.7%), however, PRM changes were minor in 78 cases (31.3%) and major in 14.5% of plans. Changes were mostly suggested in CTV (n = 62, 24.9%), while PTV modifications were noted in 8.4% of cases during PRM discussion. The commonest regions undergoing changes during PRM were observed to be head and neck (57.6%), and pelvic (19.5%) malignancies.

Table 3.

Characteristics of PRM Meetings Included in the Present study (n = 249).

Variable	N (%)
Gender
Male	139 (55.8)
Female	110 (44.2)
Setting
Definitive	101 (40.6)
Adjuvant	144 (47.8)
Neoadjuvant	4 (1.6)
Intent
Palliative	37 (14.9)
Radical	212 (85.1)
Region
Brain	39 (15.7)
H&N	79 (31.7)
Thorax	55 (22.1)
Abdomen	11 (4.42)
Pelvis	39 (15.7)
Extremity	4 (1.61)
Technique
3DCRT	59 (23.7)
IMRT/VMAT	190 (76.3)

PRM, Peer review meeting. 3DCRT, 3D-Conformal radiation therapy; IMRT, Intensity modulated radiation therapy; VMAT, Volumetric modulated arc therapy.

Changes in Plan as Demonstrated by Clinicodemographic and Meeting Characteristics

The majority of plans receiving both minor and major changes were conducted in the adjuvant setting (52.6% and 52.8%, respectively) and of radical intent (88.5% and 88.9%, respectively). Plans of the head and neck region and males were significantly associated with major and minor changes; however, no associations were found between treatment technique, the patient population, residents at the PRM, or consultants at the PRM. CTV changes were significantly associated with both minor and major plan changes.

Univariate analysis revealed gender to have significant associations with plan changes of PRM on both regression models. Female patients were found to have significantly lower odds of major plan changes (OR (95% CI) = 0.43 (0.20 – 0.95), p = 0.04) and any treatment changes (OR(95% CI) = 0.45 (0.27 – 0.75, p = 0.002).

Discussion

Radiation oncology is an intricate discipline associated with complex treatment planning and delivery processes including consultation, simulation, target volume delineation, immobilization, treatment delivery and quality control. With the advancements in radiation techniques, treatment planning has become more complex over the past two decades, thereby increasing the likelihood of errors [8]. In order to mitigate the risk of error, comprehensive quality assurance (QA) programs, peer review processes, double-check procedures, chart rounds and standardized treatment planning protocols are some of the important measures employed regularly [9]. The present study aimed to evaluate the impact of the peer review process in leading to changes in treatment planning. Our results demonstrated PRM most employed during adjuvant treatment planning, with 45.7% of plans undergoing changes. Additionally, we also noted that during daily PRM, changes were significantly associated with both gender and regions of treatment.

Target volume delineation in radiation treatment planning has been reported to have significant interobserver variability in several studies [10, 11]. Literature has demonstrated that 6%-7% of radiation treatment plans were suboptimal and required modifications [12]. Several studies have also reported significant inter-physician variability in producing target volumes and radiation plans for conformal radiotherapy [13-16]. In the present study, we report changes in 117(45.7%) plans. The PRM changes were classified as minor, major or missing contours (Table 4). All the suggested PRM changes were then incorporated in treatment plan by the primary team. Conversely, a systematic review of published data by Brunskill et al. concluded that there were plan modifications in about 10.8% of the cases across literature [17]. Major changes were found in approximately 1.8% of the cases, with the majority of the recommended changes concerning target volume delineations [17]. However, in our study, minor changes accounted for 31.3% of cases, while major changes were suggested in 14.5% of cases. Changes in contouring (CTV: 24.9% and PTV: 8.4%) were the most common factors leading to plan modifications, followed by RT doses (4.8%) and treatment decisions (2%). These findings closely align with those reported in similar reviews17,18,19. These discrepancies may be explained by the treatment regions of selected studies in the review. The majority of studies reported the breast as the most common region undergoing PRM, with included literature demonstrating a presence of up to 49.7% of cases [18]. In contrast, most cases included in our study were in the H&N region (31.7%), a setting associated with considerable sensitivity and target volume heterogeneity [19, 20]. Additionally, H and N malignancies were noted to be significantly associated with changes in planning during PRM, in contrast to breast malignancies [17]. Similarly, our results also demonstrated an association of planning changes with H&N malignancies, further emphasizing this hypothesis. Furthermore, our results also demonstrated the importance of plan change, and its potential association with the category of change. Previous literature has typically not evaluated these characteristics, and thus, it is potentially plausible that our study evaluated a greater frequency of CTV cases, thus resulting in greater prevalences of changes. As such, these discrepancies may account for the contrast in our findings; however, further research is necessitated to warrant this assumption.

Table 4.

Changes in Plan as Demonstrated by Clinicodemographic and Meeting Characteristics (n = 246)

Variable	No Change (n = 132)	Minor Change (n = 78)	Major Change (n = 36)	P-value
Gender
Male	61 (46.2%)	49 (62.8%)	26 (72.2%)	0.006
Female	71 (53.8%)	29 (37.2%)	10 (27.8%)
Setting
Definitive	50 (37.9%)	36 (46.2%)	15 (41.7%)	0.221
Adjuvant	81 (61.4%)	41 (52.6%)	19 (52.8%)
Neoadjuvant	1 (0.8%)	1 (1.3%)	2 (5.6%)
Intent
Palliative	23 (17.4%)	9 (11.5%)	4 (11.1%)	0.411
Radical	109 (82.6%)	69 (88.5%)	32 (88.9%)
Region
Brain	31 (23.5%)	7 (9.0%)	1 (2.8%)	0.002
H&N*	26 (19.7%)	32 (41.0%)	20 (55.6%)
Thorax	35 (26.5%)	13 (16.7%)	7 (19.4%)
Abdomen	7 (5.3%)	3 (3.8%)	1 (2.8%)
Pelvis	21 (15.9%)	14 (17.9%)	4 (11.1%)
Extremity	1 (0.8%)	2 (2.6%)	1 (2.8%)
Technique
3DCRT	31 (23.5%)	17 (21.8%)	10 (27.8%)	0.782
IMRT/VMAT	101 (76.5%)	61 (78.2%)	26 (72.2%)
Category of change (n = 94)
CTV	1 (0.8%)	44 (56.4%)	16 (44.4%)	0.02
PTV	0 (0.0%)	20 (25.6%)	1 (2.8%)
Doses	0 (0.0%)	6 (7.7%)	6 (16.7%)
Patient Population
Adult	115 (87.1%)	62 (79.5%)	31 (86.1%)	0.322
Pediatrics	17 (12.9%)	16 (20.5%)	5 (13.9%)
Residents at PRM
<= 3 residents	74 (56.1%)	39 (50.0%)	16 (44.4%)	0.406
> 3 residents	58 (43.9%)	39 (50.0%)	20 (55.6%)
Consultants at PRM
> 4 doctors	83 (62.9%)	53 (67.9%)	21 (58.3%)	0.578
<= 4 doctors	49 (37.1%)	25 (32.1%)	15 (41.7%)

Published literature has shown that in most of the centers peer review meetings are held either weekly or twice weekly, as daily meetings are often considered less feasible [5]. We switched from weekly to daily peer review meetings to enhance the quality of discussions, resulting in even more frequent suggestions for changes during the meetings, being 45.7% compared to 22.4% in the previous study [21]. This can be attributed to a reduced number of cases discussed daily, allowing for more extensive discussion on each case. Furthermore, daily PRMs also mitigated the risk of potential treatment delays as observed in the case of weekly peer review [22]. The duration of peer review meetings varies across different centers. In our department, we typically discuss 4 to 5 cases during daily peer review meetings, which usually last for about an hour. As per Mitchell et al, the average duration of case discussion requiring a major change is 10 minutes and 6 minutes for a case requiring no or minor change [23]. As such, daily PRMs may enable greater rigor and greater allocation of time, thus enabling increased frequency of changes, however, it is imperative that future research evaluates these variables to demonstrate such relationships.

In our study, we observed that the majority of changes, 51 out of 249 (20.4%) plans, were recommended when three radiation oncologists were present in the meeting. Despite this, the number of residents and number of consultants demonstrated no association with increasing odds of treatment planning. As such, it may be possible that the number of individuals present on PRMs is impacted by other factors, such as the case load of the meeting, which may be confounding our results. Furthermore, literature has also demonstrated that the experience of the faculty attending PRM may impact the quality of PRMs. This could be explored further and possibly form the basis of site-specific peer review meetings, as practiced in some centers [24-26].

This study has several implications. Firstly, our results demonstrated that daily PRMs may lead to greater frequency of plan changes, with certain regions and change categories being associated with the degree of change. Similar literature published previously in this setting for weekly PRMs demonstrated lower frequencies of changes and differing associations [21]. These associations may be explained by the frequency of meetings and also by the increase in quality of radiation techniques in Pakistan. At the time of conduction of the previous study, IMRT was less frequent (25.9% of plans), while in the present study the majority of participants had received IMRT (76.3%). This technique has several advantages, however, due to nuances in its application, considerable care and precision is required in its implementation [27]. As such, it is possible that these advancements in therapy delivery may motivate greater provider care in PRM review, and further manifest as increase in planning changes. Furthermore, our results demonstrated a high prevalence of H&N plans discussed, with this region being associated with plan change. The burden of H&N cancer is high in Pakistan, with delays in diagnosis and unique tumor morphologies manifesting as late-stage disease with considerable OAR involvement [28]. Thus, in such settings, it may be beneficial to implement frequent PRMs, as these enable discussion of a large volume of high-risk cases, thus enabling efficient identification of such cases and appropriate plan review.

Our study has several limitations which must be considered when interpreting our results. Notably, we did not document the total meeting duration, and the number of cases discussed in each meeting. Such data could have offered valuable insights into the quality of the discussions held. Additionally, information regarding the time elapsed from peer review to the first day of treatment could have helped determine any potential delays. Furthermore, our analysis must be viewed with caution, as some variables had lower frequencies of observations thus reducing the statistical power and potentially leading to interpretations that may not be reflected with larger sample sizes. Lastly, despite the high prevalence of plan changes, our results did not determine whether such recommendations manifested in actual improvement. Future research is warranted to determine the prevalence of plans requiring changes actually undergoing such changes, as in a high-volume center, literature has demonstrated that such recommendations are not always followed accordingly.

Peer review is vital for quality assurance in radiation oncology treatment planning. Daily peer review meetings are both feasible and effective, promoting thorough discussions that enhance patient care and provide valuable learning opportunities for residents. These sessions foster a collaborative environment, driving continuous improvement in treatment accuracy and professional development.

Author Contribution Statement

Laraib Khan: Manuscript writing, Data Collection. Aahan Arif: Data Analysis. Maham Khan: Manuscript writing. Mariam Hina: Manuscript writing. Tooba Ali: Data Collection. Bilal Ahmed: Drafting. Maria Tariq: Review. Asim Hafiz: Review. Nasir Ali: Review. Ahmed Nadeem Abbasi: Review. Bilal Mazhar Qureshi: Conceptualization.