Virtual Radiation Oncology Peer Review is Associated With Decreased Engagement and Limited Case Discussion: Analysis of a Prospective Database Before and During the COVID-19 Pandemic

Introduction

Peer review (PR) of radiation therapy (RT) plans is an important component of safety and quality in radiation oncology and leads to identification of deviations and plan modification in approximately 10% of cases.^1,2 PR “chart rounds” conferences usually include cases being reviewed with approximately 3 minutes of discussion between treating physician and independent peer reviewers. Rates of deviation detection may vary based on venue, timing, and other factors.^2,3 Depending on the institution, PR can represent a variety of activities at different times throughout the RT planning process, and the extent can vary widely for various RT intents and indications.^4,5

The COVID-19 pandemic has resulted in significant changes to clinic operations and radiation oncology practice.^6,7 In accordance with public health guidelines, in-person activities have transitioned to a virtual format, including weekly PR. Recent data suggest that virtual PR results in statistically significantly lower rates of identified deviations.⁸ Although this may be a result of increased provider-peer agreement during this period, other counterproductive features of the virtual format may affect PR outcomes. We analyzed our prospectively maintained departmental database for differences in the rate of deviations and alterations in the PR discussion itself between pre- and post–COVID-19 pandemic periods.

Methods and Materials

Our weekly chart rounds conference is a prospective review of upcoming treatment before or within 1 week of starting. For each case, the signed prescription and an exported PDF summary document of the RT plan are displayed from within the electronic medical record (MOSAIQ; Elekta AB, Stockholm, Sweden) with a short patient presentation, which may be followed by a group discussion. At our institution, an in-depth review of the target/normal tissue structures and the plan within the treatment planning system are out of the scope of the weekly chart rounds meeting. However, some of our physicians and trainees do perform slice-by-slice reviews within their own site-specific conferences that are not captured within the current analysis. A prospective REDCap database (created in April 2019) captures deviation and peer discussion data (Table 1).⁹ On March 30, 2020, chart rounds transitioned to the virtual WebEx (Cisco Systems, San Jose, CA) platform using an otherwise similar process. Conference attendance, format, workflow, and methodology all remained similar to in-person, and our department never adopted a work from home policy. Unfortunately, we do not have data on faculty attendance. Throughout the COVID-19 pandemic, the individuals that comprised core clinical operations within our department (faculty radiation oncologists, physicists, dosimetrists, and trainees) continued patient care activities on-site, except in extreme circumstances, which were overall uncommon. In this institutional review board approved study, RT plans reviewed between April 15, 2019, and November 22, 2021, were dichotomized as either in-person or virtual based on the virtual transition date. Descriptive statistics were performed for each group and comparisons of proportions between groups were done using a 2-sample test of binomial proportions. A deviation was defined as any change to any aspect of the patient’s prescription or treatment plan that was recommended by peer reviewers during the meeting. Discussion items were also collected and were defined as a topic that was discussed during the PR of that individual case, whether or not it resulted in a recommended change. Due to the nature of the data collection, weekly attendance metrics were not available for the in-person group, but were collected for the virtual group—all attendees appearing in the participant list in the teleconference were logged along with each individual case. These records were queried to identify the presence of each of 8 clinical faculty attendings for each of the 89 weeks represented in the virtual group. Statistical analyses were performed using R (R Foundation for Statistical Analysis, Vienna, Austria).

Table 1.

Rates of deviations and discussion for radiation therapy cases reviewed in-person or virtually

	Overall	In-person	Virtual	P value
Total (n)	3372	1120	2252
Deviation identified	138 (4.1)	82 (7.3)	56 (2.5)	< .001
Discussion topics
None	2847 (84.4)	813 (72.6)	2034 (90.3)	< .001
Treatment intent*	39 (1.2)	24 (2.1)	15 (0.7)	< .001
Dose/fractionation†	161 (4.8)	93 (8.3)	68 (3.0)	< .001
Diagnosis‡	38 (1.1)	24 (2.1)	14 (0.6)	< .001
Prescription site label‡	42 (1.2)	26 (2.3)	16 (0.7)	< .001
Normal tissue exposure§	150 (4.4)	78 (7.0)	72 (3.2)	< .001
Target coverage§	91 (2.7)	53 (4.7)	38 (1.7)	< .001
Treatment technique†	53 (1.6)	43 (3.8)	10 (0.4)	< .001
Target selection§	42 (1.2)	31 (2.8)	11 (0.5)	< .001
Concurrent chemotherapy†	40 (1.2)	26 (2.3)	14 (0.6)	< .001
Image guidance†	3 (0.1)	2 (0.2)	1 (0.0)	.258
Protocol flag in electronic medical record‡	2 (0.1)	1 (0.1)	1 (0.0)	.554
Special physics consult‡	2 (0.1)	1 (0.1)	1 (0.0)	.554
In vivo dose measurement§	2 (0.1)	1 (0.1)	1 (0.0)	.554
Dose constraints§	4 (0.1)	2 (0.2)	2 (0.1)	.604
Bolus†	10 (0.3)	8 (0.7)	2 (0.1)	.003
Topics discussed (per patient) (%)				< .001
0	2847 (84.4)	813 (72.6)	2034 (90.3)
1	410 (12.2)	228 (20.4)	182 (8.1)
2+	115 (3.4)	79 (7.1)	36 (1.6)

All results are reported as n (%). Discussion topics are labeled according to the following categories:

^*Treatment intent

^†Radiation therapy dose, prescription, or treatment technique

^‡Documentation/nomenclature

^§Target/normal tissue

Results

In total, 3489 RT plans were reviewed and 3372 plans were evaluable with date and deviation data: 1120 in-person and 2252 virtually. Although attendance records were not available for the in-person group, the number of clinical faculty (out of 8 total) that were present at each chart rounds teleconference was calculated. At least 5 faculty physicians were present for 80 of the 89 (89.9%) conferences, at least 6 were present for 68 (76.4%), and at least 7 were present for 49 (55.1%). The overall deviation rate was 4.1%. At least 1 topic was discussed in 15.6%. The most common discussion items were dose/fractionation (4.8%), normal tissue exposure (4.4%), target coverage (2.7%), and treatment technique (1.6%) (Table 1).

When comparing in-person versus virtual PR, there was a significantly lower rate of identified deviations in the virtual group (7.3% vs 2.5%; P < .001). The depth and breadth of PR discussion per case also declined: 72.8% of in-person cases had no discussion compared with 90.3% of virtual cases (P < .001). One topic was discussed in 20.4% of in-person cases and 8.1% of virtual cases, and 2 or more topics were discussed in 7.1% and 1.6%, respectively (P < .001). Rates of discussion by topic are described in Table 1. The 15 potential discussion topics were grouped into major discussion categories (treatment intent, RT dose/prescription/technique, documentation/nomenclature, and target/normal tissue) for further analysis of the breadth of discussion between in-person and virtual PR. The proportion of cases that had at least 1 item discussed within each of these categories is shown in Figure 1. Relative to the in-person group, rates decreased for treatment intent (from 2.1% to 0.7%; P = .001), RT dose/prescription/treatment technique (from 14.0% to 4.0%; P < .001), documentation/nomenclature (from 4.1% to 1.3%; P < .001), and target/normal tissue (from 12.9% to 5.0%; P < .001).

Fig. 1.

Bar plot of the proportion of cases with topics discussed in in-person and virtual peer review conference. All comparisons are statistically significantly different between in-person and virtual groups (P ≤ .001).

To better understand the relationship between case discussion and rate of deviation identification, we performed a subgroup analysis of only cases that had at least 1 topic discussed during PR of that case. Among specific cases that had active discussion (307 in the in-person group, 218 in the virtual group), there was not a significant difference in rates of identified deviations (P = .46), with in-person discussions resulting in a 26.7% deviation rate and virtual discussion resulting in a 23.9% deviation rate.

Discussion

A retrospective report assessed differences in radiation oncology PR for 274 in-person versus 195 cases discussed virtually during the COVID-19 pandemic.⁸ They similarly demonstrated a significant decrease in the number of deviations among cases reviewed virtually compared with in-person. Overall, it was concluded that the virtual process was a timely method for plan evaluation, did not lead to treatment delays, and seemed to increase the consensus among providers. It is unclear why the implementation of a virtual format would lead to a change in the number of deviations identified in routine PR, if this represents a favorable finding, and what factors may account for these findings. We sought to investigate our departmental weekly radiation oncology chart rounds database for differences in error detection between in-person and virtual formats. Because data on case-by-case discussion topics were already being prospectively collected, we also performed a detailed analysis of the effect of the virtual transition on number and breadth of topics discussed.

Our analysis of 3372 prospectively recorded RT plan reviews provides unique insight into this issue with detailed information on the types of discussions occurring for each case. After conversion to virtual conferences, we noted a significant decrease in deviations and an increase in the number of cases with no discussion at all. Although this could theoretically represent improved consensus among providers, it seems unlikely that providers would no longer find any items to discuss or be less prone to encounter common errors as a result of the virtual format. It is a possibility that clinicians have become more selective with their comments in a virtual setting in an effort to avoid talking over/interrupting others. However, an alternative and potentially alarming interpretation of these data may be that the remote conferencing process could potentially promote less meaningful engagement and accountability from providers. These findings are likely subject to multiple confounding factors that are not accounted for in this analysis, and this interpretation should be made with caution. In this study, a causal association between virtual radiation oncology chart rounds and a decrease in deviation detection cannot be reached. However, concern remains that less engaged RT plan PR (regardless of the driving factors related to virtual format) increases the chances of critical errors going unidentified. Further validation of this study in other institutions with differing standards and systems is required. When doing so, it will be critical to carefully consider the professional, systematic, interpersonal, psychological, and sociologic factors that may have a downstream effect on the quality of radiation oncology PR.

There are multiple technical issues that can limit the virtual PR experience: voice echo, screen delays, and/or poor image/audio quality. At our institution, the in-person visual experience entailed a medium-sized conference room that seats 50 people with multiple large flat-screen televisions for display. The virtual experience was individualized on each participant’s workstation. Our department has a full-time, dedicated, onsite information technology analyst, and any visual glitches or lagging was promptly corrected during the virtual meetings. For that reason, we rarely, if ever, had any inherent issues with the conferencing software video or audio. We do not anticipate that significant visual issues contributed to these findings, as it was potentially easier to see the presenter’s screen when maximized on an individual high-resolution workstation monitor rather than from approximately 5 to 10 feet away in a conference room. On the other hand, we hypothesize that the ability to simply minimize the chart rounds presentation and divert one’s focus to other tasks may have contributed to these findings. Even when all systems are operational, the ability to mute one’s audio/video feed provides an opportunity to divide one’s attention away from PR in favor of other time-intensive clinical or administrative responsibilities.¹⁰ For example, the option to “call in” while driving presents obvious hazards. Careful and attentive inspection of several aspects of the RT plan are essential for rigorous PR. Based on these data, it is concerning that interruptions, distractions, and multitasking that were not as prevalent during an in-person conference may lead to less attentive PR with the potential for uncaught errors.¹¹

In addition to the technical challenges associated with virtual PR, there are numerous physical, cognitive, psychosocial, and emotional effects of the virtual meeting format (as well as the context in which the transition to virtual occurred) that may have also affected these findings. Differences in attendance between in-person and virtual chart rounds conferences may have affected the error detection rate as well as the discussion outcomes. Unfortunately, owing to the nature of the in-person data collection, comparisons of attendance between the 2 groups are not possible. However, attendance data abstracted from the virtual group show a majority of clinical faculty were present in 90% of conferences and three-quarters of the faculty were present in 76%. It is possible that changes in amplitude and distribution of faculty attendance after the transition to virtual PR affected these findings. Other barriers to effective, impartial discussion include the lack of nonverbal cues or other intangible, “personal” aspects of safety culture that are very likely to be lost in the virtual format, particularly if the text-based communication function is used in an attempt to avoid interrupting the ongoing presentation.¹² Despite the fact that participants of each virtual chart rounds meeting were each at their workstations within the department in the vast majority of cases (work from home at our institution was not adopted except in extreme circumstances), there were likely great changes to the perceived distance between participants after the transition from in-person to virtual meetings. Perceived distance, a symbolic concept of perceived (rather than spatial or temporal) proximity between team members, is significantly associated with decreased levels of team collaboration.^13,14 It may also play a role in the promotion and maintenance of trust between team members as well as team creativity, all of which are critical for quality radiation oncology PR.¹⁵

There are multiple features of the virtual meeting format that cause significant disarray in many professional settings. These include meeting log in/hosting difficulties; camera, microphone, or speaker issues; meals during the teleconference; and the complications of working from home and the requisite background distractions.¹⁶ All of these issues represent significant barriers to effective communication for both case presenter and reviewer. It is unclear whether the lack of these interactions contributed to these findings in the context of radiation oncology PR, and further attention to this issue is needed. Along with the required virtual transition, multiple other work-related and personal stressors were likely present at the time of the transition and for at least several months after. Additionally, with the global increase in virtual meetings, “Zoom fatigue” may have also contributed to the differences noted in this study.¹⁷ It should be noted that it is not possible to draw a direct causal relationship between virtual PR and lack of deviation detection. Further consideration and investigation of these findings is warranted to more accurately determine methods and policies to improve the quality of PR and enhance patient safety.

There are still benefits to virtual PR. In addition to the decreased risk of COVID-19 transmission, screen sharing may demonstrate plan details with higher resolution than in-person displays. There are also several potential strategies to decrease distractions during virtual PR. These include careful attendance tracking (and efforts to maintain high attendance of virtual chart rounds); designation of intradepartmental working groups consisting of clinicians, medical physics, dosimetry, therapy, and staff for targeted discussion as needed; a video-on policy; and/or other participation requirements. Considering these findings, we plan to strongly encourage all participants to maintain audio/video for the duration of the meeting and pursue further prospective testing of these interventions on PR discussion and deviations. Further research regarding barriers to engagement in virtual patient care teleconferences such as radiation oncology chart rounds is needed to inform these policies, though ultimately, departmental preferences and practices that maximize engagement may vary between institutions. It may also be important to consider requesting individuals to volunteer as the “peer reviewer of record” to better encourage personal interaction/involvement. Considering the uniqueness of chart rounds to each individual institution, it is important that departments consider these findings when evaluating their radiation oncology PR processes. This study is limited by its nature as a single-institution review and may be detecting institutional findings that may not generalize to the broader radiation oncology community. Considering the ubiquity of virtual PR, these findings require further validation at other institutions to better understand the mechanisms and potential interventions to improve PR.

Conclusions

Necessitated by the COVID-19 pandemic, virtual radiation oncology chart rounds resulted in decreased depth and breadth of PR discussion and lower deviation identification. There are multiple potential factors affecting these findings that require further study to better understand the medical and societal implications of radiation oncology PR in the virtual format. In the meantime, policies to more closely mimic in-person interactions may improve clinician engagement and maintain the rigorous PR needed to ensure optimal RT plan quality.

Data sharing statement:

Deidentified data available upon reasonable request.