Enhancing the role of case-oriented peer review to improve quality and safety in radiation oncology: Executive summary

Abstract

This report is part of a series of white papers commissioned for the American Society for Radiation Oncology (ASTRO) Board of Directors as part of ASTRO's Target Safely Campaign, focusing on the role of peer review as an important component of a broad safety/quality assurance (QA) program. Peer review is one of the most effective means for assuring the quality of qualitative, and potentially controversial, patient-specific decisions in radiation oncology. This report summarizes many of the areas throughout radiation therapy that may benefit from the application of peer review. Each radiation oncology facility should evaluate the issues raised and develop improved ways to apply the concept of peer review to its individual process and workflow. This might consist of a daily multidisciplinary (eg, physicians, dosimetrists, physicists, therapists) meeting to review patients being considered for, or undergoing planning for, radiation therapy (eg, intention to treat and target delineation), as well as meetings to review patients already under treatment (eg, adequacy of image guidance). This report is intended to clarify and broaden the understanding of radiation oncology professionals regarding the meaning, roles, benefits, and targets for peer review as a routine quality assurance tool. It is hoped that this work will be a catalyst for further investigation, development, and study of the efficacy of peer review techniques and how these efforts can help improve the safety and quality of our treatments.

Outline of the full report (available online only at www.practicalradonc.org).

• 1.0
  Introduction

  • 1.1
    Current Peer Review Recommendations within Radiation Oncology
  • 1.2
    Prior Work on Peer Review in Radiation Oncology
  • 1.3
    Programmatic Peer Review and Maintenance of Certification Programs
  • 1.4
    Peer Review: QA for Professional Qualitative Decisions

    • 1.4.1
      Peer Review versus Process Control
    • 1.4.2
      Rationale for Case-Specific Peer Review in Clinical Radiation Oncology
  • 1.5
    Context of Current Report
• 2.0
  The Traditional Approach to Case-Oriented Peer Review in Radiation Oncology

  • 2.1
    Chart Rounds
  • 2.2
    Prospective Pretreatment Tumor Boards
• 3.0
  Categorizing Possible Targets for Peer Review

  • 3.1
    General Considerations
  • 3.2
    Categorizing Targets for Peer Review
  • 3.3
    Pre-Planning, Physician-Focused Tasks
  • 3.4
    Treatment Planning—Dosimetry/Physics-Focused Tasks
  • 3.5
    Treatment Delivery—Therapist-Focused Tasks
  • 3.6
    Prioritizing the Possible Targets for Peer Review
  • 3.7
    Operational Implementation/Prioritization
• 4.0
  Implementing Effective Peer Review

  • 4.1
    Example Process Improvements
  • 4.2
    Example Technological Improvements
  • 4.3
    Peer Review in the Context of Evolving Roles
  • 4.4
    Peer Review in the Context of Education
• 5.0Discussion
• 6.0Summary of General Recommendations
• 7.0Conclusions

White papers on patient safety in RT

The full report is part of a series of white papers addressing patient safety commissioned by the American Society for Radiation Oncology (ASTRO) Board of Directors as part of ASTRO's Target Safely Campaign. The full length document was approved by the ASTRO Board of Directors on September 11, 2012 and has been endorsed by the American Association of Physicists in Medicine, American Association of Medical Dosimetrists, and the American Society of Radiologic Technologists. The document has also been reviewed and accepted by the American College of Radiology's Commission on Radiation Oncology. These organizations have a long history of supporting efforts toward improving patient safety in the United States.

This report is related to other published reports of the ASTRO white paper series on patient safety, including those on intensity modulated radiation therapy (IMRT) and stereotactic body radiation therapy (SBRT), and those still in preparation. There are sections of this report that defer to guidance in these reports.

1.0. Introduction

Peer review, also known as audit and feedback, is a valuable tool central to quality management or quality assurance (QA) programs.¹

While peer review has been accepted as an important aspect of quality efforts (especially of physicians' decisions) in radiation oncology for many years, there is currently little specific guidance and limited published literature. The goals of this report are to:

• a.provide a summary of current recommendations;
• b.review potential peer review targets and to discuss prioritization and rationale; and
• c.propose improvements in processes or technology that may facilitate or improve peer review, and acknowledge associated challenges.

1.1. Current peer review recommendations within radiation oncology

Available only at www.practicalradonc.org.

1.2. Prior work on peer review in radiation oncology

Brundage et al² assessed the real-time pretreatment review of 3052 treatment plans over 8 years. They found that such pre-radiation therapy peer review was feasible, and that plan modifications were recommended in approximately 8% of cases. A similar prospective study by Boxer et al³ noted peer review-recommended changes in 8/208 patients (4%). A posttreatment peer audit of ~ 80 cases also noted that ≈ 5% of patients had apparent controversial/concerning medical decisions made regarding things such as treatment intent, dose, and fractionation.⁴

These latter 2 studies used peer review audit tools created by the Royal Australian and New Zealand College of Radiologists³ and The Cancer Institute in Singapore.⁴ These peer review audit tools have evolved over time in response, in part to systematic assessments of their reproducibility.⁴ Such tools have the potential to improve practice quality, to alter patient management,^{4, 5, 6} and to provide an effective means to document changes in patient management.⁷ The experiences of these groups, and components of their existing audit tools, may prove useful in future initiatives.

The chief residents of 57/71 North American academic radiation oncology facilities responded to an anonymous survey addressing “peer review chart rounds.”⁸ Their key findings include the following: (1) providing protected time and monitoring attendance was associated with better participation by senior faculty; (2) review of routine external beam cases was more common (where 80% of institutions reported peer review of all cases) than for radiosurgery or brachytherapy (where 58%, and 40%-47% of centers reported review of cases, respectively); (3) 60% reported that the duration of chart rounds was < 2 hours per week (range, 1–6); (4) the median time spent per patient was 2.7 minutes (range, 0.6-12 minutes); and (5) minor and major changes were relatively uncommon as a result of chart rounds; 14% of respondents estimated that minor changes (eg, small multileaf collimator change or request to repeat a port film) were requested in ≥ 20% of cases, and 61% of respondents estimated that minor changes were requested in < 10% of cases; 11% of respondents estimated that major changes (eg, to the dose prescription or treatment plan) occurred in ≥ 10% of cases, and 75% of respondents estimated that major changes occurred in < 10% of cases.⁸

1.3. Programmatic peer review and maintenance of certification programs

Available online only at www.practicalradonc.org.

1.4. Peer review: QA for professional qualitative decisions

1.4.1. Peer review versus process control

This paper concentrates on patient-specific peer review in radiation oncology; ie, peer review for items linked to a specific patient (eg, dose distribution), rather than global processes that effect patients more broadly (eg, machine calibration). Patient care processes can be broadly divided into a series of medical (often qualitative) decisions that often do not have clear right or wrong answers (eg, a physician's prescription of dose-volume) and a series of more quantitative technical tasks or programs to implement the prescribed treatment (typically executed by teams of therapists, dosimetrists, and physicists) (Fig 1). This report concentrates on ways that peer review can be used to help improve the safety and quality related to professional decisions made by members of the radiation oncology team (left side of Fig 1).

Figure 1.

A quality management program must address medical and qualitative steps (left side) as well as technical and quantifiable process-related steps (right side) to implement the medical directive. The left side is the focus of this report.

2.0. The traditional approach to case-oriented peer review in radiation oncology

2.1. Chart rounds

Peer review is widely practiced, largely through “chart rounds.” During these sessions, members of the treatment team review each case (eg, doses, fields, treatment plans, patient setup, etc).^{8, 9, 10} A recent survey suggested that such routine peer review occurs in essentially all North American centers with accredited residency training programs.⁸

2.2. Prospective pretreatment tumor boards

Tumor boards, with multiple physicians and members of the healthcare team from diverse disciplines, are a second common form of peer review. The group renders opinions and advice for individual patients (eg, whether or not to use RT). Such a multidisciplinary discussion is often crucial, especially in complex cases; posttherapy discussions are less useful.¹¹

3.0. Possible targets for peer review

3.1. General considerations

Available online only at www.practicalradonc.org.

3.2. Categorizing targets for peer review

Candidate decisions or actions that can be targeted for peer review are summarized and divided into 6 categories corresponding to the simplified radiation therapy process (Table 1, available online only at www.practicalradonc.org). For each task, Table 1 describes the goal of peer review, the person who typically performs the peer review, the ideal timing, and where applicable, features that may facilitate the peer review. A refined prioritization of targets is provided in the section “Prioritizing the possible targets for peer review.”

3.3. Preplanning, physician-focused tasks

The first series of process steps are components of medical decision making that come before the treatment planning process. For some of these items, there are objective treatment guidelines that might be useful in streamlining/automating peer review.

3.4. Treatment planning—dosimetry/physics-focused tasks

The latter table entries largely focus on treatment planning and treatment delivery. Physicians, dosimetrists, therapists, and physicists all have critical roles in these tasks and in their peer review. The diversity of individuals involved reflects the large number of hand-offs and “back and forth/iterations” that occur during the planning/delivery process. The very nature of these processes demands a rigorous review and QA system.

The seemingly infinite number of possible treatment plans, and the large amount of time needed to generate and QA a plan, make peer review of treatment planning challenging. Often the planner and physician's experience dictates when the “best” possible plan has been generated. This is a critical step, where peer review might be particularly helpful. Furthermore, consideration should be given to nonmedical issues; eg, a complicated treatment approach may not be appropriate if the patient is uncooperative.

3.5. Treatment delivery—therapist-focused tasks

Therapists are at the “end of the line" and are often expected to perform a broad review of all prior activities; eg, “Is this the correct patient and correct site?” Many of these have clear right or wrong answers and are in reality “quality/process control” issues (ie, not technically “peer review”). However, almost all treatment delivery errors manifest themselves at the treatment machine. Given this fact, it is clear that in order to maximize safety and quality, multiple therapists are needed at the treatment machine to check on each other, verify all the necessary information, and try to prevent upstream errors from getting to the treatment, while also using peer review on qualitative issues such as setup accuracy, immobilization, and image review.

Even with multiple therapists per machine, modern radiation therapy expects the radiation therapists to perform the following numerous physical tasks in the treatment room (eg, patient setup); at the treatment machine console (eg, retrieving data from medical records, image review/manipulation); and elsewhere (eg, retrieving patients from the clinic lobby). Having tasks performed in multiple locations promotes a “divide and conquer” approach that may undermine therapist peer review. However, given the many varied tasks and speed with which they are performed, the use of a second person to review the actions, measurements, data entry, and machine control aspects of the therapists’ job is a critical part of a good QA program. The second set of eyes is a crucial check of all the technical steps performed by the therapists during each treatment fraction. The use of timeouts (similar to a presurgery checklist in an operating room) to verify the physician's prescription, the prescribed dose programmed into the machine, and the patient's identity help the therapist team review these most crucial aspects of a treatment.

3.6. Prioritizing the possible targets for peer review

Table 1 (available online only at www.practicalradonc.org). includes a lengthy list of possible peer review targets, and addressing each item is not practical. Thus, prioritization is needed. Candidate “high priority” targets should be those with a high potential risk to the patient, and a low probability that an “error” will be detected downstream (Figure 3, available online only at www.practicalradonc.org). There are some data to help guide this prioritization. In the studies from Boxer et al³ and Brundage et al,² target volume coverage was the item most often changed during peer review. In the Brundage et al study, 8% of plans were identified as requiring modification. The most common reasons for modification related to planning target volumes (31%), protection of critical structures (15%), selection of treatment volumes (eg, nodal volumes included or excluded, 11%), selection of dose (11%), and dose distribution (6%). These domains correspond to current patterns of peer review in Ontario.²

Table 2.

Prioritization of targets for peer review

Item for peer review	Prioritization	Rationale for priority level	Timing of peer review and associated comments	Example clinical situations where peer review is anticipated to be particularly useful
1) Decision to include radiation as part of treatment	Level 2	Guidelines often exist, but these decisions are often individualized	Pretherapy preferred	Unusual/nonguideline cases
2) General radiation treatment approach	Level 3	There are many guidelines and best practice statements that address this issue. If standard dose/volume constraints are respected, patient risks are low regardless of the specific RT approach taken.	Preradiation preferred. Altering some aspect of the treatment approach once RT has been initiated can be cumbersome (eg, image guidance approach), while other aspects are more easily changed during RT. The safest environment is one where mid-treatment changes are minimized.	Retreatment cases
3) Target definition⁎	Level 1	Every patient's tumor is different and visualization on different types of images can vary. Each image fusion is unique.	Pretreatment peer review of how targets are defined (eg, which images and which "pixels") is critical as mistargeting can lead to poor clinical outcomes. Preplanning review is ideal but is not critical for every case.	Tight margins; eg, SBRT
4) Normal tissue image segmentation	Level 3	There are atlases for normal tissues.	Review of normal tissues can be done during RT since the risks are less (especially for fractionated regiments). Normal tissue pre-RT peer review needed for single and hypofractionation cases.	Tight margins; eg, SBRT
5) Planning directive (dose/volume goals/constraints for targets and normal tissues)	Level 2	Patient risks are low if standard dose/volume limits are respected. Guidelines and best practice recommendations often exist, but these decisions are often individualized.	Preplanning or pretreatment
6) Technical plan quality	Level 2	Normal tissue dose/volume guidance documents are generally available, but the compromises between normal tissue vs target doses are often patient specific.	For conventional fractionation, this may be acceptable to perform during RT, as there is usually an opportunity to alter the plan. The safest environment is one where mid-treatment changes are minimized.	IMRT, SBRT
7) Treatment delivery (eg, patient setup)	First day is Level 1, especially for curative cases. Other days are Level 2.	The first day's setup is critical to avoid systematic errors and their propagation.	Therapist peer review of setup must be done pre-RT for the first fraction, and ideally for all subsequent fractions. Portal or localization image peer review must be done before the second treatment. Physicist and physician involved with pretreatment QA for complex cases (eg, SBRT).	IMRT (since portal or localization imaging often does not provide independent assessment of target volume location)

Level 1 indicates highest priority for peer review (where there are marked interpatient variations), Level 2 next highest (where there are often guidelines/atlases to aid in decision), and Level 3 the next (other targets for peer review).

RT, radiation therapy; IMRT, intensity modulated radiation therapy; SBRT, stereotactic body radiation therapy.

^⁎Target definition includes the decision regarding the need for multimodality imaging, the fusion of the images, and the target definitions on the images.

A study from Johns Hopkins used failure modes and effects analysis (FMEA) to identify priority targets.¹⁰ The tasks with the highest risk probability included wrong isocenter used, wrong markings used, incorrect patient in the record and verify system, incorrect contours used, and wrong computed tomographic simulation data entered for the patient.¹⁰ However, this approach is not ideal, as many of the events that occur in the clinic are not predicted by the FMEA (Ford, personal communication, 2011). Additional work is certainly needed to refine this prioritization. The recent survey of North American teaching centers noted that the issues most commonly modified by peer review related to normal tissue exposure, the prescribed dose/fractionation schedule, target coverage, and treatment technique.⁸

A suggested prioritization of targets for peer review, associated justifications, and applicable clinical situations are listed in Table 2. Specific recommendations for these targets are given below.

• (1)The physician's decision to treat is considered as a level 2 priority for peer review as this is often aided by clinical practice guidelines. Retreatment settings are often particularly challenging (especially when normal tissue constraints in the target volume are compromised) and peer review may be particularly useful.
• (2)General treatment approach decisions (eg, treatment goal [curative, adjuvant, or palliative]; modality [eg, brachytherapy versus various external beam approaches]; nonconformal or conformal, image guidance, and motion management approach) are often made by the physician in consultation with dosimetrists, physicists, and therapists, and thus peer review of these issues should ideally be multidisciplinary. This is considered as level 3 as these decisions are often aided by clinical practice guidelines and recommendations.
• (3)Image segmentation/contouring. Delineation of the target volumes is the physician's responsibility. Normal tissue segmentation is typically done primarily by dosimetrists, physicists, or other specially trained staff. Structure delineation is often guided by multimodality image registration that is typically performed by dosimetrists, physicists, or other specially trained staff. Peer review of segmentation/fusion (as well as the images chosen for segmentation) should ideally be done prior to treatment planning as much of the subsequent work is dependent on the details of the segmentation/fusion. This is one of the most important medical decisions that likely would benefit from peer review. Because there are significant, often marked interpatient variations in the target volumes, and because mistargeting can lead to poor clinical outcomes, this is considered as one of the most critical areas for peer review (ie, level 1), especially in patients being treated with curative intent and with highly conformal approaches (as is often the case for image guided radiation therapy [IGRT] and SBRT). Segmentation of the normal tissues is considered level 3 as this is often guided by atlases.
• (4)Planning directive. The planning goals (eg, desired dose-volume parameters and limits) are often based on existing guidelines. Nevertheless, these decisions are often somewhat qualitative, and are thus considered level 2. This peer review should ideally be performed prior to initiation of treatment planning (particularly if the planning process is anticipated to be time consuming and highly dependent on the planning goals), or prior to the initiation of therapy (if the planning process is less time consuming and less dependent on the planning goals). This might be most important in the settings where there are not clear clinical guidelines (eg, the retreatment setting) or when normal tissue and target constraints are in conflict. Similarly, changes in the planning goals or image segmentation made during the course of therapy (ie, adaptive therapy) may also benefit from peer review.
• (5)Technical plan quality. Evaluation of the plan's quality, relative to the planning goals, is usually made by the treating physician, in consultation with the dosimetrist/physicist who is typically familiar with the technical tradeoffs and compromises made during planning. Peer review for planning evaluates 2 types of decisions from the planning process: the physician-driven clinical tradeoffs (ideally addressed through physician peer review), and the more technical aspects of the plan's ability to achieve the desired dose-volume results with reasonable complexity and deliverable fields (ideally addressed through dosimetrist/physicist peer review).
• (6)Setup accuracy and consistency. A radiation therapist is responsible for daily setup accuracy, and thus a second radiation therapist should ideally provide daily review; ie, 1 therapist setting the patient up with a second verifying, or the 2 therapists working together and checking each other. In more challenging cases (eg, SBRT), a physicist and/or physician should also provide peer review of the therapist's activities. Similar peer review should ideally be performed for other therapist activities such as review of daily pretreatment setup images, or review of respiratory gating parameters. This is identified as a critical target for peer review, especially in curative cases, and in IMRT and SBRT cases where port films do not provide an independent assessment of the treated volume.

3.7. Operational implementation/prioritization

See full text version available only at www.practicalradonc.org.

4.0. Implementing effective peer review

Implementing effective peer review can be challenging. There are often competing demands and rigid computer-workflow systems that do not readily facilitate or support peer review. Table 3 (available online only at www.practicalradonc.org) offers a list of potential barriers to effective peer review, possible interventions, and recommendations that may help improve the effectiveness of peer review. The section below offers a more detailed description of examples of suggested improvements and tools that might help make peer review more effective and efficient. The themes raised are potentially more important than the precise methods of implementation.

4.1. Example process improvements

The full details of this section are available in the full text version (available online only at www.practicalradonc.org). The outline of this section is as follows:

• a.Management
• b.Allotting necessary resources and time
• c.Facilities
• d.Creating a collaborative atmosphere
• e.Knowing each other's names
• f.Minimizing distractions and maintaining clearly-defined roles and expectations

4.2. Example technological improvements

This section recommends a number of changes in software and other technologies that the vendors should consider since they may help improve the efficiency, effectiveness, and usefulness of various peer review activities. The outline of this section is as follows:

• a.Integration of peer review tools into our routine workflow (eg, within treatment planning/management systems);
• b.Tools to streamline peer review of target doses/prescriptions;
• c.Tools to assist peer review of normal tissue exposures;
• d.Tools to facilitate peer review of segmented anatomy; and
• e.Inter-center connectivity

4.3. Peer review in the context of evolving roles

The changing practice of radiation oncology is leading to modification of traditional tasks ascribed to each team member. Table 4 (available online only at www.practicalradonc.org) illustrates some of these changes in tasks and how this might have some implications for peer review.

4.4. Peer review in the context of education

See full text version available online only at www.practicalradonc.org.

5.0. Discussion

See full text version available online only at www.practicalradonc.org.

6.0. Summary of general recommendations

Peer review of important nontechnical decisions has the potential to improve the quality of radiation therapy patients receive. Thus, it should be embraced by leadership and staff, and be considered part of the standard practice.

Leadership need to empower the staff to be involved in peer review activities (eg, facilitate and support their involvement) and provide the necessary infrastructure for efficient and effective peer review (eg, adequate space, image display capabilities, access to electronic records, support staff to help monitor and facilitate review, software tools).

Peer review should be conducted in the context of an open and just culture, such that staff do not feel threatened by the peer review process. For example, people who are found to have made an honest mistake should not face punitive measures. On the other hand, people should be held accountable for failing to participate in peer review activities.

Clear expectations for the “content” and the conduct of peer review efforts (eg, the what, when, where, and how) are necessary. Among the many targets relevant to improving safety and quality with peer review methods, the most obvious high-priority targets include the following.

• (1)Physician. Physicians are indicated in 5 of the 7 prioritized items for peer review. In descending order, they include the following: level 1, target definition; level 2, decision to include radiation as part of treatment, planning directive; level 3, general radiation treatment approach, normal tissue image (Table 2).
• (2)Dosimetrist. Level 2, technical plan quality (Table 2).
• (3)Therapists. The first day of treatment delivery is considered level 1, high-priority peer review for radiation therapists, especially in curative cases, IMRT, and SBRT cases where there is no independent assessment of the targeted volume. Other treatment days are considered level 2, next highest priority (Table 2).
• (4)Physicists. Level 1 for pretreatment setup verification for complex cases (eg, SBRT). Level 2, technical plan quality for treatments in general (Table 2).

The specific goals and targets of peer review should be clearly specified, and the results of each peer review effort ideally should be tracked. Creative ways to actively monitor the clinical utility of peer review are needed to reassure staff that their time is well spent and to identify opportunities for improvement.

Users and vendors should collaborate to define and create software and hardware tools that can help make peer review more efficient and effective, and to keep track of these peer review activities (eg, providing the ability for annotation in the electronic medical record).

For small practices, where peer review is particularly challenging, we encourage the creation of peer review relationships among physicians from separate (perhaps distant) practices.

The principles of peer review should be included in the curriculum in educational programs. Students should be included as participants in (or at least observers of) peer review as it can be educational, promotes a culture of respectful questioning (ie, students observe professionals questioning each other), and reinforces the role and utility of peer review as part of routine clinical practice.

Developing a successful comprehensive peer review program requires the concerted activities of many people. Example actions central to the peer review mission are outlined in Table 5 (available only at www.practicalradonc.org), for leadership, staff, and vendors, who can all help facilitate improved safety and quality through more effective peer review.

7.0. Conclusions

See full text version available online only at www.practicalradonc.org.

Supplementary materials.

The following are the supplementary materials related to this article.

Supplementary Material (pdf)

PRRO_200_Supp_Material_file.pdf