Innovative Education Method for a More Effective, Faster, and Valued Training in Radiation Therapy Treatment Planning

Abstract

Purpose

Because of the automation of radiation therapy, competencies of radiation technologists (RTTs) change, and training methods are challenged. This study aims to develop, and pilot test an innovative training method based on lean management principles.

Methods and Materials

A new training method was developed for lung cancer treatment planning (TP). The novelty is summarized by including a stable environment and an increased focus on the how and why of key decision making. Trainees have to motivate their decisions during TP process, and to argue their choices with peers. Six students and 6 RTTs completed this training for lung cancer TP. Effects of the training were measured by (1) quality of TP, using doses in organs at risk and target volumes, (2) perceived experiences (survey), measured at baseline (T0); after peer session (T1); and 6 months later (T2). Finally, training throughput time was measured.

Results

At T0, RTTs showed a larger intragroup interquartile range (IIR) (2.63Gy vs 1.51Gy), but lower mean doses to heart and esophagus than students (6.79Gy vs 8.49Gy; 20.87Gy vs 24.62Gy). At T1, quality of TPs was similar between RTTs and students (IIR: 1.39Gy vs 1.33Gy) and no significant differences in mean dose to heart and esophagus (4.48Gy vs 4.69Gy; 17.75Gy vs 18.47Gy). At T2, students still performed equal to RTTs (IIR: 1.07Gy vs 1.45Gy) and achieved lower maximum dose to esophagus (44.75Gy vs 46.45Gy). The training method and peer sessions were experienced positive: at baseline (T0): 8 score on a scale 1-10, directly after the peer sessions; (T1): 8 by the students and 7 by the RTTs, after 9 months; (T2): 9 by the students and 7 by the RTTs. Training throughput time decreased from 12 to 3 months.

Conclusions

This training method based on lean management principles was successfully applied to training of RTTs for lung cancer TP. Training throughput time was reduced dramatically and TP quality sustained after 6 months. This method can potentially improve training efficiency in diverse situations with complex decision-making.

Introduction

Radiation therapy has been a fast-developing field for many years with several technological and process innovations.¹ Sufficient and competent employees appear to be the most important predictive determinant for timely innovation implementation.² Automation, specifically driven by artificial intelligence (AI) techniques, is entering the radiation therapy domain.3, 4, 5, 6 It is expected to significantly impact day-to-day work, especially for radiation therapy therapists (RTTs). Where AI has effects on employees’ competences and therefore on education, the adoption of AI is also dependent on the level of organizational learning.⁷ With these developments, the roles of RTTs evolve and the required competences of RTTs further increase.⁸ The professional focus is shifting from performing tasks to controlling the equipment (understanding what happens)⁶ as well as taking more individualized decisions to optimize the individual patient's treatment. Therefore, the expertise in medical decision making and background knowledge will become increasingly important for RTTs,^5,6 which present challenges for training of professionals. However, the numerous articles currently being published and announcing a lasting and profound transformation of radiation therapy planning practices focus mainly on the content of AI and not on training and skilling RTTs. This study contributes to this gap in scientific literature by testing a new way of learning. In addition to the quality aspect of the competences, the efficiency of training is important. The decreasing opportunities of training in clinical practice, due to increased automation, enhances the need of innovative training methods further, not only to ensure the quality of competences but also to limit training time. Currently used training methods are stretched, and ideas are required to improve learning in clinical practice and increase its efficiency for starting and experienced RTTs. Little research has been published on the innovation of education methods in radiation therapy.

Job instruction (based on Toyota's lean principles)⁹ could help to improve the learning curve and therefore improve training efficiency. According to Toyota, all work can be divided into 2 categories: the physical task and the related knowledge of the work. For work which involves a lot of implicit knowledge, determining the quality objectively is difficult, which complicates decision making. Treatment planning (TP) in radiation therapy includes a lot of decision making during the process based on a lot of implicit knowledge. The ambiguity further increases because multiple professions have to work together to ensure the patient's quality treatment (radiation oncologists, physicists, and RTTs). For the more complex TPs, this ambiguity and the large amount of implicit knowledge results in differing outcomes dependent on individual competencies, which is undesirable for quality assurance. Job instruction is a method which reduces ambiguity by explicating implicit knowledge⁹ and includes 3 parts: (1) job breakdown, which provides a method to analyze the work: determine what is important, how certain aspects of the job should be performed, and why it is important to do it exactly that way; (2) a structured method to prepare and train the student; (3) create an individual training plan for every employee to develop employee talent continuously (see Appendix E1 for an example of job instruction in radiation therapy TP).

Automation of TP and contouring is already widely accepted and is also (partly) working its way into actual irradiation of patients. TP is a critical subprocess in radiation therapy as this has a direct impact on the treatment quality for the patient. It entails a multidisciplinary process where medical decision making meets knowledge on the technical opportunities. TP quality depends on many aspects, starting from commissioning and quality management of the TP system, including its measured input data and detailed understanding of TP system models and limitations.¹⁰ A lot of implicit knowledge is involved to judge if an individual TP presents the desired outcome. Because of automation, different competences are required.⁵ Guidelines on the quality of the TP are present in most institutes, specifying tolerance doses to the organs at risk (OARs) and the required coverage of the target volume. However, it remains difficult to objectivate the absolute quality of the TP. Following the general rule as low as reasonably achievable for doses delivered to the OARs, even when the plan fulfills clinical constraints, does not guarantee that this is the most optimal plan for the given patient. This ambiguity results in long training times for RTTs to become competent.

We hypothesize that if we base our RTT training for TP skills on the fundamentals of job instruction, we will sustainably improve RTT's TP competences and background knowledge. This could result in improved TP outcomes with less variation among RTTs in the short term and RTTs being increasingly able to deal with the challenges expected from automation in the long term. In addition, we hypothesize that the required training time will decrease.

The aim of the current study was first to develop a more effective and efficient training method to keep RTTs sustainably skilled for the complex skills of TP and second to pilot-test the effect and sustainability of the new training method at several timepoints after training.

Methods and Materials

Study design

The study design is a single-case study and includes the development of a training method (part 1) and its evaluation with pre and post measures (part 2) to assess its efficacy and efficiency. We selected TP for lung cancer because complexity is high due to variable tumor locations in relation with the many surrounding OARs that must be considered. The described training is meant for trainees who already completed basic RTT training including basic TP competences.

Part 1: Development of the Training Method

Training materials were developed to improve the training for lung cancer TP treated with 30 fractions and volumetric modulated arc therapy TP following the structured 3 step approach of job instruction method (lean principles), see Appendix E1:

• 1.Job breakdown by experienced RTTs. RTTs determined the key points: the most critical points for the TP outcome. For every key point, a what (was done), a how (it was done), and a why (it was done) were determined (see Appendix E2) and implemented as an interactive presentation to be used as a tool during training.
• 2.Structured training: a separate education patient database was set up where trainees could practice in an arranged and safe environment independently from the clinic. A training structure was developed for trainees including a set of training tasks with increasing difficulty, finalized with a test to illustrate personal development, see Appendix E3.
• 3.Feedback to the individual trainee, including individual reflection by coaches and interpersonal reflection during a peer session on the how and why; questions were formulated to test trainee's detailed background knowledge on multiple moments in time to increase in-depth feedback and reflection opportunities (see Appendix E4). The training structure and all materials were incorporated in a learning management software to make it easier to apply for the trainee. To further enhance the learning curve, the trainee received feedback from peers by means of a peer discussion after their individual training.

To facilitate in-depth peer discussion, a tool called plan quality assurance has been created to visualize and objectivate individual's TP outcomes. This tool was used to compare TP output (eg, dose-volume [DV] parameters) between participants, the available norm (eg, clinical plan or expert plan) or the protocol constrains. The tool helps to objectivate the outcomes and thus was used to increase reflection on clinical decision making among peers. The tool can also be used to quantify the variance in TP outcomes and decision making within the group of trainees. Ninety-minute peer discussion sessions were scheduled with small groups of peers to discuss the results on 1 selected case.

The novelty of the training method can be summarized by including a more stable environment and an increased focus on the deep understanding by explicating the how and why of the key decision-making points during the TP process instead of just following protocols like recipes. Trainees have to motivate their different decisions during the TP process and, in the end, have to interact and argue their choices with peers.

Part 2: Evaluation of the Efficiency and Efficacy of the Training Method

Study population

Six students who just finished their RTT training at school completed the new training. The control group consisted of 6 experienced RTTs with at least 10 years TP experience for thoracic tumors.

Intervention

The students were trained for lung TP using the aforementioned training method. The experienced RTTs did not receive additional training but participated in the peer sessions.

Measurements, endpoints, and analysis

To evaluate the training method, we compared a lung cancer TP made by (experienced) RTTs and students using 3 endpoints:

• 1.Effectiveness: quality of TP was measured by intragroup variation, group interquartile range, and medians in DV parameters of TPs for lung cancer for the students and the experienced RTTs. DV parameters were determined for planning target volume (PTV): PTV inclusive a margin 95% of prescribed dose (V95%), 0.03 cc of the PTV volume is allowed to receive >115% of the prescribed dose (D0.03 cc), lung mean dose (D_mean), maximum 60% of the contra lateral lung volume is allowed to receive 5Gy but as low as reasonably achievable is applicable (V5 Gy), heart D_mean, heart D0.03 cc, esophagus D0.03 cc, esophagus D_mean, spinal cord 0.03 cc of the OAR volume is allowed to receive more than the maximum allowed dose for the organ at risk (D0.03 cc), mediastinal envelope D0.03 cc. Difference in interquartile range was used to quantify the difference in within group agreement between students and RTTs (Flinger test; P value <.05). Difference in median was used to quantify the agreement between the groups (Mann Whitney test; P value <.05). No correction for multiple testing was applied as the main focus of this pilot was to develop a new training method.
• 2.The perceived value of training: measured by the perceived experiences of the involved RTTs as an evaluation of educational activities on level 1 of the Kirkpatrick taxonomy.¹¹ A survey was distributed among the involved RTTs collecting quantitative scores.1, 2, 3, 4, 5, 6, 7, 8, 9, 10
• 3.Training time: measured by training throughput time in days for a student RTT to make a complex TP for lung cancer in clinical practice; the throughput time for training and for making the TPs were registered by the RTTs themselves. For the experienced RTTs, the involved education experts were asked to estimate the historical throughput time of training.

Endpoints 1 and 2 were measured at 3 timepoints: baseline (T0, end of training), after the peer session using the plan quality assurance tool (T1), and after 6 months (T2) (see Fig. 1). The TPs used for T0 and T1 were different to make sure the respondents did not memorize the key points but were able to transfer their competencies to different patient anatomy. For T1 and T2, identical TPs were used because of the longer period (6 months) between timepoints.

Figure 1.

Design of the pilot evaluating the designed training method for radiation therapy therapist's treatment planning competencies of lung cancer patients. Abbreviations: OAR = organ at risk; PTV = planning target volume; TP = treatment planning.

Results

All 6 students completed the training. All 6 experienced RTTs participated in the peer sessions. The questionnaire on subjective evaluation was filled out by 6 students at T0, 5 students and 6 RTTs at T1, and 4 students and 5 RTTs at T2.

Effectiveness: Intergroup variation (endpoint 1)

At T0, both groups showed little within-group variation. For most analyzed DV metrics, no statistically significant difference in interquartile range was observed (see Fig. 2) except for spinal cord, where students showed smaller within-group variation than employees. The employees achieved a significantly lower heart D_mean, heart V30 Gy, and esophagus D_mean with similar PTV coverage. The groups were small, which can highly impact statistical testing results. Although not statistically significant, it seems that students at T0 made different choices than experienced RTTs: students had a lower D_mean lung and a higher D_mean heart, while for experienced RTTs this was the other way around. Some of the experienced RTTs also put a higher priority of dose reduction in spinal cord.

Figure 2.

Medians and variances of dose volume parameters for (a) PTV (planning target Volume), (b) lung, (c) heart, (d) esophagus, (e) spinal cord, and (f) mediastinal envelope directly after individual training completed and before the peer session (T0), directly after the peer session (T1), and 6 months after (T2) for trainees and experienced radiation therapy therapists. The star indicates a significant difference (Flinger test P value < .05) in variance between the students and the employees. The bullets indicate a significant difference in medians (Mann Whitney test P value < .05) between the students and employees. Abbreviations: GTV = gross target volume; PTV = planning target volume.

After the peer session (T1), almost no difference between the groups was recorded. The students showed better within group agreement for the PTV coverage (V95%) than the employees, but the employees achieved lower D0.03 cc to mediastinal envelope and D0.03 cc to PTV. The students presented a large variation for lung D_mean, but difference with employees was not statistical relevant.

After 6 months (T2), no difference between students and employees was perceived. Students achieved statistically significantly lower D0.03 cc to esophagus in comparison to the employees.

Perceived value of training (endpoint 2)

The students were positive about the new training method and experienced increased background knowledge, see Table 1. Both students and experienced RTTs experienced the discussion in the peer sessions as worthwhile, although the students were more positive.

Table 1.

Reported experiences of the students and radiation technologists (RTT) participating the peer sessions

Before peer session, after individual training (T0)		Students
≥1: negative; 10: positive		1	2	3	4	5	6	Average
1	How did you experience the structured method of training? Very negative (1) - very positive (10)	8	10	6	8	9	9	8
2	How different did you experience the new training method compared with the training we usually use in practice? Not different at all (1) - very different (10)	10	10	5	9	10	9	9
3	How did you experience you registering the clinical decision-making steps in detail? Absolutely not valuable (1) - very valuable (10)	7	7	6	6	7	8	7
4	How did you experience the level of complexity of the education treatment plans to practice your competencies for clinical practice? Absolutely inadequate (1) - very adequate (10)	8	9	5	8	7	9	8
5	How confident do you feel to perform treatment plans in clinical practice? Absolutely not confident (1) - absolutely confident (10)	9	7	9	7	7	7	8
6	Do you have enough experience to complete a standard treatment plan in clinical practice? Not enough experience (1) - absolutely enough experience (10)	9	9	9	9	8	8	9
7	Do you have enough experience to complete a complex treatment plan in clinical practice? Not enough experience (1) - absolutely enough experience (10)	5	7	7	6	5	5	6
8	Do you experience that your insight/background knowledge increased due to following this specific (new) method of training? No difference from usual training methods (1) - increased background knowledge compared with usual training methods (10)	9	9	2*	5*	5	8	8

Directly after peer session (T1)		Students							Experienced RTTs
≥1: negative; 10: positive		1	2	3	4	5	6	Average	1	2	3	4	5	6	Average
1	How did you experience discussing the treatment plan outcomes and decision making in detail with peers? Very unpleasant (1) - very pleasant (10)	8	9	7	8	NA	8	8	7	7	8	7	8	7	7
2	How much did this peer discussion benefit your training outcome/existing knowledge? Absolutely nothing (1) - very much (10)	9	9	6	7	NA	8	8	7	6	7	7	5	6	6
3	How did your insight/background knowledge increase due to the discussion in the peer session? No increased background knowledge (1) - a lot of background knowledge gained (10)	6	9	5	6	NA	7	7	6	2	7	4	2	NA	4

Six months after peer session (T2)		Students							Experienced RTTs
≥1: negative; 10: positive		1	2	3	4	5	6	Average	1	2	3	4	5	6	Average
1	How did you experience discussing the treatment plan outcomes and decision making in detail with peers? Very unpleasant (1) - very pleasant (10)	10	NA	8	NA	7	10	9	7	7	8	7	8	NA	7
2	How much did this peer discussion benefit your training outcome/existing knowledge? Absolutely nothing (1) - very much (10)	8	NA	7	NA	9	9	8	6	6	9	6	3	NA	6
3	How did your insight/ background knowledge increase due to the discussion in the peer session? No increased background knowledge (1) - a lot of background knowledge gained (10)	9	NA	7	NA	7	8	8	7	1	9	5	1	NA	5

Additional questions (T2) ≥1: negative; 10: positive		1	2	3	4	5	6	Average	1	2	3	4	5	6	Average
4	I feel more competent to make a lung treatment plan in clinical practice than I did before the peer session Do not agree at all (1) - absolutely agree (10)	3	NA	6	NA	8	6	6	3	2	9	3	1	NA	4
5	How competent do you feel to make a lung treatment plan? Very incompetent (1) - very competent (10)	9	NA	8	NA	7	6	8	9	8	9	9	9	NA	9
6	How experienced are you to make a standard treatment plan? Absolutely insufficient experience (1) - absolutely sufficient experience (10)	9	NA	9	NA	9	7	9	9	9	9	9	9	NA	9
7	How experienced are you to make a complex treatment plan? Absolutely insufficient experience (1) - absolutely sufficient experience (10)	8	NA	8	NA	6	5	7	9	9	7	8	9	NA	8
8	Peer sessions should be scheduled for every treatment site Do not agree at all (1) - absolutely agree (10)	10	NA	7	NA	9†	9	9	8	5‡	10	7	6§	NA	8

Abbreviations: NA = not available; RTT = radiation therapy therapists.

^⁎Only partly trained by the new training method.

^†Comment respondent: “I think it is very useful to schedule periodic peer sessions. All RTTs plan different and you can learn a lot from tips & tricks from peers.”

^‡Comment respondent: "Not relevant for palliative TPs: too much time effort versus importance of outcome.”

^§Comment respondent: “Although the peer session did not change my experience and background knowledge, I found it interesting to discuss the TPs and the differences in decision making.”

Training throughput time (endpoint 3)

Development of training materials (TPs, e-learnings, touchscreens, and questionnaires) took about 48 hours for lung cancer TP. The throughput time for a trainee to become fully competent for easy and complex TP for lung cancer decreased from about 12 months to 3 months using the new training (Fig. 3).

Figure 3.

Summary of training elements and required time for training. Abbreviation: RTT = radiation therapy therapist.

Discussion

The training based on job instruction elements for lung cancer TP showed positive results in this pilot study. The throughput time for students to become fully competent for complex TP improved from 6 months to 12 months by means of training in daily practice to 3 months using the structured training in the education environment with equal quality results compared with experienced RTTs. The quality of competences seemed to sustain 6 months after training completion.

In addition, the data suggested that at T0 the variation of TP outcomes within the group of students was smaller than within the group of experienced RTTs, which could indicate that the decision making within the group of students was more uniform. This could be the result of more structure, an increased focus on decision making, and explicating the why and reflection throughout the training instead of focusing on TP outcomes alone. Both groups seemed to benefit from the peer discussions because they made more unanimous decisions resulting in improved TP outcomes for all.

The students rated the interpersonal reflection during the peer sessions as positive for their background knowledge. All involved employees experienced the peer sessions as positive. Some experienced RTTs brought a lot of TP experience to the session and got fewer new insights to improve their own competences, whereas others learned some essential new insights, explaining the larger diversity in opinions among RTTs. Students valued that their outcomes were compared to those of experienced RTTs, increasing insight in their own level of competence and their self-confidence. Where the average level of perceived confidence for complex TP did not improve much, we saw that the perceived confidence improved for the individuals scoring low before the peer session. However, numbers were small. All respondents rated the peer session as a helpful tool to keep up competences especially for complex TP.

Following our hypothesis, describing the what, the how, and the why in a structured manner seemed to increase understanding according to the theory described in Toyota Talent, which states that including the why of actions increases background knowledge and compliance to the how.⁹ Simon Sinek also amplified defining the why (golden circle theory) because the why influences the linguistic brain, responsible for decision making and behavior, and, therefore, is beneficial to inspire people for behavioral change. The how and the what are processed in the neocortex of the brain, which enables people to reason and analyze. However, without direct involvement of the linguistic brain, where people's feelings are processed, behavioral change will likely not occur.¹² Educating is about changing behavior. Including the why in training seems, therefore, essential to increasing compliance. Including the why is also an important step to realize triple-loop learning.¹³ Although proof of triple-loop learning effects is still limited, in theory, it should account for organizational learning and changing behavior and attitudes¹³ and should be essential to become a learning organization.¹⁴ Where double-loop learning is focused on including the how with the what,¹⁵ this educational theory shows many resemblances to the breakdown process of job instruction. Most learning is realized by doing in real practice (70%) and reflecting on it (20%), and only a very limited amount (10%) is due to formal (theoretical) learning sessions.¹⁶ Therefore, peer reflection in clinical practice seems an essential part of effective education methods. To realize these in-depth reflections, a safe learning environment is essential.¹⁷ The environment should also include different contexts and challenges of real daily practice.¹⁸ Because the TP process involves a lot of implicit knowledge, it is difficult to manage RTT competences. Explicating implicit knowledge is beneficial and makes knowledge transferable; however, is difficult.^19,20 If we manage this, the adoption of AI in clinical practice has more potential as well.⁷ Although this seems worthwhile, few learning interventions manage to do this. We provide promising results on explicating implicit knowledge.

Little research in educational programs for RTTs is performed. Internationally, there is little conformity in RTT education and RTT tasks, resulting in large differences on RTT competency levels.^21,22 Lombarts describes that self-reflection, self-regulation, and pursuit of excellence are key in optimal professional performance for medical specialists.²³ Reflection among peers is becoming more integrated in medical education and radiation therapy, especially for TP competencies. Recent examples describe peer reflection on an international scale, which might be an opportunity to increase reflection even further and guarantee a level of experimentation and innovation.²⁴ An increasing number of research articles regarding learning models for automated TP software are being published. However, few publications have been published on how professionals have to interact with this automation.

This pilot involved only a small number of respondents, including only 2 groups of 6 professionals. Therefore, the presented results must be carefully interpreted, and no strong statistical conclusions can be drawn. However, we feel that these results are promising because they indicate a positive effect of this training method both in interobserver variation and throughput time. More extensive research in more contexts with more professionals and in other departments is recommended.

In the lung TP training, only 6 TP cases were incorporated. Because of the diversity in tumor locations and surrounding OARs, these cases represent many TPs in clinical practice. The level of complexity increased during training to ensure the learning curve of trainees. However, this is not a standalone training for students with zero TP competencies. Completion of the basic RTT training including basic TP competences is required to start this complex TP training.

We incorporated a timepoint 6 months after the peer session to evaluate sustainability of competences. This is a relatively short period of evaluation. We expect that repeating the peer sessions on periodic base would be necessary to keep up the complex TP competences.

Development of materials was relatively labor intensive, with about 48 hours spent on the treatment site–specific training preparation. However, this is only once for every training and will become less time consuming if employees/experts become acquainted. The peer session and all relevant materials are not only useful for students or new RTTs but can also be used to repeat training and guarantee high competence levels.

In the broader context, we expect that the structured training method, involving increased argumentation on decision making throughout the TP process (why), evaluated in this pilot can be useful for training of all tasks, including complex decision making. The process breakdown benefits the educational program and helps to standardize the process and reveal implicit knowledge through discussions of all relevant process steps. The 3 steps of job instruction method combined with creating a safe environment to practice skills and educate trainers in their training skills seem promising to enhance learning. Reflection within the individual and among peers is beneficial for learning, especially when reflection is focused on the decision making (why) instead of only looking at the outcomes (see Fig. 4). This combination of elements seems to improve efficacy, efficiency, and sustainability of training.

Figure 4.

Framework for sustainable education in complex settings.

Conclusion

Training based on job instruction elements combined with individual and peer reflection yielded positive results on the training's effectiveness, efficiency, and perceived value for a pilot of lung cancer TP training. Although developing training materials was labor intensive, the reduction of throughput time for students to become competent for complex TP to only 3 months was a striking benefit. In addition, interpersonal variance decreased, resulting in sustainably higher TP quality. This structured training method improved training efficiency in situations with complex decision making and can potentially be used in continuous education settings to maintain competences in radiation therapy TP and in other contexts based on implicit knowledge.

Disclosures

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix. Supplementary materials

Supplement A. roadmap how to design and perform training

Supplement A. Example of a roadmap how to design and perform training very complex skills with a large amount of implicit knowledge/ in-depth understanding

Supplement B. Example of a breakdown process

Supplement B. Example of a breakdown process sheet presenting the what, how and why for some key steps in lung TP

Supplement C. Increasing training difficulty

Supplement C. Example of increasing difficulty in training TPs

Supplement D. In-depth questions for lung TP

Supplement D. In-depth questions for lung TP to facilitate inter- and intra-person reflection