Framework Development: Standardizing Definition of Advanced Practice Radiation Therapy Activities for Clinical Workload Quantification

Highlights

• •First systematic attempt to measure the clinical impact of advanced practice (AP) radiation therapy roles.
• •Consensus building involving 20 advanced practice radiation therapists and 14 their managers.
• •List of 20 advanced practice (AP) activities and its definitions was generated.
• •Proposed framework that allows standardized and continuous monitoring of AP clinical impact.

Abstract

Purpose

Advanced practice (AP) in radiation therapy (RT) is being implemented around the globe. In an effort to advance the understanding of the similarities and differences in APRT roles in Ontario, Canada, a community of practice (CoP) sought ways to provide quantitative data on the nature of APRT clinical activities and the frequency with which these activities were being executed.

Methods

In 2017, a consensus building project involving 20 APRTs and 14 radiation therapy (RT) department managers in Ontario was completed to establish a mechanism to quantify APRTs’ clinical impact. In Round 1 & 2, expert feedback was gathered to generate an Advanced Practice (AP) Activity List. In Round 3: 20 APRTs completed an online survey to assess the importance and applicability of each AP Activity to their role using Likert scale (0–5). A final AP Activity List & Definitions was generated.

Results & discussion

Round 1: Forty-seven AP activities were identified. Round 2: 3/14 RT managers provided 145 feedback statements on Round 1 AP Activity List. The working group used RT managers’ feedback to clarify AP activities and definitions, specifically merging 33 unique AP activities to create 11 inclusive AP activities and eliminating 8 activities identified from Round 1. The most inclusive AP activity created was #1 New Patient Consultation, this AP Activity is merged from 7 unique AP activities. Incorporating RT managers’ feedback with the internal AP clinical workload lists from 2 Ontario cancer centres resulted in a revised AP Activity List with 20 AP inclusive activities. Round 3: 14/20 APRTs provided Likert scores on this revised list. The most applicable AP activities (mean score) were #16 Technical Consultation (4.0), #15 Contouring Target Volume (3.8) and #2 Planning Consultation (3.8); the least applicable was #18 MR Applicator Assessment (0.9).

Conclusions

This is the first systematic attempt to build consensus on AP clinical activities. Non-clinical APRT activities related to research, education, innovation, and program development were not in the scope of this project. The Final AP Activity List & Definitions serves as a framework that allows standardized and continuous monitoring of AP clinical activities and impact.

Introduction background

Advanced practice (AP) in radiation therapy (RT) is being implemented around the globe, however, it is clear that vast differences exist between the definitions and expectations of advanced practice radiation therapists (APRTs) from jurisdiction to jurisdiction [1] making concrete interpretation of what APRT is and what these professionals can do difficult at best. In addition to this issue, there is a paucity of data measuring the impact that APRTs have on their local departments or services, or on the system overall.

In an effort to advance the understanding of the similarities and differences in APRT roles in Ontario [2], [3], [4], and to make inroads on producing evidence of the impact these roles are having, one advanced practice community of practice (CoP) [5], [6] was formed to identify strategies for augmenting the kind of data being collected to demonstrate the impact of AP work on the radiation treatment system and on patient care. A community of practice is a group of experts gathered to promote knowledge creation and exchange. This CoP consisted of all APRTs hired in the province and representative from the provincial cancer agency. Very little has been written about quantitative workload measure for allied health professionals, and none exists for APRTs. As such, this CoP undertook a project to develop and implement a tool to provide quantitative data on the nature of APRT clinical activities and the frequency with which these activities were being executed [7].

Workload measurement is common in the Ontario health care system, as is the case in radiation treatment programs across the province [8]. The system uses a list of codes that RT professions select to confirm what RT activities they perform in the execution of their clinical work [9], [10]. These data permit analysis of workflow and workload in the system at one point in time or over time. Permission was sought and granted to use this system for the purposes of capturing APRT clinical workload data as well. In order to create the codes necessary for workload capture, the CoP undertook a consensus building project to prepare a standardized list of AP activities that could be incorporated into the existing workload data system for the quantification of clinical impact of existing APRTs [7]. This paper describes the processes used to develop a framework for consistent measurement of APRT clinical activities and workload in Ontario, Canada. In Ontario, the title ‘Clinical Specialist Radiation Therapist’ (CSRT) was assigned by the Ministry of Health and Long Term Care to describe APRT in Ontario, so the terms CSRT and APRT are used synonymously here [4], [11].

Methods

Selection of experts

The consensus building process involved different experts in three rounds of data collection. In rounds 1 & 3, all 22 members of the CoP were invited, which included 20 APRTs, one CoP advisor and one specialist from the Radiation Treatment Program at the provincial cancer registry. At the time of study, the 20 APRTs had an average of 6.8 years of experience in their advanced positions (range from 1 − 11 years) representing 7 different areas of specialization, across 8 cancer centres in Ontario (see Table 1). In round 2, all 14 RT managers in the province were invited to participate.

Table 1.

Years in APRT role of community of practice members (Q4 2017).

	Palliative	Brachytherapy	H&N	SBRT	Breast	Adaptive	Skin	Total
Senior (>8 yrs.)	2		2		1		1	6
Junior (4–8 yrs.)	3	3		1	1	1		9
New (<4 yrs.)	3			1		1		5

Abbreviations: H&N = head and neck, SBRT = stereo-tactic radiation therapy.

The advanced practice (AP) workload working group (WG)

The AP Workload WG consisted of 7 APRT CoP members: 6 APRTs (1 palliative, 2 brachytherapy, 1 head & neck, 1 stereotactic body radiation therapy and 1 adaptive radiation therapy APRTs) representing 3 cancer centres and 1 specialist from the Radiation Treatment Program at the provincial cancer registry.

Consensus building

Consensus building is a process involving a good-faith effort to meet the interests of all stakeholders and seek a unanimous agreement [12]. This project followed a trajectory similar to the modified Delphi technique, a qualitative research approach through gathering of expert opinions on a topic until reaching a consensus [13]. In the consensus building project, multiple rounds of data gathering from experts and data collection surveys, including pre-determined qualitative and quantitative questions, were used.

Data sources

The provincial cancer registry uses a workload code system [10] to document and evaluate patient related radiation therapy activities. The WG used this existing code system as the foundation to assign codes to patient related clinical activities performed by APRTs. The second data sources were two distinct internal AP clinical workload capturing lists that had been developed and implemented independently in 2 cancer centres that employed APRTs. Centre 1 and Centre 2 had identified 18 and 19 [14] AP activities respectively that their APRTs reported on regularly. (Appendix 1).

Round 1: AP Activity List generation

In Q2 2017, the WG invited 20 APRTs to identify and provide descriptions on the AP clinical activities in their respective roles via email. The clinical activities collected were anonymized and collated by the WG leads and the provincial cancer registry specialist and presented to the WG. The WG reviewed and compared these activities to the existing provincial RT clinical workload capturing system [9] to identify any duplication to generate a Round 1 AP Activity List (Fig. 1).

Fig. 1.

Summary of data collection.

Round 2: RT Managers’ feedback & internal AP lists incorporation

In Q3 2017, the CoP advisor and the provincial cancer registry specialist met with the RT managers to provide the rationale of tracking AP activities and to invite them to provide detailed feedback on (1) the appropriateness of the activity being tracked separately as an AP activity and (2) the clarity of the description of each activity. These comments were thematically analyzed by the provincial cancer registry specialist and shared with the WG for consideration in drafting the next version of the list. The WG also compared the Round 1 AP Activity List with the internal AP Workload Capturing Lists from Centre 1 and Centre 2 (Appendix 1). The WG members collated feedback from both sources and generated a Round 2 AP Activity List. The WG then defined each AP activity on this list to create a Round 2 AP Activity List & Definitions (Fig. 1).

Round 3: APRT feedback

In Q2 2018, all 20 APRTs in the CoP were invited to complete an electronic survey (SurveyMonkey^TM). Using a 5-point Likert Scale (5 = very frequently, 1 = not likely, 0 = not applicable N/A), participants were asked to rate the perceived frequency of undertaking each activity in their work for each of the AP Activities and also provided their feedback on AP activity definitions. This information was used to create the Final AP Activity List and Definitions. (Fig. 1).

Results

Three rounds of data collection was performed from Q1 2017 to Q4 2018. The consensus building project concluded with a Final AP Activity List and Definitions.

Round 1: AP Activity List generation

The WG gathered 47 AP activities performed by APRTs (Appendix 2). The AP activities collected were diverse and from every time-point of a typical radiation therapy patient care pathway: 1) triage, 2) new patient consultation: obtaining consent, physical exam, 3) treatment planning: technical consultation for planning, delineation of target, organ-at-risk (OAR) or field placement, 4) treatment delivery: image assessment, and 5) weekly review or follow up visits. The APRTs also provided a list of activities performed under a medical directive: ordering diagnostic imaging, lab work and medications independently and conducting virtual new patient consultations for patients that live from a distant from their cancer centre [15]. Finally the brachytherapy APRTs listed AP activities they performed in the brachytherapy patient care pathway. The WG categorized them into 4 categories: general activities, medical directives, telemedicine and brachytherapy. The WG recognized that AP activities would likely be different for APRTs in different institutions, including some identified AP activities that were deemed advanced practice in one institution could be a radiation oncologist or a regular radiation therapist duty in other RT departments, but the WG decided to keep these AP activities as part of the initial list for RT managers’ feedback. In this round of consensus building, Round 1 AP Activity List consisted of 47 AP activities was generated.

Round 2: RT Managers’ feedback & internal AP lists incorporation

The WG received responses from 3 of 14 RT managers (response rate = 21 %) on the Round 1 AP Activity List. They provided a total of 145 statements: general activities (95), medical directive (6), telemedicine (11) and brachytherapy (33) (Appendix 2). The WG sorted these comments into 3 themes for action: Advanced Practice Activities created (37), AP activities description that requires revision (62), AP activities eliminated from the AP Activity List (46). Key actions taken were extracted and presented in Table 2.

Table 2.

Actions taken based on managers’ comments.

New AP Activities Created	AP Activities Description Revised	AP Activities Eliminated
1.Creating a New Patient Consultation code to reflect APRT activities 2.Incorporating small clinical activities into one code, e.g. New Patient Consultation includes Patient assessment, Consent, Dictation 3.Conducting patient review and observation visits on behalf of radiation oncologist is considered advanced practice activity	1.Depending on which centre, the use of words “intake”, “case”, “patient” and “visits” requires standardization. 2.Patient Navigation is also performed by clinic clerks and regular radiation therapists in some centres 3.Referral to other health care professionals is also performed by regular radiation therapists in some centres 4.Patient Education is already performed by regular radiation therapists, therefore more detailed description is required to classified this activity as advanced practice 5.Contouring OARs is already performed as a regular planning radiation therapist, the experts suggested to add the approval of contour in the description to classify as advanced practice 6.Dose Accumulation is already performed by regular planning radiation therapist, to qualify for an advanced practice activity there needs to be an assessment component 7.Deformable image registration, image import/expert, image registration/fusion are already performed by regular radiation therapist, to qualify for an advanced practice activity there is an assessment and approval by CSRT, that may lead to corrective actions 8.Telemedicine codes need to be combined	1.Brachytherapy planning (even using MR images), intra-vaginal applicator insertion and removal, are already performed by regular brachytherapists in some centres and therefore does not consider advanced practice activity 2.If brachytherapy intra-vaginal applicator sizing involves a physical examination, it should be captured as physical exam instead

Using RT managers’ feedback, the WG merged 33 out of 47 unique AP activities identified in Round 1 to create 11 inclusive AP activities, namely #1 New Patient Consultation, #2 Planning Consultation, #3 Patient Education, #7 Clinical Examination, #9 Virtual pre-treatment consultation, #10 Virtual Interaction, #11 Virtual Follow Up Consultation, #12 Triage/Intake, #15 Contouring Target Volumes, #16 Technical Consultation, #17 Critical Image Assessment and Approval. From round 1, 6 activities identified by APRTs were kept, namely #5 Follow Up Consultation, #6 On Treatment Assessment, #8 Multi-disciplinary Pre-treatment Consultation, #14 Patient Navigation, #18 MR Applicator Assessment, #19 Dose Accumulation & Adaptive Decision. Finally, 8 activities were eliminated from Round 1.

Then, the WG further cross checked the list with the internal AP Clinical Workload Capturing Lists from Centre 1 and Centre 2 (Appendix 1) to decide on final action items. From this internal AP Clinical Workload Capturing Lists, the WG decided that #4 Treatment Completion Consultation, #13 Care Co-ordination and #20 Request Replan or Re-scan were also relevant APRT activities. As a result, the WG produced a Round 2 AP Activity List that consisted of 20 AP activities in 5 clinical competency categories: patient interactions, multi-disciplinary consult, virtual consultation, resource optimization and technical activities. Finally, the WG finalized definitions for each of these 20 AP activities to create the Round 2 AP Activity List & Definitions (Table 3).

Table 3.

Final AP activity list & definitions.

Code	AP Activity	Definition	Score
Patient Interactions
01	New Patient Consultation	APRT is present for consultation in clinic (new to centre or program or new disease sit/complaint not directly related to previous consultation) related to treatment decision and independently assesses and/or counsels the patient; reviews imaging. As a clinical examination is included in the new patient consultation code already, do not charge the clinical examination code for the same session.	3.4
02	Planning Consultation	APRT performs consultation with patient related to planning and/or treatment procedures after decision to treat has already been made (consult is done in planning). Maximum one code per treatment course.	3.8
03	Patient Education	APRT assesses and discusses with patient re: staging, outcomes and other information relevant for referral to other services or initiation of radiation planning	3.6
04	Treatment Completion Consultation	APRT assessment of patient upon completion of XRT to establish patient needs and follow up care plan	2.7
05	Follow Up Consultation	APRT is present at the follow up consultation and independently assesses and/or counsels the patient	2.9
06	On Treatment Assessment	APRT attends treatment unit or is paged to assess patient for symptom assessment and management (troubleshooting issues, non-scheduled)	2.7
07	Clinical Examinations	APRT obtains history, performs physical examination or bimanual examination. (Not to be charged during new patient consultation. Use the “New Patient Consultation” code instead.)	2.2
Multi-disciplinary Consult
08	Pre-treatment Consultation	Discussion with other health care providers to finalize patient’s treatment plan; includes review of imaging or other tests	3.2
Virtual Consultation
09	Virtual pre-treatment consultation/ assessment	APRT has discussion with patient regarding appointment preparation/ screening or performs pretreatment consultation	1.7
10	Virtual Interaction	Any virtual communications made to patient/family members outside of the definition of the virtual consultation/intervention	2.1
11	Virtual Follow Up Consultation	APRT performs virtual follow-up consultation with patient post treatment	2.1
Resource Optimization
12	Triage/Intake	APRT triages patient referrals to ensure appropriate/efficient clinical interactions for visits or procedures (Internal/external, new/return pts)	2.4
13	Care Co-ordination	Assessment of all aspects of patient treatment and care plan in order to ensure optimum patient experience and care (orders imaging, makes referrals or contacts other health care providers)	3.5
14	Patient Navigation	Researching and directing patients to services and resources; increasing access to care; removing barriers; facilitating treatment decisions	3.6
Technical
15	Contouring Target Volumes	APRT contours target volumes for the purpose of RT treatment planning. One code per treatment course	3.8
16	Technical Consultation	APRT consults with radiation therapy team members to provide instructions with respect to technical aspects of planning or treatment, QAs treatment plan. One code per instance.	4.0
17	Critical image assessment and approval	APRT reviews planning CTs, planning MRIs for acceptability or reviews CBCT registration as physician substitute to determine treatment initiation. One code per imageset/registration.	3.2
18	MR Applicator Assessment	APRT approves placement of brachytherapy applicator and positioning maintenance during pre-brachy MR procedure	0.9
19	Dose accumulation & adaptive decision	Delivered dose assessment – Perform dose of the day calculation, evaluation and recommend adaptive intervention. One code per assessment point.	1.7
20	Request replan or re-scan	Ordering additional imaging for response assessment/replan.	2.2

Round 3: APRT feedback

Fourteen electronic survey responses were received from the 20 APRTs in the CoP (70 % response rate). Mean scores ranged from 0.9 to 4.0 (5 = very frequently, 1 = not likely, 0 = not applicable N/A) (Table 3). The activities that were perceived as the most applicable for APRT roles were #16 Technical Consultation, #15 Contouring Target Volumes, #02 Planning Consultation with mean score of 4.0, 3.8 and 3.8 respectively. The AP activity that received the lowest score, i.e. the least applicable to APRTs, was #18 MR Applicator Assessment (mean score = 0.9).

Discussion

After many years of implementing pilot APRTs roles across Ontario, the incumbent APRTs, their managers and the APRT project leaders had gained extensive experience in defining AP and measuring its impact. It was felt that this formal consensus building exercise would harness this expertise and produce a final, validated list of AP clinical activities (and associated descriptions) that could be used locally to continue monitoring APRT impact, and potentially be adopted or adapted in other jurisdictions who are just in the development phase of AP to facilitate data generation without the extended exploratory phase.

Classification of “advanced” activities

The exercise to ascertain what element of an activity made it “advanced” was time consuming. To warrant a new, unique AP activity, an activity had to be clearly differentiated from the regular clinical RT activities that already had a code assigned. For example, for #03 Patient Education, the RT managers’ feedback and WG agreed that discussion of cancer prognosis is required in order to classify the patient education activity as advanced. This activity can either take place during a consultation or prior to CT/MR simulation.

Another example relates to contouring of target volumes and organs at risk (OARs). The Round 1 AP Activity List contained a number of tasks assigned to contouring activities, but RT managers’ feedback and WG discussion led the WG to decide that OAR contouring should be removed from the list because the completion of OAR contouring was being completed differently at the various clinical sites such that it could not be consistently included or excluded as a distinct AP activity. However, contouring of target volumes could consistently be considered an AP activity (i.e. #15 Contouring Target Volumes). Teasing out these particulars also revealed the need for a clear and consistent description for each of the codes so that they would be used in the same way by all APRTs.

Identifying / Standardizing APRT activities

The consensus building project took longer than expected (approximately 2 years) but was very valuable for the CoP and was necessary to ensure the highest quality product. One of the most important discoveries early on (Round 1) was the diverse nature of the APRT roles across the province. The 47 AP activities represented the work of 7 different APRT specialties, some with multiple APRTs fulfilling similar roles in different centres, others with only 1 of its kind in the province. Even when there were multiple APRTs working in similar roles (ie. palliative APRTs employed in 8 cancer centres [10], [11], [12]) how they worked very differently in their local environment. Some APRTs were delegated target contouring [16], [17], [18] activity while some APRTs performed independent activities such as RT planning orders and image approvals that the WG classified as Medical Directives [19]. This variability resulted in a list of activities that was less cohesive than originally anticipated.

The AP activities collected from APRTs (Round 1) also showed that it was important to create a separate Virtual Consultation category for clinical workload capture, because a number of APRTs were conducting virtual pre-treatment consultation and follow up consultations (AP Activity #9-11) for patients using provincial accredited video conference system for patients from a geographic distance. One APRT facilitated and reported on a virtual stereotactic radiotherapy consultation service for patients with disease that could be suitable for this treatment who lived at a distance from the facility. The implementation of the service resulted in significant time and cost savings for these patients residing, on average, 600 km outside of the Local Health Integration Network (LHIN). [15] It was also identified that there is a distinct skillset and competencies required to conduct virtual patient interactions compared to those conducted in-person. [20], [21] To ensure patient safety, additional training is required for the use of new technology and a modified approach to virtual assessment.

Grouping unique AP activities into inclusive AP activities

Identifying where AP activities were deemed similar enough to be categorized under a single inclusive AP activity or warranted a separate AP activity took much discussion and time. Efforts to make each AP activity relevant and specific enough while keeping it generalizable to more than one APRT was a challenging balance for the WG to strike. An example of this is the #01 New Patient Consultation which was originally divided up into 7 discrete AP activities and is now an inclusive AP activity that encompasses consultation, physical exam, dictation, consent, radiation prescription entry, medication prescription, independent tests ordering and/or interventional procedures ordering. (Appendix 2).

Methodology

In addition to the extent of discussion and negotiation required for the consensus building project, the WG also learned other lessons on the methodological approaches used in the consensus building exercise. Most notably, after reviewing the Likert score assigned to each AP activity in Round 3, it was decided that mean scores did not dictate the validity of each AP activity, hence would not dictate which AP activity remained on the list or not. Those with low scores (ie. #18 MR Applicator Assessment) did not necessarily have to be excluded due to low mean score (0.9), but rather reflected the specificity of the AP activity. Despite the fact that an activity was only performed by one APRT in the province did not negate its importance on the list. The WG learned that the exercise of assigning a Likert score to each activity was likely not as useful as originally thought and that the value in the consensus building project was more in deciding whether an APRT activity was an AP activity or not, regardless of how many APRTs performed it.

Department Buy-in

Finally, the WG felt quite strongly that department buy-in would be important throughout the process. The RT managers’ group was kept apprised of the work being done, including several face-to-face presentations and meetings. Despite almost unanimous agreement that AP activity measurement was important, response rate to the Round 2 was very low (3/14). The lower than expected response rate could have been due to time constraints and could have been impacted by the fact that only 8/14 of the managers actually employ an APRT in their department at that time. However, the 3 managers that did respond had extensive experience in designing and implementing APRT roles and they provided robust feedback that was extremely valuable to the process. The managers’ feedback served as a pivotal determinant in combining unique AP activities into inclusive AP activities and the elimination of the 8 activities performed by APRTs as identified in Round 1. (Appendix 2) It is expected that getting manager feedback on new AP activity that will be added in the future will be a less onerous task and may result in a higher response rate.

Also it was important to note that Centre 1 and Centre 2 had implemented internal AP clinical workload capturing lists prior to this initiative. While these centres endorsed and adopted the new AP Activity List & Definitions, they noted that some of their department specific AP activities, that were not included in the Final AP Activity List, would continue to be captured separately. For example, #13 Care Co-ordination, that tracks assessment of patient treatment plan including referrals to other health care providers, was originally represented by five different AP activities in Centre 1 to meet their goal of tracking the frequency of inter-professional interactions between palliative care, radiologist, referral physician, nurse and radiation oncologist. As such, they will continue to collect that information internally.

Strengths and weaknesses

One of the strengths of this work includes being the first systematic attempt to build consensus in APRTs and in RT departments for AP clinical activities measurement. To add to the rigor of the consensus building project, experts (the APRTs and the RT managers) consulted and commented anonymously on the Round 1 AP Activity List. The response rate of Round 3 was 70 % that is relatively high. The endorsement of the provincial cancer registry throughout the process also lent legitimacy to the formal adoption of the final list. This has implications for supporting sustainability of the APRT roles using this framework to quantify clinical workload of all APRTs.

One limitation of this work is the limited group of experts that can be drawn upon for input into the process. With APRTs being implemented in 8 of 14 centres in Ontario, and nowhere else in Canada at the time of consensus building project, the number of professionals familiar with the concept of APRT and its practical implications is low. Despite this, the in-depth discussions amongst the CoP members and the extensive feedback received by the managers provided rich and deep information upon which to make decisions.

The Round 3 online survey also did not record the specific APRT roles, or time in position, of the respondents as the WG had felt it was important to keep the survey responses anonymous at the time of data collection. It was therefore impossible to stratify responses based on area of specialization or level of experience.

With respect to the Final AP Activity List & Definitions itself, as mentioned above, it only included clinical activities performed by APRTs. Also, this Final AP Activity List & Definitions does not capture the duration of each AP activity, which could be important information to gather for sustainability and decision making. Finally, it is recognized that given how new the APRT role is and how it continues to evolve, the WG recognizes that this final list will need to be routinely reviewed and updated to reflect current APRT practice.

Future developments

The CoP will obtain approval to use the Final AP Activity List & Definitions and proceed with technical implementation into the RT-electronic medical record system. A pilot study to test the usability of the Final AP Activity List & Definitions in clinical practice is essential and plans are underway to implement this. Results of the pilot project will also inform the eventual assignment of “duration” to enhance the information provided from code capture.

Since the scope of this work was focused on recording clinical and patient-related AP Activity List, non-clinical APRT workload could not yet be tracked by this new framework. Future work will focus on developing mechanisms to capture other activities that are essential to APRTs’ responsibilities including, but not limited to, the development of patient education materials [22], [23], staff teaching/education/supervision [24], [25], research [26], [27], [28], [29], community outreach [30], workflow development [31], [32], [33], [34], [35], new technology integration, implementation, process improvements and evaluations [36], [37], [38], [39] and knowledge translation [40], [41], [42], [43].

Conclusions

This study presents a successful attempt in building consensus on definitions for the measurement of AP clinical activities in Ontario. The value of this consensus building exercise is that it harnessed the existing extensive experience of all APRTs in Ontario – knowledge and expertise that had been built over many years of experimenting with approaches to reliable and valid data collection for measuring the impact of AP. The Final AP Activity List & Definitions sets out a framework that will form the basis of a pilot project to analyze how consistently the AP activities are interpreted and used, as well as how generalizable the AP activities are across the diverse APRT roles and the different cancer centres. If proven to be useful, this framework could allow decision makers to measure and analyze the impact and contribution of APRTs in the health care system and make informed decisions on future strategic directions both inside Ontario and beyond.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix 1: Internal AP clinical workload capturing lists

No.	Centre 1	Centre 2	Description
1	NEW	New Patient Consultation	APRT is present for consultation in clinic (new to centre or program) related to treatment decision and independently assesses and/or counsels the patient; reviews imaging
2	CONS	Planning Consultation	APRT performs consultation with patient related to planning and/or treatment procedures after decision to treat has already been made (consult is done in planning)
3		In-patient Consultation	APRT consultation with in-patients on the ward to discuss treatment related issues and assess side effects (does in-patient need a separate code? Does it matter whether the consultation took place in clinic or ward?)
4		Treatment Completion Consultation	APRT assessment of patient upon completion of XRT to establish patient needs and follow up care plan
5		Follow Up Consultation	APRT is present at the follow up consultation and independently assesses and/or counsels the patient
6	OBS	On Treatment Assessment	APRT attends treatment unit or is paged to assess patient for symptom assessment and management (troubleshooting issues, non-scheduled)
7		Clinical Examinations	APRT obtains history, performs physical examination or bimanual examination
8	CALL	Phone - Assess	APRT provides assessment over telephone
9	TEACH	Specialized Education	APRT provides complex, specialized patient education that provides information beyond that covered with standard radiation therapy teaching
10–14	PC/RAD/REF/RN/RO	Pre-Treatment Consultation	Discussion with other health care providers to finalize patient’s treatment plan; includes review of imaging or other tests
15	CA	Virtual pre-treatment consultation/ assessment	APRT has discussion with patient regarding appointment preparation/ screening or performs pretreatment consultation
16	TCI	Virtual Interaction	Any virtual communications made to patient/family members outside of the definition of the virtual consultation/intervention
17	FLUP Call (15 m)	Virtual Follow Up Consultation	APRT performs virtual follow-up consultation with patient post treatment
18	Triage/Intake	Triage/Intake	APRT triages patient referrals to ensure appropriate/efficient clinical interactions for visits or procedures (Internal/external, new/return pts)
19	RTRN
20	PPC	Care Co-ordination (new)	Assessment of all aspects of patient treatment and care plan in order to ensure optimum patient experience and care (orders imaging, makes referrals or contacts other health care providers)
21	TPI (Patient Intervention)
22		Patient Navigation	Researching and directing patients to services and resources; increasing access to care; removing barriers; facilitating treatment decisions
23	TARGET	Contouring Target Volumes	APRT contours target volumes for the purpose of RT treatment planning
24		Technical Consultation	APRT consults with radiation therapy team members to provide instructions with respect to technical aspects of planning or treatment, QAs treatment plan
25			APRT reviews planning CTs, planning MRIs for acceptability or reviews CBCT registration as physician substitute to determine treatment initiation
26		MR Applicator Assessment	APRT approves placement of brachytherapy applicator and positioning maintenance during pre-brachy MR procedure
27			Delivered dose assessment – Perform dose of the day calculation, evaluation and recommend adaptive intervention.
28		Replan	Ordering additional imaging for response assessment/replan.
Total	18	19

Appendix 2: Round 1 & 2 results summary

No	Round 1 AP Activity	Description	Round 2 Managers' Feedback**	Actions
General Activities
1	Triage	Intake and review of patient for appropriateness for visits, procedures and treatment (Internal/external, new/return pts)	2; clarify intake, patient, visit definition	AP Activity #12
2	Communication	Patient or family (includes virtual ie. Emails, texts)	5	AP Activity #10
3	Clinical consultation without patient	Communication with health care professionals (includes virtual i.e. emails, texts) regarding a patient)	7; terminology coordination of care recommended	AP Activity #8
4	Consultation with patient	History taking /clinical exam /Review of imaging	2	AP Activity #1
5	Dictation	Dictating in RT-EMR in addition to the radiation oncologist dictation note	3; AP, incorporate to NP consult code	AP Activity #1
6	Technical consultation for planning, peer review and treatment	Peer review, QA plan, PTV and clinical structures and/or technical consultation on the treatment unit	5; AP must lead review	AP Activity #16
7	Scheduled Support	Review and Observation Visits	5; AP, conducts review visits	AP Activity #6
8	In person Follow up visits	Scheduled post XRT visits in follow up clinics in person	6; AP, used for f/u and treatment review	AP Activity #5
9	Consent	Obtaining informed consent from patients and/or POA	4: consent for tx or procedure	AP Activity #1
10	Rx Radiation	Completing radiation prescription entry	5	AP Activity #1
11	Rx Medication	Prescribe medication for a patient	4; incorporate to NP consult code	AP Activity #1
12	Patient Navigation	Researching and directing patients to services and resources as well as appointment and location navigation.	5; regular therapist or admins do this in some centres	AP Activity #14
13	Referral	Referral to other health care professional	3; requires judgement to create a referral	AP Activity #12
14	Patient Education	Educating patient on radiation procedures, possible side effects and management of those side effects.	5; clarification on what is AP Pt Ed vs. Regular Pt Ed	AP Activity #13
15	Telephone Follow up visits	Telephone follow up	4; done by regular therapist in some centres	AP Activity #11
16	SIMPLE Delineation of target/OAR/field placement	Target/OAR and field contouring based on complexity and time to complete	5; delineate or approve target is AP; OAR is not	AP Activity #15
17	COMPLEX Delineation of target/OAR/field placement	Target/OAR and field contouring based on co mplexity and time to complete	4; delineate or approve target is AP; OAR is not	AP Activity #15
18	Clinical Mark up	Performing a mark-up of a radiation field with a patient	2; delineate tx area	AP Activity #15
19	Image assessment and interpretation	Assessing rigid /deformable image registration	4; regular planner’s activity in some centres	AP Activity #17
20	Image import/export	Transfer of image sets	4; regular planner’s activity in some centres	Eliminated
21	Image registration	Fusion	4; regular planner’s activity in some centres	Eliminated
22	Image approval	Approval of an image registration	3; not needed in some centres	AP Activity #17
23	Dose Accumulation	Programming/computation /assessment	3; regular planner’s activity?	AP Activity #19
24	Independent test ordering	Order diagnostic imaging, lab work, prescription medication	2; remove the word ‘independent’ to gain consistency re: medical, procedure, RT Rx orders	AP Activity #1
25	Order interventional procedures	Order invasive procedures, peripheral IV insertion, access central lines, catheterization, rectum immobilization, gastric tubes, etc.	2; align interventional and invasive procedures with overall CCO definition	AP Activity #1
26	Perform invasive procedures	Peripheral IV insertion, access central lines, catheterization, rectum immobilization	2: appropriate to track	AP Activity #2
Telemedicine
27	Consent(T)	Telemedicine specific	3; combine telemedicine activities	AP Activity #9
28	NP Consultation (T)	Telemedicine specific	1	AP Activity #9
29	Follow-up (T)	Telemedicine specific	1	AP Activity #11
30	Communication (T)	Telemedicine specific	1	AP Activity #10
31	Education (T)	Telemedicine specific	1	AP Activity #3
32	Teaching (T)	Telemedicine specific	1	AP Activity #3
33	Image Assessment (T)	Telemedicine specific	2	AP Activity #17
34	Scheduled support (T)	Telemedicine specific	1	AP Activity #10
Brachytherapy
35	Insertion and assessment of intra-cavitary applicator	Insertion of intra-cavitary applicator (GYN, esophagus, lung, GU)	4: generalize insertion of applicator to all sites	AP Activity #18
36	Registration of catheter and applicator	Planning: registration of applicator/catheter	3: responsibility varies between centres (MRT, physicists)	Eliminated
37	Dose optimization	Planning: dose optimization & plan publishing	4: regular MRT’s duty in diff centres	Eliminated
38	Treatment accessory fabrication	Treatment accessories prep, not AP	4	Eliminated
39	Intra-vaginal applicator insertion	Insertion of intra-vaginal applicator (GYN)	3: generalized insertion of applicator to all sites	Eliminated
40	Applicator removal	Removal of Applicator	3: highly site dependent, intra-cavitary vs. interstitial	Eliminated
41	IVA sizing	Measuring of patient’s vaginal length in clinic	3: requires physical exam	AP Activity #7
42	Machine shop coordination	Coordination of customized applicators	4	AP Activity #16
43	Intra-op Assist for intra-cavitary or interstitial applicator insertion		1: only 1 centre has APRT performing this activity	AP Activity #2
44	TRUS for LDR Prostate Volume Study		1: only 1 centre has APRT performing this activity	AP Activity #2
45	Physical examination		1	AP Activity #1 & 7
46	Contouring: Target		1: responsibility varies between centres (MRT, physicists)	AP Activity #15
47	Contouring: OARs		1: responsibility varies between centres (MRT, physicists)	Eliminated

^**number of comments collected from managers; summary/theme of comments collected.