Students’ Perspectives on Their Experience in Clinical Placements: Using a Modified Delphi Methodology to Engage Physiotherapy Stakeholders in Revising the National Form

Abstract

Purpose: We developed an evidence-informed Student Evaluation of the Clinical Placement form. This form gives students the opportunity to share their feedback and perceptions of their clinical placement experiences and provides meaningful data to all stakeholders. Method: We used a modified Delphi process to engage a sample of national stakeholders: physiotherapy clinical education leads of academic departments, centre coordinators of clinical education, clinical instructors, and students. An expert consultant panel, in addition to the investigators, reviewed the responses from each round and helped develop the questionnaire for the subsequent round and finalize the evaluation form. Results: The response rate was 65.3% (47 of 72) for Round 1, 76.6% (36 of 47) for Round 2, and 100% (36 of 36) for Round 3. After three rounds of questionnaires, 89% of participants thought that the evaluation form met their needs. Conclusions: We developed a revised Student Evaluation of the Clinical Placement form that is informed by the literature and meaningful to all stakeholders. This form is being implemented in physiotherapy university programmes across Canada to enable students to share their experiences at clinical sites.

Clinical placements are an integral component of entry-to-practice physiotherapy programmes in Canada. Placements give physiotherapy students the opportunity to apply theory to practice in the clinical environment. Canadian physiotherapy programmes require students to complete a minimum of 1,025 hours in clinical education placements throughout their entry-level master’s degree physiotherapy programme.¹ Clinical placements occur at sites affiliated with each university.

Clinical placement agreements are used to establish a relationship between the university and the clinical sites. Although the terms of the agreements vary, in general universities agree to prepare and support the students during the placements, and the clinical sites agree to provide safe opportunities for students to apply their knowledge and skills and develop their clinical reasoning and professional competencies.¹ The agreements also include a notation that the clinical instructor (CI) – the term used to refer to a health professional, typically a registered physiotherapist – assigned to supervise a student during the clinical placement will evaluate the student’s performance on clinical placement and that students will have an opportunity to share feedback about their learning experiences with their CI and the clinical site. Once an agreement is established with a clinical site, the CIs at the site are oriented to the placement expectations and clinical placement processes by a university faculty or staff member. This orientation typically includes the placement evaluation forms used by both the students and the CIs.

Because universities are committed to strengthening their relationship with clinical sites and to establishing high-quality learning opportunities for their students, the feedback that students provide about their learning experiences is valued and meaningful to the other key stakeholders: the CI, site managers, and the clinical education team at the university. This feedback is also instrumental in developing high-caliber placement opportunities for future students at a clinical site.

To enable students to give their feedback at the mid- and final points of clinical placements – to evaluate their placement experiences with their CI and the hosting team at the clinical site – most Canadian physiotherapy programmes require them to complete a clinical placement evaluation form. This form reflects the students’ experiences at the clinical site and their engagement with the CI. At the end of the placement, students submit their completed form to their university’s physiotherapy programme.

Here are specific examples of how the information on a completed form can be used by different stakeholders:

• Academic clinical education leads of university physiotherapy departments use the information to learn about students’ experiences at a site, further develop partnerships or strengthen affiliations with sites, generate appreciation letters to host sites, and nominate CIs for awards.

• Centre coordinators of clinical education (CCCEs) are the individuals who coordinate placements at a clinical site. They use the information to learn about students’ experience at the clinical location. The feedback can also be used to reinforce their current practices or offer suggestions for improvement.

• CIs supervise students during a placement and receive feedback about their teaching and the students’ experience. CIs use this form to reflect on their teaching style and to identify strengths and opportunities for improvement when working with future students.

• Students provide input and feedback to their CI and site as well as to the university.

The National Association for Clinical Education in Physiotherapy (NACEP) consists of the clinical education leads from each physiotherapy university programme in Canada. This group provides leadership in clinical education in the broad context of education, practice, and research in the country. Part of NACEP’s role is to establish national assessment or evaluation processes for clinical education supported by ongoing research and evaluation. NACEP members also identify and collaboratively share resources that promote and support clinical education for physiotherapy.

It was as a result of the mandate to develop and share resources that the original Student Evaluation of Clinical Internship form was developed more than 10 years ago. However, university programmes, CCCEs, CIs, and students note that it has not been updated since then, and it is not clear whether it was based on best practice in developing evaluation forms, such as including the perspectives of multiple stakeholders. For example, Courtney-Pratt and colleagues² have reported on the development of a quality evaluation for acute care nursing placements, which included the perspectives of both student and nursing student supervisor. The evaluation embraces a sense of welcome and belonging and support to meet learning needs.² Therefore, NACEP identified the need to revise the national form in keeping with best-practice teaching behaviours in clinical education and to provide meaningful information to all stakeholders.

The purpose of this research study was to develop an updated, evidence-informed Student Evaluation of the Clinical Placement form that would be concise and meaningful to all stakeholders by engaging with Canadian physiotherapy academic departments, CCCEs, CIs, and physiotherapy students.

Methods

Delphi process

The Delphi process is an accepted method for building consensus among a group of people.³ It consists of multiple requests (known as rounds) for stakeholder input to gain consensus on topic areas. A traditional Delphi method uses open-ended questions to collect information in the initial round, and a modified Delphi uses a structured instrument to begin the Delphi process (e.g., close-ended questions) and as the platform for subsequent questionnaires.³ Using a modified Delphi process is considered an appropriate option if information about a project area is available.³

In this study, we used the Web-based tool SurveyMonkey (SurveyMonkey, San Mateo, CA) to distribute an online questionnaire that consisted of both open-ended and close-ended questions (modified Delphi) to each participant for each round of the consensus-building process. Individuals who completed one round were invited to participate in subsequent rounds. Participants were given a summary of anonymous and aggregated responses from the previous round, allowing them to reflect on their own and the group’s responses. This study was approved by the Health Sciences Research Ethics Board at the University of Toronto. All participants provided informed consent.

To review the individual stakeholder responses, a consultant expert panel was developed. This panel consisted of the investigators and five volunteers from NACEP representing British Columbia (one), Ontario (three), and Quebec (one). The role of the panel was to review the responses and develop the questionnaire for the next round of the modified Delphi process.

Participants

We used a purposive sample by means of a snowball sampling method.⁴ Clinical education academic leads from Canada’s 15 entry-to-practice university physiotherapy departments, as well as the three academic regional coordinators – who coordinate clinical education for certain geographical areas and work collaboratively with universities (two from northern Ontario, one from Newfoundland and Labrador) – were invited to participate in the study (hereinafter termed academics). All academics were then asked to recommend one person from each of the other subgroups of key physiotherapy stakeholders: current students, CIs, and CCCEs. To ensure national and diverse representation of stakeholders engaged in physiotherapy clinical education, the academics recommended CIs from both privately and publicly funded clinical sites. This snowball sampling method ensured representation from all key stakeholders across the country; it allowed for a maximum of 72 participants. Students were eligible to participate if they had successfully completed at least two full-time caseload-carrying placements. CIs were eligible to participate if they had supervised a student in the previous year. CCCEs were eligible to participate if they had coordinated physiotherapy student placements at their clinical site in the previous year.

Questionnaires

The modified Delphi process was used to gain consensus about which clinical education teaching behaviours identified by the participants would be included on the final clinical placement form. We generated a list of items to be included in the first round of the study; these items were derived from the literature and from evaluation forms currently used in both Canadian and international (United States, Ireland) physiotherapy programmes, if publicly available. Because the research methods varied across the literature, we extracted the items from each study in a slightly different way. For example, from the study by Greenfield and colleagues,⁵ the six items that we extracted were the behaviours that the novice physiotherapists thought were most important in their experience. From the study by Lee and colleagues,⁶ we extracted the 10 highest rated characteristics of effective clinical educators, as rated by students, nursing students, and clinical educators. The questionnaires were available in English only, and all sections in all three rounds included text boxes to capture open-ended comments.

Rounds 1 and 2

In Rounds 1 and 2, the questionnaire had three sections: Section A, consider the items that should be included; Section B, consider the response scale that should be used on the form; and Section C, demographic information.

Section A of Round 1 asked participants to respond to the question “The following items should be part of the revised form completed by students regarding their perceptions of their learning experiences at the clinical site” on a 5-point Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree). In Section A of Round 2, participants were asked, “Should these behaviours be part of the revised form completed by students regarding their perceptions of their learning experiences as the clinical site?” and responded either include or do not include.

In Section B of Round 1, participants were asked to rate four rating scales, which would be used for each item on the evaluation of the clinical placement form. (These scales are shown in Box 1.) The following response options were provided: I like this scale, I am indifferent about this scale, and I don’t like this scale. Participants were also asked to respond yes or no to the question “Should each item have a non-applicable option?”

Box 1. Round 1, Section B: Four Rating Scales.

Strongly disagree/disagree/neutral/agree/strongly agree

Poor/fair/good/very good/excellent

This did not occur/This occurred sometimes/This occurred consistently

Disagree/neutral/agree

In Section B of Round 2, participants were asked to identify their preferred rating scale from the following two options: strongly agree/agree/neutral/disagree/strongly disagree or strongly agree/agree/disagree/strongly disagree. The difference between the rating scales was that the first option was a 5-point scale that included a neutral response.

Section C of Rounds 1 and 2 posed demographic questions about each participant’s subgroup, province of work (study for students), years of experience in clinical education, age, and gender identity.

Round 3

Round 3 consisted of Sections A and B. Section A consisted of a draft evaluation form, which students would complete to reflect their experiences at the clinical site; the form was based on the responses from the previous rounds and input from the expert consultant panel. Participants were asked to answer yes or no to the following question: “This proposed form to be completed by students regarding their perceptions of their experiences at the clinical site will meet my needs.” Section B included demographic questions about each participant’s subgroup, province of work (study for students), years of experience in clinical education, age, and gender identity.

Analysis

For each round, the response frequency for each question in each section was calculated using Microsoft Excel 2016 (Microsoft Corporation, Redmond, WA). The summary of responses and all of the open-ended comments were then reviewed independently by each expert on the consultant panel and collaboratively by the panel as a whole. An a priori decision was made to include items that received 80% or more approval. Before the expert panel meeting, all open-ended text responses were also sent to the panel for review. During the meeting, each of the comments was considered, and members of the panel decided on an action for each. In Rounds 1 and 2, the panel also developed a draft of the subsequent questionnaire, which was finalized by two investigators (Mori and Quesnel) and distributed by Mori. These investigators determined a priori that consensus for Round 3 would be achieved if 80% or more of the participants had responded yes to the draft form presented in Section A of Round 3. After Round 3, the expert panel reviewed the responses and all open-ended comments and finalized the evaluation form. For each round, demographic data from Sections A and B were analyzed separately and distributed as a summary. To protect the participants’ anonymity, only Mori had access to all the downloaded data.

Results

Item identification, Round 1

A literature search was conducted to identify studies in the health professional literature that examined effective teaching behaviours and teaching behaviours that were valued by students from a variety of health professions. We reviewed studies from physiotherapy,^5,7 nursing,^6,8–¹³ medicine,¹⁴–¹⁷ and dentistry.¹⁸

Those teaching behaviours that were identified as the most helpful or valued by faculty, learners, or both were captured verbatim and recorded in a spreadsheet by two investigators (Mori and Quesnel). For example, the teaching behaviour “Provides support and encouragement to students,” from Ludin and Fathullah, was included in the list.¹¹ Table 1 shows the study, sample, and number of behaviours extracted from the study.

Table 1.

Source articles and resources used to develop teaching items in round 1

Study	Type	Sample	No. of behaviours extracted
Babenko-Mould et al. (2012)8	Survey	Undergraduate RNs	5
Chitsabesan et al. (2006)9	Interview study	Postgraduate RNs, undergraduate RNs, nurse practitioners, MDs	6
Conigliaro & Stratton (2010)14	Observed practice	MDs	15
Fluit et al. (2010)19	Systematic review of questionnaires used to assess clinical teachers in medical education	Medical clerks, residents, or peers for assessing clinical teachers	8
Goertzen et al. (1995)15	Interview study	MD clerks’ ratings of rural family medicine preceptors	7
Greenfield et al. (2014)5	Interviews and focus groups with novice physiotherapists	PT students	6
Jarski et al. (1990)7	Survey	PT students	20
Keely et al. (2010)16	Completion of teaching encounter cards by 3rd- and 4th-year clerks	MD clerks	9
Lee et al. (2002)6	Survey	Undergraduate RNs	20
Li (1997)10	Survey	Undergraduate RNs and nurse educators	21
Ludin & Fathullah (2016)11	Survey	Undergraduate RNs	10
Mogan & Knox (1987)12	Survey	Undergraduate RNs	22
Polyzois et al. (2010)18	Survey	Dentistry students and faculty	5
Schronrock-Adema (2012)20	Survey	MD residents	15
Sweet & Broadbent (2017)13	Survey	Undergraduate RNs	43
American Physical Therapy	Physical Therapist Student Evaluation: Clinical	National form	32
Association21	Experience and Clinical Instruction (2003)
National Association for Clinical Education in Physiotherapy	Student Evaluation of the Clinical Internship (unpublished)	National form	45
Université d’Ottawa/University of Ottawa	PT student’s evaluation of the clinical placement (unpublished)	University-specific form that was shared with the PI	17
University of Limerick	Clinical Education Quality Audit (unpublished)	University-specific form that was shared with the PI	12

RN = registered nurse; MD = medical doctor; PT = physiotherapy; PI = principal investigator.

A total of 318 teaching behaviours and characteristics were extracted from the reviewed studies and existing forms. Duplicate behaviours and characteristics were removed, and similar items were grouped together so that 108 teaching behaviours and characteristics remained.

Round 1 questionnaire results

To facilitate completion of the Round 1 questionnaire, Mori and Quesnel grouped the 108 teaching behaviours from the literature search into 13 categories representing similar concepts (Table 2). The questionnaire presented four response scale options for teaching behaviours or characteristics; the responses are also presented in Table 2. Table 3 shows the responses to the demographic questions for all three rounds.

Table 2.

Round 1 Questionnaire: Categories, Number of Items, and Responses to Rating Scale Questions

Section A: Consider the items that should be included
Category	No. of items
Orientation	8
Objectives	2
Environment	4
Opportunities	15
Team	5
Teaching	18
Clinical reasoning	15
Interpersonal	7
Role modelling	6
Feedback	8
Communication	8
Assessment	8
Other	3
Section B: Consider the response scale that should be used on the form
Rating scale option	Response, no. (%)
	I like this scale	I am indifferent about this scale	I don’t like this scale
Strongly disagree/disagree/neutral/agree/strongly agree (n = 47)	40 (85.1)	7 (14.9)	0
Poor/fair/good/very good/excellent (n = 45)	9 (20.0)	15 (33.3)	21 (46.7)
This did not occur/This occurred sometimes/This occurred consistently (n = 46)	18 (39.1)	16 (34.8)	12 (26.1)
Disagree/neutral/agree (n = 44)	7 (15.9)	14 (31.8)	23 (52.3)
Should the rating scale have a non-applicable option? (n = 47)
Yes	26 (55.3)
No	21 (44.7)

Table 3.

Demographic Information Provided in Rounds 1, 2, and 3

Characteristic	Round; no (%) of participants*
	1; n = 47	2; n = 36	3; n = 36
Experience in clinical education, y, average (range); no. of missing values Participant subgroup†	12.77 (0-35); 2	15.32 (1-40); 2	16.15 (1-40); 3
Academic clinical education lead	17(36.2)	16(44.4)	16(44.4)
CCCE	9(19.1)	8 (22.2)	8 (22.2)
Cl	11 (23.4)	8 (22.2)	10(27.8)
Student	10(21.3)	7(19.4)	7(19.4)
Province represented‡
British Columbia	4 (8.5)	4(11.1)	4(11.1)
Alberta	3 (6.4)	1 (2.8)	1 (2.8)
Saskatchewan	3 (6.4)	3 (8.3)	3 (8.3)
Manitoba	3 (6.4)	3 (8.3)	3 (8.3)
Ontario	17(36.2)	10(27.8)	11 (30.6)§
Quebec	13(27.7)	11 (30.6)	10 (27.8)§
New Brunswick	1 (2.1)	1 (2.8)	1 (2.8)
Nova Scotia	2 (4.3)	1 (2.8)	1 (2.8)
Newfoundland and Labrador	1 (2.1)	1 (2.8)	1 (2.8)
No response	0 (0.0)	1 (2.8)	1 (2.8)
Gender identity
Female	34 (72.3)	26 (72.2)	27 (75.0)§
Male	12(25.5)	8 (22.2)	8 (22.2)
No response	0 (0.0)	1 (2.8)	1 (2.8)
Prefer not to answer	1 (2.1)	1 (2.8)	0§
Age group, y
18-24	4 (8.5)	2 (5.6)	3 (8.3)§
25-34	11 (23.4)	9 (25.0)	9 (25.0)
35-44	13(27.7)	8 (22.2)	7 (19.4)§
45-54	8(17.0)	8 (22.2)	7 (19.4)§
55-64	9(19.1)	7(19.4)	8 (22.2)§
65-74	1 (2.1)	1 (2.8)	1 (2.8)
Did not respond	1 (2.1)	1 (2.8)	1 (2.8)

^*Unless otherwise specified.

^†Some participants selected multiple categories.

^‡There was no representation from Prince Edward Island or Yukon, Northwest, and Nunavut Territories

^§Individuals changed their answers between Rounds 2 and 3.

CCCE = centre coordinator of clinical education; CI = clinical instructor.

Round 1 had an overall response rate of 65.3% (47 of 72 participants). Of the 108 final items we obtained, 32 received 80% or higher approval (participants agreed or strongly agreed that they should be included), and 31 received 70%–80% approval. Because many items had a high approval rating, the expert consultant panel attempted to combine those that were repetitive to minimize duplication. This was done by reviewing items and developing wording that broadly captured the specific concepts of two or more items. Most participants (85.1%) preferred a rating scale with the wording strongly disagree to strongly agree. Many (55.3%) were in favour of including non-applicable in the rating scale.

Round 2 questionnaire results

Participants who completed the Round 1 questionnaire were invited to participate in Round 2, and the response rate was 76.6% (36 of 47 participants responded). We developed the round 2 questionnaire on the basis of the responses from Round 1 and the suggestions from the expert consultant panel. In Round 2, participants shared their opinions on the suitability of 40 teaching behaviours or items and voted on an overall statement. Table 4 presents a summary of the quantitative results from Round 2.

Table 4.

Round 2 Responses

Item	Responses; no. (%)
	Include	Total no.
Section 1
1. I was adequately oriented to emergency and safety procedures.	30 (83.3)	36
2. I was adequately oriented to the site and clinical area (e.g., equipment, supplies, workload measurement statistics, documentation standards).	34 (94.4)	36
3. The expectations of the internship, roles, and responsibilities were discussed in the first week of the internship (e.g., learning/teaching style, preferred methods of feedback).	34 (94.4)	36
4. The student and the Cl discussed, negotiated, and revised the student's specific learning goals and objectives throughout the placement to ensure that the placement met the needs of both parties.	31 (86.1)	36
5. The student and Cl established a relationship that was collegial, non-threatening, and respectful of both the student and Cl.	21 (58.3)	36
6. The Cl helped me identify and make use of practice opportunities (e.g., patient meetings, variety of conditions) and resources (e.g., in-services) to augment my knowledge and learning.	28 (80.0)	35
7. I was expected to take responsibility for my learning, with support and encouragement from my Cl.	26 (74.3)	35
8. I was provided supervision/autonomy/independence appropriate to my level/experience/competence.	31 (86.1)	36
9. There was adequate opportunity to formulate assessments, develop treatment plans, and progress interventions/treatments.	34 (94.4)	36
10. There was opportunity to create discharge plans for patients/clients.	23 (65.7)	35
11. The amount and type of supervision I received helped me maximize my growth and development in this setting.	20(57.1)	35
Section 2
1. The Cl was approachable and accessible.	22(61.1)	36
2. Time was available with the Cl to discuss patient/client management.	29 (82.9)	35
3. The Cl shared their knowledge and experience.	15(44.1)	34
4. The Cl facilitated the development of my skills (e.g., interviews, assessments, clinical skills and techniques, analysis, intervention plans, communication, collaboration).	31 (88.6)	35
5. The Cl helped me develop my clinical reasoning skills.	16(45.7)	35
6. The Cl asked me questions to elicit underlying clinical reasoning (e.g., clarifications, probes, reflective questions, etc.).	17(48.6)	35
7. The Cl encouraged me to use evidence to support clinical practice.	25 (69.4)	36
8. The Cl encouraged the use of critical thinking through a reflective process of questioning and discussion.	20 (58.8)	34
9. The Cl provided support and encouragement to the student.	21 (60.0)	35
10. The Cl appeared open minded and non-judgmental.	19(54.3)	35
11. The Cl role modelled clinical skills and clinical reasoning.	21 (60.0)	35
12. The Cl was a resource for clinical skills and clinical reasoning.	21 (58.3)	36
13. The Cl modelled appropriate professional and ethical behaviours.	30 (85.7)	35
14. The Cl modelled effective patient and/or family communication skills.	26 (74.3)	35
15. The Cl demonstrated skill in active listening.	15(42.9)	35
16. The Cl facilitated communication by initiating discussion that may have been difficult or confrontational around an issue of concern.	17(47.2)	36
Section 3
1. I had opportunity to collaborate with others (e.g., health care team, administrative staff, insurance professionals), which facilitated my learning in this setting.	31 (86.1)	36
2. I was provided timely feedback on my performance.	19(55.9)	34
3. I received ongoing and regular feedback.	21 (58.3)	36
4. I was encouraged to reflect and self-assess my performance.	28 (80.0)	35
5. The feedback I received objectively identified my strengths and areas for improvement.	24 (68.6)	35
6. Based on the feedback received, a plan was developed to provide opportunities for ongoing improvement.	25 (69.4)	36
7. I was encouraged to provide feedback to others.	11 (30.6)	36
8. The Cl was receptive and responsive to my feedback.	26 (74.3)	35
9. The Cl observed me to inform their assessment of my clinical performance.	22 (64.7)	34
10. The CI's written evaluations of my performance were consistent with the informal formative feedback I received throughout the experience.	27 (75.0)	36
11. The Cl linked the grading on my evaluation to specific examples of my performance as an objective and fair representation of my performance.	30 (85.7)	35
12. The midterm evaluation was useful in identifying additional learning needs related to meeting the objectives for this clinical experience.	31 (91.2)	34
13. The CCCE was helpful and available in dealing with clinical internship issues.	28 (80.0)	35
As an overall item, please select your preferred wording between the two options below
Overall, the internship provided the learning experience required to develop competency in this area of practice, appropriate to my clinical level.	27 (77.1)	35
I felt the placement was an overall valuable educational experience.	8 (22.9)
We received contradicting comments regarding a 5-point response scale (includes a neutral option) versus a 4-point response scale (no neutral option). Because the strongly agree-strongly disagree scale was clearly preferred, we are asking you to choose between two options: a 5-point or a 4-pointresponse scale. Please select your preferred scale below.
Strongly agree/agree/neutral/disagree/strongly disagree	23 (63.9)	36
Strongly agree/agree/disagree/strongly disagree	13(36.1)
In Round 1, you were asked whether the rating scale should have a non-applicable item. The responses were nearly split, with 55% of participants indicating that they would like a non-applicable item. In reviewing the data from Round 1, the expert consultant panel thought that the items presented in Round 2 were applicable to a wide variety of practice settings and therefore is proposing a response scale without a non-applicable response option. Do you feel strongly that a non-applicable option should be included on the response scale?
Yes	21 (58.3)	36
No	15(41.7)

CI = clinical instructor; CCCE = centre coordinator of clinical education.

In reviewing the Round 1 data, the expert consultant panel believed that the items presented in Round 2 applied to a wide variety of practice settings and proposed a response scale without a non-applicable option. Therefore, in Round 2 participants were asked, “Do you feel strongly that a non-applicable option should be included on the response scale?” Most participants (58.3%) responded yes.

In this second round, 63.9% of the participants preferred to have a neutral option in the 5-point agreement rating scale. In the open text comment boxes, they questioned the response option neutral. These insightful comments guided the investigators and expert panel in making decisions about neutral and non-applicable options. For example, participants commented,

Such a difficult word to understand! Can anyone ever be “Neutral”…? or are they just trying to be careful and cautious…? “Neutral” has pros and cons. If the majority chose “neutral” its root cause will be harder to determine than if people chose agree/disagree. Can it be replaced with “did not observe” or something to indicate why the rater has no opinion? Or force comment if “neutral” is chosen? (Round 2, Participant 3)

Sometimes neutral might be used if not observed or just feeling things were OK yet not great. (Round 2, Participant 35)

I think a neutral or a non-applicable [option] is important in any scale as there are some questions that you might just not feel one way about or the other. (Round 2, Participant 33)

I like the scale without neutral. Choosing neutral is non-committal and doesn’t give the site good information. Some people may be inclined to choose neutral rather than disagree. (Round 2, Participant 7)

If there is a “not assessed” option, then I wouldn’t need the neutral. (Round 2, Participant 9)

Round 3 questionnaire results

In Round 3, we developed a draft student evaluation of the clinical placement form, one that integrated all the feedback from the previous rounds, and shared it with the participants. The response rate for Round 3 was 100% of eligible participants (36 of 36 participants responded). A total of 88.6% of participants voted yes on the following: “After reviewing the information items, please answer this question. This proposed form to be completed by students regarding their perceptions of their experiences at the clinical site will meet my needs.”

As a result, we obtained consensus on the draft form from the participants. A total of 20 individuals submitted open-ended comments that were reviewed by the investigators and expert consultant panel. After the investigators and expert panel discussed the open-ended comments, minor changes were made to the draft form, and the investigators and expert consultant panel agreed on the final evaluation form (reproduced in the Appendix.)

Summary of open-ended comments

Throughout the modified Delphi process, open-ended comments about one questionnaire informed the decisions made for the next one. Sometimes the comments reinforced the fact that the process was effectively moving toward updating the evaluation form:

Looks like a good collection of questions to ask important specifics about student perception of their placement experience and retains important overall commentary on the most positive aspects of their placement and areas for improvement of the placement experience. The number of questions does not seem like an overly long list (good). Reviewing these questions at both midterm and final evaluation of the placement would be useful. (Round 3, Participant 23)

Looks great – more concise and relevant. (Round 3, Participant 14)

However, the open-ended comments also provided useful suggestions that challenged the investigators and expert panel to consider different perspectives.

I feel that 8 and 11 are repetitive – they could be combined or 11 removed. I am concerned about the use of the word collegial in number 5 – as the student may interpret this to be more friendly than required in a student–instructor relationship. If the word collegial was removed to include only non-threatening and respectful then I would agree with including it. (Round 2, Participant 28)

Seeing it laid out succinctly under these headings, I see more overlap and redundancy now, and think it could be consolidated even more. S. 1, Items 4–8 are all CI evaluation and belong in S. 2. This may require some merging of items, such as S 1.5 and S 2.3. (Round 3, Participant 3)

Section 3, number 6 isn’t clear. Is the student providing performance feedback to patient families or are they soliciting feedback from these sources? (Round 3, Participant 16)

I do not like the scale – with undecided as the middle option – I prefer a neutral. I think the student needs to have “decided” on the items. (Round 3, Participant 2)

I would like to see the students’ perception of the CCCE’s role as well. This role is NOT operationalized equally in different organizations. However, feedback from students is necessary when justifying and evaluating this key role. (Round 3, Participant 29)

Items on the evaluation form

The final form is composed of three sections. The first section, “The Site,” includes items regarding orientation, expectations, learning environment, opportunities, and working with the CCCE. The second section, “The Clinical Instructor,” reflects the teaching behaviours of the CI and includes items such as support, encouragement, availability, facilitation of skills and clinical reasoning, modelling of effective practice, and responsiveness to feedback.

The third section, “Feedback and Assessment of the Student,” includes items regarding the exchange of feedback and assessment of the student. Some items (marked with an asterisk on the final questionnaire) were deemed to be not mandatory – such as Section 1, Item 7 (“There was adequate opportunity to: conduct assessments and analyze findings; develop and progress treatment plans; deliver treatment interventions; plan for discharges”), because not all placements are clinical in nature or have opportunities for discharge and Section 1, Item 9 (“The CCCE was helpful and available in dealing with clinical placement issues”), because a CCCE might not be present at all placements.

Discussion

The purpose of this research study was to engage a sample of national stakeholders – physiotherapy clinical education leads of academic departments, CCCEs, CIs, and students – to develop an evidence-informed student evaluation of the clinical placement form. This form would give students the opportunity to share their feedback about their clinical placement experiences and provide meaningful data to all stakeholders.

Sample

We invited the clinical education leads from 15 universities and three regional coordinators to participate in our study; each was then asked to identify representatives from the three other stakeholder groups. This yielded a maximum sample of 72 participants. Round 1 had an overall response rate of 65.3%; Round 2, a 76.6% response rate; and Round 3, a 100% response rate, demonstrating a committed sample. Although the response rate for Round 1 was low, it is similar to that of other modified Delphi studies.²² The response rates for Rounds 2 and 3, however, exceeded recommended response rates (≥ 70%) for Delphi studies.²³ The consistently high response rates in Rounds 2 and 3 support the validity of the results from these rounds.²³

Although we had representation from most areas of Canada, approximately one-half to two-thirds of the participants came from Ontario and Quebec. This reflects our snowball recruitment strategy: because there are five university programmes and two regional coordinators in Ontario and five university programmes in Quebec, there was an increase in representation from those two provinces. This representation is similar to the Health Workforce Database produced by the Canadian Institute for Health Information (CIHI) in 2016,²⁴ which reported that 58.1% of physiotherapists in Canada work in Ontario (38.0%) and Quebec (20.1%). Participants who identified as female in our study varied between 72.3% and 75.0%. This range is similar to the national CIHI data from 2016,²⁴ which reported 74.4% as being female.

There was a higher representation of younger (aged <34 y) and older (aged >55 y) participants in Round 3 compared with the national data. In our study, 33.3% of the participants in Round 3 were aged 34 years or younger (among those who responded to the question about their age group), compared with 30.3% in the national dataset. This is reasonable because our sample included students, who are not represented in the CIHI dataset. Our sample was also slightly older than the national dataset: 25.0% of the participants were aged 55 years or older, and the national dataset reports that 16.3% of physiotherapists are aged 55 years or older.

The largest stakeholder group in our round 3 sample was the academic clinical education leads category at 44.4%. CIs formed the next largest category at 27.8%, followed by CCCEs at 22.2% and students at 19.4%. Although it would have been desirable to have equal representation from all stakeholder groups across all three rounds, the investigators and expert consultant panel were satisfied with the representation.

This comparison of our sample with national data on the physiotherapy profession indicates that the participants in our study could be considered representative of those who are currently working in the profession.

Value of open-ended comments

Over the course of the study, the expert consultant panel valued the comments entered in response to the open-ended questions. Occasionally, the panel considered the open-ended comments to be so insightful that they took precedence over the a priori 80% consensus cutoff. Therefore, the investigators and expert panel used a collaborative decision-making process to reach a consensus for the draft of the new evaluation form. In Round 1, many items received 80% or higher approval; therefore, the investigators and panel needed to determine which items would stay and which would be eliminated from future rounds.

The comments provided in the text boxes proved especially helpful for making decisions about having non-applicable and neutral response options. For example, on the basis of the conflicting comments from Round 2, an undecided option was presented in Round 3 to replace neutral on the rating scale. Participants, however, expressed concern about undecided; therefore, the final form includes neutral as the middle option. It is important to note that 88.9% of the participants accepted the form presented in Round 3, which affirmed that it was suitable to meet the needs of all stakeholders and that an acceptable process had been used.

Final form

The items included on the final evaluation form are teaching behaviours that reflect effective teaching practices and quality educational experiences in clinical education.^2,25,26 Many items reflect the factors that Hutchinson identified as contributing to an effective educational climate that would be motivating and relevant to a student.²⁵ Items in the sections “The Clinical Instructor” and “Feedback and Assessment of the Student” reflect the recommendations of Lefroy and colleagues for the process of obtaining feedback and its content in clinical education.²⁶ The items in “The Site” and “The Clinical Instructor” sections identify important teaching behaviours because they establish a positive learning culture, which Watling²⁷ stated was the forerunner to gaining effective feedback and to establishing effective rapport between the CI and student. These are also critical teaching behaviours that contribute to the R2C2 Facilitated Feedback Model²⁸ in which building relationships and exploring reactions is foundational to help explore the experiences and ultimately coach for change in the student’s performance.

Variability in responses to demographic questions

For each round of the modified Delphi process, participants were asked to complete the final section of the questionnaire, which contained the demographic questions. There was some intra-participant variability in how the same individual answered these questions, such as the province where they worked and their age group category, despite the questions being the same. It is unclear why this variability occurred, and in future, it may be preferable to capture the demographic data from participants only in Round 1. This would minimize the response burden for participants in subsequent rounds and minimize the chances of response errors.

This study had a number of limitations. First, the questionnaire we developed was only available in English and therefore excluded physiotherapists who spoke only French. Another limitation is that the expert consultant panel consisted only of academics, who also responded to each round of the questionnaire. This could be considered an advantage because the objective of selecting participants for a Delphi study is to identify individuals who can speak knowledgeably from the position of the group to which they belong and are invested in the outcome of the decision-making process.²⁹ However, these academics may have also had the opportunity to over-exert their opinion in the questionnaire responses and in the panel discussions. We hope that having six members on the expert consultant panel and two investigators (who did not respond to the survey and acted as facilitators during the panel discussions) mitigated any one individual’s opinion.

Conclusion

We used a modified Delphi methodology to gather national input from physiotherapy academic departments, CCCEs, CIs, and students to develop a revised student evaluation of the clinical placement form that would be informed by the literature and meaningful to all stakeholders. The investigators and expert consultant panel collaborated to review the data from each round of questionnaires and develop subsequent ones as well as to refine the final version. This form was released to all Canadian physiotherapy programmes in February 2018.

This form is now being implemented in physiotherapy university programmes across Canada with plans to translate it into French to enable more students to share their experiences at clinical sites. In addition, this form can potentially be adapted for use in other health professions that include clinical education in their professional training programmes. Once the form’s measurement properties are explored, we will be able to demonstrate quality clinical placements and systematically evaluate them. In addition, future research might use this form to determine whether students are more likely to return to a clinical site for employment after graduation if the placement experience was highly rated.

Key Messages

What is already known on this topic

Stakeholders value students’ perceptions of their clinical placements. This feedback from students is used to strengthen relationships between academic institutions and clinical sites and can improve teaching behaviours and reinforce best practice in clinical education.

What this study adds

This study used an informed process to engage with Canadian physiotherapy academic departments, centre coordinators of clinical education, clinical instructors, and physiotherapy students to develop an updated, evidence-informed clinical placement evaluation form that is concise and meaningful to all stakeholders.

Appendix: Student Evaluation of the Clinical Placement (Final Form)

National Association for Clinical Education in Physiotherapy/Association Nationale d’Éducation Clinique en Physiothérapie

Clinical Facility:	__________________________________	Clinical Instructor:	__________________________________
Placement:	__________________________________	Clinical Instructor:	__________________________________
Dates:	__________________________________	CCCE:	__________________________________
Area of Practice	__________________________________	Student:	__________________________________

The purpose of this form is to: 1) Foster communication between the clinical instructor (CI) and student; 2) Provide constructive feedback to the clinical instructor; 3) Provide feedback to the facility/agency on the student’s experience; 4) Provide feedback to the Director of Clinical Education (DCE) regarding the clinical experience.

Instructions for completion:

• Students must complete this form at the mid- and final points of the placement and review the completed form with their Clinical Instructors (CI(s)) and Centre Coordinator of Clinical Education (CCCE) as applicable.

• Comments are extremely valuable and are strongly encouraged. Please append additional comments if required.

• Please check the appropriate rating box for each item according to the following scale: SA = Strongly Agree; A = Agree; N = Neutral; D = Disagree; SD = Strongly Disagree

• The form is to be returned to: Director of Clinical Education/Academic Coordinator of Clinical Education; University address

Section 1: The Site	Mid-point					Final
	SA	A	N	D	SD	SA	A	N	D	SD
1. I was adequately oriented to the site and clinical area (e.g., emergency and safety procedures, equipment, supplies, workload measurement statistics, documentation standards).						-	-	-	-	-
2. The expectations of the placement, roles, and responsibilities were discussed in the first week of the placement (e.g., learning/teaching style, preferred methods of feedback).
3. The environment was welcoming, non-threatening, collegial, and respectful.
4. Opportunities and resources were identified to augment my knowledge and learning (e.g., patient meetings, variety of conditions, in-services).
5. I was encouraged to take responsibility for my learning.
6. The amount and type of supervision I received was appropriate to my level/experience/ competence.
7. There was adequate opportunity to:*
i) conduct assessments and analyse findings.
ii) develop and progress treatment plans.
iii) deliver treatment interventions.
iv) plan for discharges.
8. I had opportunity to collaborate with others (e.g., health care team, administrative staff, insurance professionals), which facilitated my learning in this setting.
9. The CCCE was helpful and available in dealing with clinical placement issues.*
Comments at mid-point:	Comments at final point:

Section 2: The Clinical Instructor (please append an additional sheet for each CI): The CI . . .	Mid-point					Final point
	SA	A	N	D	SD	SA	A	N	D	SD
1. . . . provided support and encouragement for my learning.
2. . . . appeared open minded and non-judgmental.
3. . . . was regularly available for discussion and/or consultation.
4. . . . facilitated discussions to review, negotiate, and revise my specific learning goals and objectives regularly.
5. . . . facilitated the development of my skills (e.g., interviews, assessments, clinical skills and techniques, intervention plans, communication, collaboration).
6. . . . helped me develop my clinical reasoning skills (e.g., use of clarifications, probes, reflective questions, etc.).
7. . . . encouraged me to use evidence to support clinical practice.
8. . . . . modelled effective physiotherapy behaviours (e.g., clinical skills, clinical reasoning, professional and ethical behaviours, patient and/or family communication).
9. . . . was receptive and responsive to my feedback.
Comments at mid-point:	Comments at final point:

Section 3: Feedback and Assessment of the Student	Mid-point					Final point
	SA	A	N	D	SD	SA	A	N	D	SD
1. I was encouraged to reflect and self-assess my performance.
2. I was observed and given constructive feedback that was linked to specific examples of my performance.
3. The feedback I received was regular, timely, and objectively identified my strengths and areas for improvement.
4. Based on the feedback received, a plan was developed to provide opportunities for ongoing improvement.
5. I was encouraged to provide feedback to the CI and team.
6. The mid-point assessment of my performance was useful in identifying additional learning needs related to meeting the objectives for this clinical experience. (optional at mid-point; required at final point)*
7. The formal assessments of my performance were aligned with the informal feedback received to date. (optional at mid-point; required at final point)*
Comments at mid-point:	Comments at final point:

Overall, the placement provided the learning experience required to develop competency in this area of practice, appropriate to my clinical level.

□ Strongly Agree □ Agree □ Neutral □ Disagree □ Strongly Disagree

What were the most positive aspects of this placement?

Do you have any suggestions to improve the learning experience?

Signatures at mid-point		Signatures at final point
Student Signature:	__________________________________	Student Signature:	__________________________________
CI Signature:	__________________________________	CI Signature:	__________________________________
CI Signature:	__________________________________	CI Signature:	__________________________________
CCCE Signature:	__________________________________	CCCE Signature:	__________________________________
Date:	__________________________________	Date:	__________________________________