Physiotherapy Students’ Performance in Clinical Education: An Analysis of 1 Year of Canadian Cross-Sectional Data

Abstract

Purpose: This research examines 1 year of cross-sectional, Canada-wide ratings from clinical instructors using the Canadian Physiotherapy Assessment of Clinical Performance (ACP) and analyzes the performance profiles of physiotherapy students’ performance ratings over the course of their entry-to-practice clinical placements. Method: Canadian physiotherapy programmes that use the ACP were invited to submit anonymized, cross-sectional data for placements completed during 2018. Descriptive analyses and summary statistics were completed. Mixed-effects modelling was used to create typical performance profiles for each evaluation criterion in the ACP. Stepwise ordered logistic regression was also completed. Results: Ten programmes contributed data on 3,290 placements. Profiles were generated for each ACP evaluative item by means of mixed-effects modelling; three profiles are presented. In all cases, the predicted typical performance by the end of 24 months of study was approximately the rating corresponding to entry level. Subtle differences among profiles were identified, including the rate at which a student may be predicted to receive a rating of “entry level.” Conclusions: This analysis identified that, in 2018, the majority of Canadian physiotherapy students were successful on clinical placements and typically achieved a rating of “entry level” on ACP items at the end of 24 months.

Health care practice in Canada is constantly changing, and it is influenced by many factors, such as an aging population, implementing effective primary care reform, improving the management of chronic conditions and end-of-life care, addressing problems with patient safety, developing more integrated models of care delivery, managing the increasing costs of health care, delivering service models that promote patient centeredness, and implementing effective health technology and information systems.¹ Important qualities of health professionals who practise in this environment are as follows: autonomous, confident, self-directed, ethical, flexible, collaborative, innovative, and knowledgeable about change.^1,2

Academic programmes are responsible for preparing students to become health professionals. Although in-class components of entry-to-practice curricula are instrumental in educating learners about the knowledge, skills, and attitudes required in the field of physiotherapy, clinical education enables them to apply theory to practice, become active team members, and become socialized into the profession.^3,4

Two core documents guide the development and delivery of clinical education in physiotherapy programmes in Canada: the Clinical Education Guidelines,⁵ which identify the learning needs to achieve entry-level status, and the Accreditation Standards for Physiotherapy Education Programs in Canada established by Physiotherapy Education Accreditation Canada, which set the standards for education programmes in physiotherapy.⁶ The Clinical Education Guidelines were developed to provide “national consistency in preparing safe and effective entry-level physiotherapy practitioners.”^{5(p. 2)} They define Canadian physiotherapy clinical education as follows:

The component of entry-to-practice curriculum, in which students gain practical experience and engage in a range of professional opportunities in various clinical settings, for the purpose of learning and applying physiotherapy knowledge, skills, behaviours and clinical reasoning. Clinical Education serves to develop and refine, in a graded fashion, the practice skills, confidence, judgment, efficiency and responsibility needed by physiotherapy students for entry-level practice.^{5(p. 2)}

The guidelines recommend that a minimum of 1,025 hours of the academic programme take place in clinical placements, and 80% of these hours must be in settings that provide patient care.

In the Accreditation Standards for Physiotherapy Education Programs in Canada,⁶ the principal statements about clinical education are found in Criterion 4.3, which states,

The program has a required mix of clinical education experience that is designed to encompass essential areas of practice and settings across the lifespan, and that enables students to achieve the required competencies.

• 4.3.1Each student shall attain the required mix as defined by the program.
• 4.3.2Each student shall attain a minimum of 1025 hours of clinical education experience; the majority of these hours shall be under the supervision of a licensed/registered physiotherapist.^{6(p. 21)}

In addition, Criterion 4.4 highlights the value of student assessment across the programme, including in clinical education experiences.

There is a framework for evaluation of student achievements with clearly defined evaluative criteria and outcomes.

• 4.4.1A variety of appropriate methods and tools are used to measure student performance.
• 4.4.2The program provides appropriate and timely feedback to students about their strengths and opportunities for improvement and remediation.^6(p.21)

Thus, clinical education and assessment during clinical education are key and valued components of Canadian university physiotherapy programmes, and they play a vital role in the development of an entry-level graduate.

In physiotherapy clinical education contexts, students work full time in practice environments with multiple patients, and usually other health professionals, over an extended period of time (e.g., 6 wk). A supervising physiotherapist, the clinical instructor (CI), observes the students and gives them informal formative feedback throughout their placement. Formal assessment occurs at the mid- and final points of the placement for and by both the student and the CI. In the majority of Canadian physiotherapy programmes, the Canadian Physiotherapy Assessment of Clinical Performance (ACP) is one of the tools used by students to self-assess their performance at these time points. CIs also use the ACP to provide feedback on students’ performance.

Assessment during clinical education promotes authentic, context-specific learning by reinforcing foundational knowledge, and it influences the development of clinical skills, habits of reflection, professional behaviour, interpersonal skills, and integration of relevant and current knowledge into clinical practice.⁷ Assessing students’ performance during clinical education placements has a formative purpose because it is delivered with the intent to improve their performance on their way to becoming entry-level physiotherapists. Clinical education assessment also has a summative purpose in that it provides information to the academic programme about a student’s overall performance, from which the programme determines the student’s grade.

The ACP was developed between 2012 and 2014 and is described elsewhere;⁸ there is evidence to support its face and content validity,⁹ as well as its reliability, validity, and practicality.¹⁰ Since 2015, physiotherapy programmes in Canada have implemented the ACP to assess students in clinical education, and to date 13 of the 15 programmes use the ACP.

In 2017, a national, online, web-based ACP was developed. The online format allows the data to be electronic and facilitates the analysis of students’ performance in clinical education at the individual programme level as well as cross-programme comparisons. Analysis of these data is valuable because they have the potential to inform curriculum development, determine the range of students’ performance along the continuum of their development, and identify students who are performing exceptionally well or who would benefit from remediation. To contribute to identifying what is deemed exceptional performance or when a student is struggling to demonstrate competencies, it is important to understand the typical ratings on the ACP at different time points in entry-to-practice education, the range of those ratings, and their performance profiles.

The purpose of our research was to study 1 year of cross-sectional, Canada-wide data from the ACP to analyze the performance profiles of the ratings of physiotherapy students’ performance over the course of their entry-to-practice clinical placements.

Methods

This project was exempt from formal submission to the Hamilton Integrated Research Ethics Board because it was identified as quality improvement work (February 2019).

Compiling the data

In March 2019, we sent an email invitation to the directors of clinical education (DCEs) or academic coordinators of clinical education (ACCEs) of each physiotherapy programme in Canada (N = 13) that uses the ACP to assess students in clinical education. This email invited the DCE or ACCE to contribute the ACP data from their university physiotherapy programme from 1 calendar year (2018) so that we could compile a national data set that would be used to investigate the profiles of the ratings of student performance on each ACP criterion.

Each programme retains control of its own students’ ACP data, so each programme was given detailed instructions on how to prepare its data in an anonymous way by de-identifying the data and assigning each ACP a unique identifier (following a specified format) so that the data could be compiled into a national data set. Each programme was also required to prepare the data for compilation by removing text comments and mid-term ratings, retaining only final numerical item ratings, and adding the following columns: programme-assigned grade, placement number, number of months that a student had completed since starting the physiotherapy programme, and placement attempt number.

After each programme had anonymized and formatted its data, one investigator (BM) compiled the programmes’ data into one data set. This investigator reviewed the data set for missing or atypical data and followed up with the programmes as required to verify the accuracy of the submission. Once the compiled data set was completed, it was sent to a consulting statistician, who was not given any information about how the unique identifiers had been created. The statistician then re-coded all the unique identifiers so that the results would be anonymized for all the investigators once the data analysis was complete.

Data analysis

Descriptive analyses of the compiled data set included cross-tabulations that reported on frequencies associated with the placement number by university, placement attempt, assigned grade, and CI-recommended grade. We calculated summary statistics for each ACP item by placement number and determined correlations among the item scores.

Mixed-effects modelling assumed that students had been in their programmes for 4 months and was used to create typical performance profiles (i.e., graphs) for each criterion in the ACP. Stepwise ordered logistic regression was also completed, in which the dependent variable was the CI recommendation and the independent variables were the 21 items on the ACP. Correlations between CI recommendation and time in programme by placement were also investigated. All analyses were completed using STATA, Version 15.1 (StataCorp LLC, College Station, TX).

Results

Descriptive data

Ten physiotherapy programmes responded and contributed anonymized data for analysis; the number of placements varied. In total, the programmes contributed to ACP data on 3,290 placements, and the majority of the data came from Placements 1–5. Table 1 presents the ACP placement data by placement number, attempt number, and timing of the placement in months after the start of the programme. We selected the number of months as a common element that all programmes could report on despite their unique placement structures. These unique placement structures across programmes included wide variations in duration, timing within programme sequence, and quantity of placements and add up to at least 1,025 hours per student.

Table 1 .

Number of Ratings by Placement Number, Attempt Number, and Timing of Placement (N = 3,290)

Placement no., attempt no., or time since placement	No. of ACPs (%)
Placement no.
1	535 (16.3)
2	663 (20.2)
3	644 (19.6)
4	643 (19.5)
5	567 (17.2)
6, 7	238 (7.2)
Attempt no.
1	3,274 (99.5)
2	13 (0.4)
Missing	3 (0.1)
Time since starting physiotherapy degree, mo
4	70 (2.1)
7	155 (4.7)
8	63 (1.9)
9	211 (6.4)
10	362 (11.0)
11	127 (3.9)
12	2 (0.1)
13	98 (3.0)
14	279 (8.5)
15	57 (1.7)
16	146 (4.4)
17	230 (7.0)
18	202 (6.1)
19	90 (2.7)
20	339 (10.3)
21	185 (5.6)
22	46 (1.4)
23	215 (6.5)
24	62 (1.9)
25	164 (5.0)
26	79 (2.4)
≥ 27	108 (3.3)

ACP = Assessment of Clinical Performance.

At the final evaluation of the placements, the majority of CIs had recommended grades of “credit” (70.9%) or “credit with exceptional performance” (28.1%; i.e., passing grades). Only 1.0% of the students received a recommendation for “credit with reservation” or “no credit” (see Table 2). Table 3 provides an overview of the programme-assigned grades at the end of the placements. Figure 1 shows that time in a programme was not a significant predictor of a CI’s recommended grade at the end of a placement.

Table 2 .

Number of Placements and Associated Clinical Instructor Ratings Included in the Compiled Data Set

Placement no.	No. of ratings, end of placement
	No credit	Credit with reservation	Credit	Credit with exceptional performance	Total
1	0	1	307	159	467
2	1	6	429	160	596
3	2	4	399	139	544
4	1	5	394	137	537
5	4	3	362	160	529
> 6	2	0	146	54	202
Total placements, no. (%)	10 (0.35)	19 (0.66)	2,037 (70.85)	809 (28.14)	2,875

Note: Not all programmes require a clinical instructor to recommend a grade at the end of the placement; as a result, we had recommendations for 2,875 of the overall 3,290 placements.

Table 3 .

Programme-Assigned Grades at the End of Placements

Grade	No.
Fail	20
Pass	3,268
Not yet determined	2

Figure 1 .

Time in programme compared with CIs’ final grade recommendation at end of placement.

CI = clinical instructor; 1 = no credit; 2 = credit with reservation; 3 = credit; 4 = credit with exceptional performance.

Typical performance profiles

Typical performance profiles were generated for each of the 21 evaluative items of the ACP using mixed-effects modelling. This modelling estimated students’ typical performance on the basis of the length of time they had been enrolled in a physiotherapy programme, using the compiled data from the 10 programmes. Because the majority (97%) of placements had occurred within 27 months of the start of the programme and 2.7% of the ACP placement data occurred in Months 28–35, the ACP data beyond 27 months were deemed to be not representative and were excluded from this analysis.

Inspecting the item trajectories for each of the 21 profiles revealed only subtle differences among the ACP items. Three profiles are presented in this article to highlight these differences: 1.4 (representing expert analysis), 2.3 (representing communication), and 7.2 (representing professionalism). They include a range of baseline values (i.e., at 4 mo) and trajectory profiles (i.e., high, middle, and low) to achieve the entry-level ratings. The profiles for the other 18 items are available on request.

Profile 1.4: Expert Analysis – establishes a physiotherapy diagnosis and prognosis

The profile for this ACP item is presented in Figure 2. For all items, the greatest variability among the students occurs earlier in the programme, and their scores converge close to the 24-month time point.

Figure 2 .

Typical performance profile for Assessment of Clinical Performance Expert Analysis Item 1.4.

Solid line = typical performance pattern; dotted lines = 95% range of student-specific trajectories.

This profile demonstrates that students who have been in a Canadian physiotherapy programme for 24 months are typically assigned a rating of 9 (entry level) by their CI at the end of their placement. The range over which 95% of the students’ performance is likely to lie after completing 24 months is 8–9.5. However, because the ACP rating scale does not allow for half-points, CIs are required to assign a rating of 9 (entry level) or 10 (with distinction); a rating of 9.5 is not possible.

Profile 2.3: Communication – employs effective and appropriate verbal, non-verbal, written, and electronic communications

This profile is presented in Figure 3. It demonstrates that students who have been in a Canadian physiotherapy programme for 24 months are typically given a rating of 9 (entry level) by their CI at the end of their placement. The range over which 95% of the students’ performance is likely to lie is 8.75–9.5, suggesting that there may be less variability in the predicted final score for this item at the end of 24 months compared with Item 1.4 (see Figure 2). The divergence of the 95% range lines after 24 months in the programme reflects the fact that fewer data points are available for placements that occur beyond 24 months.

Figure 3 .

Typical performance profile for Assessment of Clinical Performance Communication Item 2.3.

Solid line = typical performance pattern; dotted lines = 95% range of student-specific trajectories.

Profile 7.2: Professionalism – respects the individuality and autonomy of the client

The profile for this ACP item is presented in Figure 4. It illustrates that students who have been in a Canadian physiotherapy programme for 24 months are typically rated at 9 (entry level) by their CI at the end of their placement. The range of the 95% predicted scores is closer together than in Figures 2 and 3, with little variance from the score of 9 just before 24 months. This is represented by the dotted lines on the figure that, although appearing to cross, actually deflect from the solid line after this point. This smaller range in the 95% predicted scores at 24 months suggests minimal variability in the predicted final score at 24 months, and the divergence of the dotted lines after 24 months reflects a small number of data points after this time.

Figure 4 .

Typical performance profile for Assessment of Clinical Performance Professionalism Item 7.2.

Solid line = typical performance pattern; dotted lines = 95% range of student-specific trajectories.

Comparing the profiles

The trajectories of the three profiles are presented in Figure 5, enabling us to note the similarities but also the subtle differences among them. In all cases, the predicted typical performance rating by the end of 24 months is approximately 9 (entry level); however, the rate at which it is predicted that a student will receive a rating of entry level is slightly different. For example, the trajectory for Profile 7.2 (top line) shows a predicted score of 9 earlier in the students’ time in the programme, whereas the trajectory for Profile 1.4 (bottom line) has a more consistent slope (i.e., it is predicted to take longer for a student to achieve a rating of entry level, but there are higher incremental changes between months).

Figure 5 .

Typical performance profiles for the three Assessment of Clinical Performance items.

Discussion

We analysed 1 year of cross-sectional, national data from the ACP to discover the trajectory of physiotherapy students’ ratings over the course of their entry-to-practice clinical placements. This was the first national analysis of clinical placement data for Canadian physiotherapy students. We learned that compiling blinded data from different university programmes to develop a national data set was feasible. Our study demonstrated that students trained in Canadian entry-to-practice physiotherapy programmes were generally successful in completing their clinical placements at all levels and meeting entry-level competency after being in the programme for 24 months. We also demonstrated that typical performance profiles could be generated using cross-sectional clinical education data.

Although national curriculum and clinical education guidelines exist in Canada,^5,11 programmes also have the autonomy to be innovative and diverse in how they deliver the curriculum.⁶ This autonomy may account for the wide range in students’ scores that we observed between 5 and 20 months. Specifically, students may have been learning different skills in the clinical setting in their own placements and across different programmes, based on when specific skills and knowledge were introduced in the curricula of the respective programmes.

This analysis of the cross-sectional data from 3,290 Canadian physiotherapy student placements demonstrates that, despite variations in how university programmes deliver their curricula, students’ clinical placement evaluations improved over time and generally attained entry-level performance in clinical education by the end of 24 months of study. The analysis also demonstrated that there was markedly less variation in the ratings awarded by CIs the closer students were to completing their programme (typically 24 months).

However, this analysis also identified that students achieved a rating of “entry level” faster on some items than on others. Specifically, the performance of physiotherapy students undertaking clinical placements in 2018 with regard to professional behaviours was rated at entry level sooner than their performance on expert analysis. Expert refers to a student’s ability to “integrate all of the Physiotherapist Roles to lead in the promotion, improvement, and maintenance of the mobility, health, and well-being of Canadians.”^{12(p. 6)} The longer duration required before achieving entry-level competence may reflect students’ need to experience a sufficient volume of exposure to situations that require them to apply theoretical knowledge to become proficient or be deemed entry level.

In addition, the importance of experience as health care providers move from being a beginner to being an expert – how well they apply their clinical skills – has previously been identified.^13,14 For example, Gilliland analysed the developmental processes of six American students enrolled in a physical therapy programme using a simulated patient case at three points during their entry-level education (first year, second year, and post-clinical).¹³ During the first year, students generated hypotheses that focused on anatomical structures to guide their assessments and reasoning, but during their second year and after their clinical experience, they generated hypotheses on the basis of diagnoses and biomechanical contributing factors.¹³

Furze and colleagues also explored the longitudinal development of student clinical reasoning capabilities across the curriculum at one doctorate of physical therapy programme in the American Midwest.¹⁴ Written student responses to a clinical reasoning reflection questionnaire were coupled with narrative responses from the Clinical Performance Instrument,¹⁵ a clinical education evaluation tool. The authors found that clinical reasoning developed gradually, moving from being scripted, procedural, and self-focused to becoming more dynamic and patient focused.¹⁴

Despite integrating simulated patients and case studies into academic courses that challenge physiotherapy students to apply their recently acquired knowledge, the opportunity to learn how to anticipate and develop confidence in applying that knowledge to a clinical scenario is likely best refined in clinical settings. However, the settings that students find themselves in and the populations with whom they interact tend to be different in each placement. As a result, repeated exposure to similar clinical scenarios in which students can apply their knowledge and gain independence seems to occur toward the end of the programme after completing all, or the majority of, their placements.

However, professionalism is an area that can be both integrated into academic courses and developed across each clinical placement. In a survey of Canadian physiotherapy programmes, on average more than three distinct teaching methods, including clinical education, were used to deliver professionalism topics in the curriculum.¹⁶ Thus, repeated and ongoing exposure to the concept of professional behaviours, across placements and academic courses, may contribute to the increased rate at which the students whose ratings were analyzed for this study achieved entry-to-practice ratings on this item of the ACP.

This national data analysis also revealed that 99.3% of the students received a recommendation of a grade of “credit” or “credit with exceptional performance” from their CIs, indicating that the CIs were confident that the students should be awarded a passing grade. In a study that explored the perceived benefits of and barriers to supervising students, the fear of having a student who might be struggling or in danger of failing is a perceived barrier to being willing to supervise students.¹⁷ Although this perceived barrier was “strongly voiced” in the study by Davies et al., in our study, it is reassuring to learn that fewer than 1% of the 3,290 clinical placement student assessments received a grade of “fail.” Perhaps this result of our study can lower the perceived barrier.

Alternatively, this low incidence of a recommended failing grade on the ACP may also be an example of the “failure to fail” phenomenon, whereby clinicians have difficulty reporting students’ unsatisfactory academic or professional performance.¹⁸ The factors that contribute to the difficult decision to recommend a failing grade range from CIs’ professional and personal considerations to student considerations, assessment tools available, institutional culture, and potential opportunities for remediation for students.¹⁸ The facilitators of recommending a failing grade include a sense of responsibility for patients, society, and the profession; support from the institution; and students’ opportunities after failing.¹⁸

An intention with the national implementation of the ACP was to support CIs’ ability to grade students in clinical education using a standardized student assessment measure. In addition, the online training module, which all students and CIs are required to complete before using the ACP for the first time, is a resource meant to lower the barriers to and increase the facilitators of CIs’ accurate recommendations regarding their students’ performance.

The ability to aggregate national data will also help facilitate these decisions to pass or fail students in clinical education. Long-term analysis of the evaluation of student performance in clinical education may also identify opportunities to inform national clinical education guidelines to address students’ skills and knowledge associated with a specific competency. For example, with increased data points, it may be possible to identify a competency that tends to be rated more poorly by CIs across Canadian programmes. This could lead to developing enhanced training resources for CIs on using the ACP, additional training or education on these specific competencies for students before placement to increase their potential for success on placements, and enhanced descriptors on the ACP to assist with evaluating placements at the varying stages of students’ education.

This study used typical performance profiling to effectively analyse the clinical education data. It is important to note, however, that in the absence of longitudinal data (i.e., full programme data for any individual students), the resulting models are performance profiles rather than growth trajectories. In the future, growth curve modelling may identify predictors of growth, which could in turn identify patterns or trends in student performance and allow programmes and CIs to better understand why a small proportion of students are not successful on their clinical placements. In addition, analyzing the CIs’ recommended mid-term marks and subjective comments along with the final evaluation data may provide further insight into why students did or did not achieve success in each placement. Future studies should identify how longitudinal analysis of ACP data can help inform curriculum standards or expectations related to the education of physiotherapy students in Canada.

This study had limitations that we must acknowledge. First, the blinding of the data occurred at the level of each DCE or ACCE, but despite the fact that we provided instructions on how to code the placement data, it is possible that the instructions were applied incorrectly, which could have contributed incorrect data to a placement period. To address this concern, when the data from each programme were being added to the master file, any atypical patterns were identified, and the data were sent back to the originating programme for quality checking. Second, although our analysis integrated end-of-placement data from 10 of the 13 programmes that use the ACP, some programmes do not use it, and others were not able to release their data to be compiled. Thus, the results need to be generalized with caution, understanding that the data reflect 1 year (2018) of placements across the majority of Canadian physiotherapy programmes.

Conclusion

This project looked at 1 year of cross-sectional, national data from the ACP and analysed the trajectory of physiotherapy students’ ratings over the course of their entry-to-practice clinical placements. Our results identified that, in 2018, the majority of students were successful on their clinical placements, and they typically achieved an entry-level rating on the ACP items at the end of 24 months. The results of this analysis also demonstrate the feasibility of generating a national data set of clinical placement data, and they provide a foundation from which future longitudinal and cross-sectional analyses of placement data can be developed.

Key Messages

What is already known on this topic

The Canadian Physiotherapy Assessment of Clinical Performance (ACP) is a valid and reliable tool that has clinical applicability. Most Canadian entry-to-practice physiotherapy programmes use it as an evaluation tool for student performance during clinical placements.

What this study adds

Typically, students enrolled in a Canadian entry-to-practice physiotherapy programme are successful on their clinical placements: by 24 months they appear to achieve a rating of “entry level” on all ACP items.