Self-Assessment: The Disconnect between Research and Rhetoric

Abstract

Purpose:

Physical therapists are expected to engage in self-assessment in order to ensure competent practice and to identify appropriate professional development activities.

Summary of Key Points:

This paper reviews the current literature on the accuracy and role of self-assessment in physical therapy. Current literature indicating that self-assessment cannot be conducted with any degree of accuracy is discussed, and a proposed reformulation of the concept of self-assessment is presented.

Recommendations:

Practical strategies are offered for clinicians to improve the potential for obtaining reliable and valid information about their own clinical performance to guide the selection of appropriate professional development activities and to promote the provision of competent patient care.

INTRODUCTION

The term “self-assessment” seems omnipresent in the lexicon of health professionals, including physical therapists (PTs). The concept is considered central to the development of professional competence and effective lifelong learning¹ and is an integral component of the self-directed learning process inherent in problem-based learning (PBL) curricula.² Self-assessment has been defined as “one part of the self assessing another part of the self's actions and outcomes.”^{1 (p. 176)} Whether we are expecting student PTs to recognize the importance of asking for help when circumstances indicate or expecting the graduate (or practising) PT to select the appropriate continuing education (CE) programme to maintain his or her competence, the expectation of self-assessment is there. This assumption, with its implicit expectation of accuracy, is rarely described but, even more importantly, rarely questioned. But what is self-assessment? And does it really work?

This paper, written with a physical therapy audience in mind, addresses the accuracy of self-assessment. A brief introduction about the role of self-assessment in self-directed learning, PBL, and autonomous practice is presented. Studies on self-assessment in physical therapy are reviewed, and other studies presented, to illustrate the current state of research in self-assessment. A reformulation of the construct of self-assessment is described, and specific strategies that can be used by clinicians to obtain objective information about their clinical competence are discussed.

BACKGROUND

In the 1970s, Knowles introduced the concept of self-directed learning.^3,4 The person engaging in self-directed learning must be able to identify the required competencies and judge his or her own level within those competencies in order to develop an appropriate learning plan.^3,4 Knowles identified self-assessment as an essential skill of self-directed learning, describing it as “the ability to diagnose one's learning needs in the light of models of competencies required for performing life roles.”^{4 (p. 184)} Self-directed learning is one of the most common ways that adults pursue learning throughout their lives,⁵ although the application of self-assessment in health professional practice and education has been questioned by some.^6,7

Self-directed learning is an integral part of the PBL curricula used by health professional schools around the world.^2,8 The goal of PBL is to allow learners to find knowledge for themselves, to contrast that knowledge with the understandings of others, and to refine that knowledge as they gain more relevant experience.⁹ PBL is reported to give students the self-study skills they will need for the rest of their professional lives.²

Self-assessment is an expectation of autonomous professionals and an integral part of professionalism.^10–12 For example, the College of Physiotherapists of Ontario expects registrants to conduct a self-assessment to identify both their strengths and their weaknesses so as to maintain the required competencies of the profession.¹⁰ Furthermore, the “Essential Competency Profile for Physiotherapists in Canada”¹³ identifies that each PT is expected to “effectively self-assess … to identify learning needs”^{13 (p. 13)} as part of his or her professional development and lifelong learning activities. The Royal College of Physicians and Surgeons of Canada has included self-assessment in the professional role description, noting that physicians are expected to “continually evaluate [their] abilities, knowledge and skills and know [their] limitations of professional competence.”¹⁴

All this rhetoric about self-assessment gives rise to a question: Is there evidence that PTs, or other health professionals, are able to conduct self-assessment accurately? A review of the literature on self-assessment in physical therapy follows.

STUDIES OF SELF-ASSESSMENT IN PHYSICAL THERAPY

Despite the fact that the term “self-assessment” appears regularly in the academic, regulatory, and professional literature, there is little information about self-assessment in physical therapy. And, while the term can be used to describe a range of activities, for the purposes of this review, “self-assessment” will refer to the activity of self-rating or judging one's performance.

A literature search for studies in English that describe research to determine the accuracy of self-assessment or the self-assessment process was undertaken using the Ovid CINAHL and Ovid MEDLINE databases. The databases were accessed online in early October 2006 for the time periods 1982 to October 2006 and 1966 to September 2006, respectively. Using the keywords “self assessment or self-assessment” combined with “physiotherapy or physical therapy,” 2 relevant studies^15,16 were identified among 236 results in the CINAHL database. The search in the MEDLINE database identified 66 articles, 3 of which were research articles;^15,17,18 of these, 2 had not been identified in CINAHL.^17,18 Re-executing the search in the Ovid CINAHL and Ovid MEDLINE databases using the keyword “self evaluation” yielded 35 and 22 articles respectively; no new research studies were identified. Finally, in the CINAHL database, the following journals were searched using the keywords “self-assessment or self assessment”: Physical Therapy, Physiotherapy Canada, the Journal of Physical Therapy Education, the Australian Journal of Physiotherapy, and Physiotherapy. These searches yielded 25, 3, 12, 10, and 2 articles respectively; no new research studies were retrieved.

This literature search identified only four studies that addressed the accuracy or the process of self-assessment in the physical therapy profession^15–18 (Table 1). Two studies examined the accuracy of self-assessment among PT students,^16,18 and two addressed the self-assessment process among both student and graduate PTs.^15,17 A brief overview of these studies follows.

Table 1.

Summary of Research on Self-Assessment in Physical Therapy

Author	Type of Study	Subjects	Focus	Context	Comparators	Hypothesis	Results
Palmer et al. (1985)18	Quantitative methods: quasi-experimental design	32 student PTs	Accuracy of self-assessment	Manual muscle testing and goniometry	Faculty rating and student rating	Viewing a videotape would improve the accuracy of self-assessment	There was no significant difference between those who did and those who did not view a videotape of themselves
Miller (1999)16	Quantitative methods: longitudinal design	54 student PTs	Accuracy of self-assessment	Student learning in tutorials	Average of peer ratings and student rating	Accuracy of self-assessment would improve over time	Poor to moderate correlations were found between self-assessment and peer rating; accuracy did not improve with time.
Orest (1995)17	Qualitative inquiry using interviews	4 PTs	Perceptions of self-assessment in practice	N/A	N/A	N/A	Three themes were identified: competence, patient outcomes, and professional development.
Musolino (2006)15	Qualitative inquiry using interviews in a phenomenological approach	7 student PTs and 4 PTs	The experience of self-assessment and reflective practice	N/A	N/A	N/A	A conceptual model and tables related to barriers and support were developed.

Orest undertook a qualitative study designed to explore PTs’ perceptions of self-assessment in their clinical practice.¹⁷ Four clinicians were purposively selected in an attempt to attain diversity on the dimensions of gender, age, and experience. During in-depth interviews, the PTs were asked to discuss the importance of self-assessment, barriers to and facilitators of self-assessment, and the role of formal training in self-assessment. Three themes were identified: competence (the relationship between self-assessment and remaining competent), patient outcome (how wanting to optimize patient outcomes is a stimulus for self-assessment), and professional development (how self-assessment was linked to professional development goals). The participants viewed self-assessment as a continuous process that benefits patients as well as themselves.¹⁷

Building on the work of Orest,¹⁷ Musolino employed qualitative methods to explore the experience of self-assessment among a sample of PT students (n = 7) and recent PT graduates (n = 4).¹⁵ Another study objective was to identify training needs in order to describe a plan by which PTs could incorporate self-assessment into lifelong learning. All participants identified the importance and value of self-assessment in relation to professional competence. Motivations for self-assessment included the desire for self-improvement, career development, and professional competence. Musolino presented a list of training needs for self-assessment addressing the affective, psychomotor, and cognitive domains for student and graduate PTs and a conceptual model of reflective self-assessment.¹⁵

While the Musolino study¹⁵ provided validation for Orest's findings,¹⁷ a major limitation detracts from its contribution to the understanding of self-assessment. Musolino reported that self-assessment and reflective practice are “intertwined,” and it is apparent that both activities were addressed simultaneously in the study. Self-assessment (the provision of a judgment about one's abilities) and reflection (“a metacognitive function that deals critically with a previous activity or thought process”^{19 (p. 295)}) are very different concepts. As a result, the findings of Musolino's study¹⁵ make a limited contribution to the present discussion, because it is impossible to separate the issues of self-assessment and reflection in the methods and findings.

Palmer et al. examined the effect of videotape replay on the quality and accuracy of student evaluations,¹⁸ based on a descriptive report by Saarinen.²⁰ In their study, 32 PT students were videotaped while conducting a patient assessment involving manual muscle testing and goniometry. The authors hypothesized that the students in the experimental group, who viewed a videotape of their assessment before scoring their performance, would produce more accurate self-assessments than the control group, who rated themselves without seeing a videotape of their performance. Items in the assessment dealt with communication (e.g., “introduces self and explains the procedure”) and use of the goniometer. The correlations between student and instructor ratings were 0.51 and 0.56 for the experimental and control groups respectively; the difference was not statistically significant. The authors note the importance of formal instruction in self-assessment for students and suggest that further research could address more feasible strategies to enhance the accuracy of self-assessment.

Within a PBL curriculum, Miller examined the relationship between self-assessment and peer assessment among 54 students in an entry-level physical therapy course dealing with kinesiology problems.¹⁶ The peer rating was an average of the scores of the four peers who had been randomly assigned to each five-member tutorial group. The assessment related to the student's learning process during the case problem. Intra-class correlation coefficients ranged from 0.42 to 0.74, indicating moderate to strong agreement between the self- and peer assessments for the case problems. There was no evidence that the level of agreement improved as the course progressed. The authors questioned the validity of using peer and self-assessment to evaluate students’ performance in this small-group learning setting.¹⁶

The literature search conducted for the present study indicates that there is little evidence on self-assessment in physical therapy. The qualitative studies confirm that self-assessment is used by PTs to gauge professional competence and to guide the selection of appropriate professional development activities. There is no research investigating the accuracy of self-assessment among graduate PTs, and only two studies involving student PTs. This lack of research exemplifies the disconnect in our profession between the rhetoric and the evidence on self-assessment. While our educational and professional literature is replete with expectations to engage in self-assessment, we have no evidence to indicate that we are able to do so successfully.

Although the results of the two quantitative studies of student PTs yielded moderate to strong correlations, we need to collect more evidence on self-assessment among graduate PTs, or to draw on the larger body of knowledge about self-assessment outside of physical therapy. In order to place the research on accuracy of self-assessment in physical therapy within the context of current research in the medical education literature, a brief overview is presented below, using selected studies.

CONSIDERING THE EVIDENCE OUTSIDE OF PHYSICAL THERAPY

Although there is much rhetoric about the importance of self-assessment, the broader evidence base suggests that most health professionals’ ability to conduct an accurate self-assessment is poor. A literature review by Gordon²¹ reported studies of health professionals in which the correlations between self-assessment of factual knowledge and tested knowledge ranged from 0.02 to 0.65, and the correlations of studies of global self-assessment ability to clinical supervisors’ ratings was < 0.32. A recent systematic review of the medical literature corroborated these findings in studies of physicians.²² Furthermore, these findings are not exclusive to educational settings in the health professions. The mean correlation between student and teacher marks for 57 studies in educational fields of science (including medicine), social science, and the arts is reported to be 0.39.²³ While the two studies involving student PTs yielded results at the higher end of the range reported by Gordon²¹ (ICC = 0.42–0.74,¹⁶ r = 0.51¹⁸), there is little reason to believe, based on the larger professional and non-professional literature, that student or graduate PTs would be more able than other professionals to self-assess their abilities accurately. Perhaps, as noted by Eva et al., the ability to self-assess is “far more complex than we thought.” ^{24 (p. 223)}

In response to these findings, Ward et al. suggested that “problems inherent in the traditional approaches for measuring self-assessment call into question this verdict on self-assessment,”^{25 (p. 76)} rather than the construct of self-assessment itself. In many self-assessment studies, participants’ scores were compared to peer or faculty ratings, a procedure that has inherent methodological problems. Ward et al. identified concerns with the “gold standard,” with the differential use of scales among students, and with group-level analyses; they advocate for an intra-individual approach (in contrast to the inter-individual approach that produces group-level estimates of accuracy) in the measurement of self-assessment to address these problems.²⁵

In the intra-individual approach, both the student and the expert rate a set of the student's skills relative to one another.²⁵ For example, the participant is expected to rank his best to poorest performance area from a list of 10 performance domains; these rankings are then compared to the faculty member's rankings. Even when this relative ranking process is adopted, the results still indicate that measures of self-assessment often correlate poorly with other measures.^26–28 While this approach did not prove to reveal robust accuracy of self-assessment, several authors have reported that the relative ranking process is an excellent way to provide feedback to an individual, because both strengths and weaknesses are reviewed, and the skills are discussed relative to one another.^26–28

Eva et al.²⁴ conducted a study to address the concerns cited by Ward et al.²⁵ They hypothesized that medical students who completed a self-assessment after writing a multiple-choice examination that was used repeatedly throughout the medical training programme (postdiction group) would be more accurate in predicting how many questions they answered correctly than those students who conducted their self-assessment prior to writing the examination (prediction group).²⁴ Students from three different class years were randomly allocated to the prediction and postdiction groups. Correlations were low to moderate across the three years, ranging from −0.12 to 0.51 in the prediction groups and from 0.35 to 0.62 in the postdiction groups. Furthermore, this cross-sectional study in a PBL environment indicated that the students’ self-assessment accuracy declined with increasing seniority. Perhaps most disconcerting was the fact that these individuals had had considerable feedback on their performance in the various content areas, and, had they simply reported the scores they received on their last examination, their predictions would have correlated with their performance in the 0.69–0.76 range for all three years.²⁵

Another common misconception is that self-assessment skills improve with practice. There is no evidence to indicate that students, despite receiving regular feedback in the form of formative and summative evaluations, become better at self-assessment over the course of an educational programme. Longitudinal studies have indicated that medical students’ self-assessments do not improve over time, whether they are in a PBL curriculum^24,29 or a conventional programme.^30,31 While one study with occupational therapy students in a PBL curriculum noted improved correlations between students’ ratings and their tutor's ratings over a 14-week unit, the authors attributed this improvement not to greater accuracy of self-assessment but to improvements in the “dance of negotiation,”^{32 (p. 73)} in which increasing experience with one another's expectations led to an improvement in agreement between the tutor's ratings and those of the students.

DISCUSSION

Despite various methodological approaches, conclusions regarding the accuracy of self-assessment do not appear to have changed. How can we, in a culture that values evidence-based practice,^33,34 continue to embrace the belief that accurate self-assessments are being conducted by ourselves and our colleagues, when the evidence strongly suggests otherwise? And what are the implications of these research findings for our responsibilities as a self-regulating and autonomous profession?

Of greatest concern are those clinicians who may be “unskilled and unaware” of their inabilities, failing to recognize their own lack of competence.^35,36 Studies undertaken with undergraduate and medical students, medical residents, and laboratory personnel suggest that those who most lack skill possess the least insight into their poor performance.^35–40 Researchers report that, when observing the behaviour of their peers, those with the least developed skills failed to gain insight into their own performance, suggesting that one requires competence in a particular domain to recognize a lack of competence in oneself.^35,39 Indeed, those who displayed the best skills in these studies were also the most accurate in their self-assessments.^35,36 PTs who are “unskilled and unaware” raise concerns for colleagues and their respective regulatory boards because of their potential to harm patients.

If our self-assessments are inaccurate, then our professional development activities may be misguided. Research findings raise concerns that inaccurate self-assessments are not congruent with the principles of self-directed learning on which professional CE is meant to be based.^39,41 In a sample of general medical practitioners in New Zealand, who were asked to identify their strengths and weaknesses in knowledge across conditions they were likely to encounter, correlations between self-assessments and test scores were poor (0.19–0.21). The authors concluded that clinicians whose self-assessments are inaccurate may be unaware of how their knowledge falls short, and thus their learning activities may be misdirected.⁴¹

There is also concern that the domain in which the health professional has not maintained competence may be one in which his or her internal motivation to learn will not outweigh the inertial drive to continue with his or her current practice.⁶ While PTs appear to be “intrinsically motivated” to pursue formal CE activities, and report they feel they improve as a result of participating in CE,⁴² there is no evidence to indicate that PTs necessarily attend CE in the areas in which they lack competence. Among physicians, for example, just the opposite has been shown to be true. In a study involving Ontario physicians, general practitioners were asked to rank a set of educational packages for various medical conditions as being of “high” or “low” preference. The authors found significant improvements in the quality of care offered to patients in the group of physicians who received the low-preference packages, but no significant improvements in the care of patients among physicians who received the high-preference packages.⁴³ This suggests that clinicians may be more likely to take courses in areas where they are already performing at a high level. Canadian PTs have been found to prefer CE in their area of interest, which adds to the concern.⁴⁴ The important relationships among competence, continuing professional education, and patient-care outcomes warrant further study.

If we cannot rely on self-assessment, then what strategies can PTs use to monitor and maintain their professional competence?

RECOMMENDATIONS

Both Orest¹⁷ and Musilino¹⁸ identified that PTs engage in self-assessment in order to gauge their clinical competence and direct their CE choices. Duffy and Holmboe suggested that “self-evaluation in the absence of credible data is unlikely to be of much value.”^{45 (p. 1138)} If our self-assessment skills are likely to be inaccurate, how then can we gain more accurate information about ourselves and our learning needs? There are several different ways in which PTs can gain reliable and valid information about themselves to supplement their self-assessment activities, including the use of colleagues and objective testing.

Clinicians need to seek feedback from external sources, such as peers and experts.⁴³ Peer assessment has been shown to be more reliable and accurate than self-assessment.⁴⁶ While students find themselves part of a culture in which receiving feedback from others (including peers or clinical supervisors or faculty) is commonplace, this is not the case for clinicians in the majority of workplaces. Evans et al. stressed the importance of a supportive, “no-blame” culture to facilitate feedback among work colleagues.⁴⁷ Might it be possible for us to change our professional culture to one that expects, welcomes, and values peer feedback in the workplace?

There are some instances in which peer assessment is already being used. Performance reviews are undertaken in many work settings. The use of multi-source feedback, whereby peers and others in the work setting evaluate behaviours such as professionalism and patient care, is one example.^22,48 While it may be beneficial to gain information from several colleagues representing different professions, persons from outside our profession will not have a full understanding of the scope and standards of our practice. Therefore, it is imperative that at least one of the team members providing feedback be another PT. Peer-review programmes like the Onsite Assessment implemented by the College of Physiotherapists of Ontario,⁴⁹ in which a trained peer assessor offers formative and summative feedback regarding practice, might provide valuable feedback about clinical practice and competence. A PBL curriculum, with its integral peer- and self-evaluation process to provide constructive feedback, might provide an excellent forum to develop skills for giving and receiving feedback.⁵⁰ It is important that clinicians identify and foster opportunities to obtain feedback from peers.

Another way for PTs to gain accurate information about their competence would be to use objective testing to identify their strengths and weaknesses⁵¹ (i.e., by promoting learning through feedback).⁵² Clinicians need easy access to self-administered tests or other forms of formal testing, and these could be presented in a range of venues.

There are several examples of objective testing used by other professions. For example, a test with questions from the certifying examination in rheumatology has been offered to delegates of the Swiss Society of Rheumatology, as well as to the Swiss Society of Physical Medicine and Rehabilitation, at their annual scientific meetings. The aim of this test is for physicians to receive accurate feedback regarding their competencies to help guide their selection of professional development activities.⁵³ The American College of Physicians offers a Medical Knowledge Self-Assessment Program that includes learning resources and a multiple-choice questionnaire (with answers) to enable the user to test his or her knowledge and judgment.⁵⁴ In the medical laboratory field, tests of clinical knowledge and problem-solving scenarios have been used to provide feedback to staff on their areas of weakness, so that additional training can be undertaken.⁴⁰ Furthermore, self-administered, Web-based tests with self-assessment questions and feedback have been found to enhance learning for medical residents in a Web-based course.⁵⁵

The availability of self-administered examinations for PTs is limited. However, various strategies have been used in the past. For example, a programme composed of a written and practical examination to assess competencies required for ultrasound treatment was made available to PTs in their workplaces and at professional meetings.⁵⁶ Furthermore, multiple-choice quizzes have accompanied published articles in Physical Therapy to enable readers to test their knowledge of the subject and objectively identify gaps in knowledge that could direct further study.^57–59 Although substantial resources to develop, maintain, and administer examinations are required, serious consideration should be given to ways in which members of our profession can be given easy access to objective feedback on their clinical competence, using readily accessible tests. Different organizations, including professional associations, regulatory bodies, and academic facilities, would be likely sources of such formal examinations.⁴⁵

Even if the PT (or other health professional) receives objective feedback about his or her performance, several hurdles remain before the desired outcome of a more competent practitioner is achieved. First, the results of the objective testing or peer feedback must be accepted as reliable and valid. Then, the proper course of action to address these identified gaps in knowledge or skills must be taken, which, as noted above, may not always be the case.⁶ The process by which one identifies the areas in which one may lack competence, and then identifies the appropriate remediation strategies, is complex and warrants further study.

FUTURE RESEARCH INITIATIVES

In response to the disconnect between professional and educational rhetoric and research findings in the area of self-assessment, Eva and Regehr have recently proposed a reformulation of the concept of self-assessment to guide future research.⁶⁰ They argued that the problem with the self-assessment literature lies not in a methodological context but, rather, in a failure to effectively conceptualize the nature of self-assessment within the context of practice among health professionals. Self-assessment, they propose, is “a complicated, multifaceted, multipurpose phenomenon that involves a number of interacting cognitive processes,”^{60 (p. S47)} including self-efficacy and self-concept, cognitive and metacognitive processes, social cognition, expertise, and reflective practice. They conclude that self-assessment is “not a stable skill, but one that will vary by content, context, and perspective.”^{60 (p. S52)} So, rather than considering research questions such as “How much do I know compared to my classmates?” or “What mark might I get on this neurology test?” self-assessment needs to be considered in terms of its relationship to the content and context in which the health professional employs this activity in his or her day-to-day practice. Undertaking further study to determine the accuracy of self-assessment is no longer appropriate.

Research employing this reformulated concept of self-assessment⁶⁰ is needed to inform physical therapy practice and education. Colliver et al. suggested that in practice, “self-assessment is inherently qualitative.”^{61 (p. 200)} Additional qualitative research may offer further insight into the role and process of self-assessment. For example, identification of situations or triggers that motivate clinicians to seek advice or resources in the context of clinical decision-making might provide insight into self-assessment in practice. Understanding the barriers and facilitators within our professional culture as to the giving and receiving of valid feedback among peers could potentially support the process of self-regulation. Furthermore, exploring factors that motivate a clinician to accept feedback and act on it could also be useful. There are many opportunities for physical therapy researchers to contribute to the body of knowledge regarding self-assessment.

CONCLUSION

Self-assessment is an inherent part of autonomous practice by health professionals and is integral to learning for both students and clinicians. Current evidence, primarily from studies among student health professionals, indicates that self-assessment cannot be carried out with an appropriate degree of accuracy. Health professionals must seek external sources of information about their abilities and performance to supplement their own self-assessments. A reformulation of the concept of self-assessment has been proposed to guide future research initiatives.⁶⁰ There is an obvious need for research, both within physical therapy and in other health professions, to understand the process of self-assessment and to identify situations and strategies that will guide the selection of appropriate professional development activities and promote the provision of competent patient care.

Miller PA. Self-assessment: the disconnect between research and rhetoric. Physiother Can. 2008;60:117-124.