Job Analysis and Student Assessment Tool: Perfusion Education Clinical Preceptor

Abstract:

The perfusion education system centers on the cardiac surgery operating room and the perfusionist teacher who serves as a preceptor for the perfusion student. One method to improve the quality of perfusion education is to create a valid method for perfusion students to give feedback to clinical teachers. The preceptor job analysis consisted of a literature review and interviews with preceptors to list their critical tasks, critical incidents, and cognitive and behavioral competencies. Behaviorally anchored rating traits associated with the preceptors’ tasks were identified. Students voted to validate the instrument items. The perfusion instructor rating instrument with a 0–4, “very weak” to “very strong” Likert rating scale was used. The five preceptor traits for student evaluation of clinical instruction (SECI) are as follows: The clinical instructor (1) encourages self-learning, (2) encourages clinical reasoning, (3) meets student’s learning needs, (4) gives continuous feedback, and (5) represents a good role model. Scores from 430 student–preceptor relationships for 28 students rotating at 24 affiliate institutions with 134 clinical instructors were evaluated. The mean overall good preceptor average (GPA) was 3.45 ± 0.76 and was skewed to the left, ranging from 0.0 to 4.0 (median = 3.8). Only 21 of the SECI relationships earned a GPA <2.0. Analyzing the role of the clinical instructor and performing SECI are methods to provide valid information to improve the quality of a perfusion education program.

Perfusionists are highly educated and trained allied health professionals who operate life support equipment during cardiac surgery and in special care areas in health care facilities. The clinical perfusion education system centers on the cardiac surgery operating room and the clinical perfusionist who serves as a preceptor or clinical instructor for the perfusion student (Figure 1).

Figure 1.

Perfusion education learning system.

Why assess? Springer et al. (1) reported “… If used effectively, performance appraisal systems keep employees focused on the vision and mission of the organization.” Instructor appraisal forms based on a valid job analysis can be an aid to improve performance, or in the case of a perfusion education program, a method to improve the quality of education.

Analyzing the role of clinical instructor and facilitating student feedback to clinical teachers is one method to improve the quality of a clinical education program (2). Health education programs should be continuously involved in total quality management (3). Student evaluation of clinical instructors (SECI) may be a valuable addition to a perfusion student experience.

Students have requested the opportunity to evaluate clinical faculty. There have been no requests for SECI from the >130 clinical instructors in the 29 clinical affiliates to The Ohio State University (OSU) Circulation Technology (Circ Tech) Program (www.amp.osu.edu/CT). OSU perfusion students presently evaluate their clinical rotation health care facility but do not evaluate clinical instructors.

A job analysis is a logical and legal requirement for performance appraisal and for evaluation instrument content validity (4). Content validity is probably more important than legal foundation when it comes to students evaluating clinical instructors within the context of this study.

Perfusion education clinical instructors perform first and foremost as perfusionists taking care of cardiac surgical patients. The job description and scope of practice for the perfusionist is well known, well studied, and published (www.amsect.org/). Perfusionists participate in a national certification process (www.abcp.org/) and are licensed in >11 states in these United States.

When a perfusionist volunteers to become a clinical instructor, they add another dimension of activities to their already important list of tasks. This study focuses on the additional tasks that are added to the perfusionist’s role during clinical instruction. The purpose of the perfusion preceptor job analysis is twofold: to analyze the education-related tasks of perfusion clinical instructor and to develop an instrument for perfusion students to use to rate their preceptors.

For the job analysis, the first goal is to list the critical tasks, cognition, and behaviorally anchored statements (5) that describe the role of clinical instructor for perfusion students. The second goal is to create an SECI instrument for perfusion students to use at the end of a clinical rotation to evaluate each of their preceptors. “Unambiguous” behaviorally anchored rating scales (BARS) described by Smith and Kendall (6) will aid clinical instructors to understand their role and aid students to evaluate the instructor.

MATERIALS AND METHODS

There are three main approaches to the job analysis used in this study: a procedural analysis, a cognitive task analysis, and a critical incident analysis (4,7–9,11). These data will be collected by observation and interview of expert clinical instructors—the subject matter experts. Some of the data collected by task and cognitive analysis come from direct observation and interaction. Critical incident identification comes from personal experience and interviews with perfusion clinical instructors. Critical incident theory (CIT) is the cornerstone for developing performance and understanding appraisal measures (10). In addition to these three methods, a review of the clinical education literature will reveal additional traits for effective clinical instructors. The cognitive task analysis was performed according to the work of Grunwald et al. (9) and Hanser (8).

Next, behavioral statements from the job analysis and the literature review were used to develop the construct for a perfusion education SECI instrument. The statements were presented to perfusion students for selection and validation. The SECI instrument was finalized, and the students were trained in the use of the SECI.

Prospectively, the SECI was used by the students in two graduating classes at OSU to evaluate their clinical preceptors beginning in winter 2004 to spring 2006. Statistical analysis was performed using SPSS version 14.0 (SPSS, Chicago, IL), and the α level was set at 0.05.

RESULTS

Effective, experienced clinical instructors were able to report the critical incidents and the consequences of those incidents. The job statements collected from expert clinical preceptors during interviews to discover their critical tasks during clinical teaching are presented in Table 1. The statements were collated into five general tasks specifically related to clinical teaching.

Table 1.

Results of job analysis.

Tasks		Incidents and Statements
Task 1	Critical incident	“Teach, monitor, and double-check the daily detailed work and skills of the perfusion student”
	Cognition	Goal is patient safety Student demonstrates competency Student gains confidence of preceptor
	Behaviorally anchored statements(s)	Carefully checks student work Monitors student for safe practice Goes out of the way to explain principles Assigns valid and reasonable homework
Task 2	Critical incident	“Model, monitor, and document student mastery of competence”
	Cognition	Evaluate student competency Compare student to performance standard Encourage student independence
	Behaviorally anchored statement(s)	Challenges the student’s thinking Forces student to confront mistakes Encourages independence on all levels Motivates student to do better
Task 3	Critical incident	“Introduce the student to, and help the student integrate into the patient care arena and team”
	Cognition	Gain team confidence in student Student shows appropriate behavior Student shows equipment-based competency
	Behaviorally anchored statement(s)	Accessible for questions Is eager to help students Is fair in student evaluation Uses positive reinforcement
Task 4	Critical incident	“Act as a clinical consultant, mentor, and role model to the perfusion student”
	Cognition	Build student knowledge base Student shows minimal knowledge Student incorporates new learning into daily activities
	Behaviorally anchored statement(s)	Able to handle crisis situations Collaborates information with student Is a positive influence Confident in their own clinical skills
Task 5	Critical incident	“Give students informal and formal feedback by completing and submitting daily case, interim, and final rotation evaluation forms”
	Cognition	Exhibit effective communication with student, and conversely Student trusts preceptor, and vice versa Student performs consistently to standard
	Behaviorally anchored statement(s)	Shows students respect Sits and talks with student after procedure Criticism is constructive Is accurate in student evaluation

Studies from the medical clinical education literature helped to identify previously described traits of preceptors such as the work of Snell et al. (12), Hamdy et al. (13), and Archer (14). Hamdy et al. (13) identified nine indicators of clinical faculty from a 2001 literature review. Archer (14) listed many statements to describe ideal clinical instructor behaviors from a student point of view. Assimilating the nine traits from Hamdy et al. (13), Archer (14), and the five groups of preceptor tasks from the job analysis, a survey was created to allow current circulation technology (CT) students to perform a paired-comparison procedure (15) to discover the top five traits. Table 2 presents the clinical student survey that includes the nine traits discovered by this method. Because the task analysis interview seemed to yield similar traits as the review by Hamdy et al., current CT students were asked to rank the nine traits. The results of the paired-comparison procedure showed the ranking of the traits based on total votes by nine students in Table 2.

Table 2.

Student survey to choose top five clinical instructor traits.

Indicator Statement	Median	Votes	Rank	SD	Mean
Encourages self-learning	7	49	1	2.7	5.4
Encourages clinical reasoning	5	41	2	1.7	4.6
Meets student learning needs	5	41	2	2.0	4.6
Represents a good role model	6	41	2	3.4	4.6
Gives continuous feedback	3	35	5	2.3	3.9
Eases student access to patient/team	5	34	6	2.5	3.8
Accessibility/availability	2	31	7	3.0	3.4
Encourages professional skills	3	29	8	1.5	3.2
Defines clerkship objectives	2	23	9	2.6	2.6

BARS from the preceptor interview task analysis were related to the top five traits from the student survey. The BARS below was used to help students use the SECI instrument. The BARS in Table 3 were used to train the students as raters to use the SECI instrument in a consistent manner.

Table 3.

BARS matched to top five preceptor traits.

Behaviorally Anchored Rating Statements from Task Analysis	Top Five Traits*
	1	2	3	4	5
Challenges the student’s thinking	X	X
Encourages independence on all levels	X				X
Motivates student to do better	X			X
Assigns valid and reasonable homework	X		X
Collaborates information with student		X	X
Sits and talks with student after procedure		X	X
Monitors student for safe practice			X	X
Shows students respect			X	X
Accessible for questions			X
Forces student to confront mistakes			X		X
Criticism is constructive			X		X
Goes out of the way to explain principles			X		X
Is eager to help students			X		X
Is fair in student evaluation			X		X
Is accurate in student evaluation			X		X
Is a positive influence				X
Confident in their own clinical skills				X
Uses positive reinforcement				X
Carefully checks student work					X
Able to handle crisis situations					X

^*Top five traits: (1) encourages self-learning; (2) encourages clinical reasoning; (3) meets students learning needs; (4) gives continuous feedback; (5) represents a good role model.

Figure 2 presents the SECI evaluation instrument created for use by CT students. The instrument is a five-dimension semantic differentiation scale of potency—strong vs. weak (16). The SECI instrument results from each student were scored to award each preceptor a good preceptor average (GPA) between 0.0 and 4.0. The following equation was used to calculate the GPA. For each of the five trait (T) scores, −2 Weak = 0 points, −1 Weak = 1 point, 0 Non-committal = 2 points, +1 Strong = 3 points, and +2 Strong = 4 points,

Figure 2.

Circulation technology SECI instrument.

$$\text{GPA}=\text{SUM(T1 points}+\text{T2 points}+\text{T3 points}+\text{T4 points}+\text{T5 points)/5.}$$ (1)

Student rating results for preceptors were entered into a database at the end of each rotation. A specific preceptor overall GPA was calculated from several student ratings for the five traits. An overall GPA for each of the preceptors at each clinical facility was calculated, reported, and statistically compared with other preceptors and other affiliates. There were 430 preceptor–student rating pairs. Ninety-two percent of the GPAs were >2.2.

Tables 4 and 5 present the ANOVA comparison for the overall GPAs for 134 preceptors, 28 students, and 24 clinical affiliate hospitals. There were no significant differences between the overall mean scores for the five traits. There were small significant differences between the scores that students awarded affiliate hospital groups overall. However, the largest significant differences were in the scores that individual students awarded individual preceptors. There were no significant differences between the clinical affiliates as rated by the students.

Table 4.

ANOVA tests of between GPA for affiliate hospitals and students.

Source	Type III SS	df	MS	F	Sig	Partial η2	Obs Power
Student	38.221	27	1.416	3.151	.000	.194	1.00
Affiliate	15.201	23	0.661	1.471	.077	.087	0.95
Affiliate*student	26.182	25	1.047	2.330	.000	.141	0.99
Error	159.080	354	0.449
Total	245.383	429

Affiliate*student is the interaction between affiliate hospitals and students; partial η² is the effect size where 0.15–0.65 is moderate; Obs power is the observed power or probability of avoiding a type II statistical error.

Table 5.

ANOVA tests of between GPA for preceptors and students.

Source	Type III SS	df	MS	F	Sig	Partial η2	Obs Power
Student	35.757	27	1.324	17.167	.000	.971	1.00
Preceptor	88.362	133	0.664	8.612	.000	.988	1.00
Preceptor*student	113.049	255	0.443	5.747	.000	.991	1.00
Error	159.080	354	0.077
Total	245.383	429

Preceptor*student is the interaction between preceptors and the students; partial η² is the effect size where 0.15–0.65 is moderate; Obs power is the observed power or probability of avoiding a type II statistical error.

DISCUSSION

The preceptor GPA will become part of an annual report mailed to each affiliate hospital by the education program. Hamdy et al. (13) presents “visual indicators of teaching and learning success” (VITALS) to report student evaluation of clinical instructors. The visual report of Hamdy et al. presented the number of students that score instructors as weak or strong. Neutral or non-committal scores are counted as weak. Figure 3 shows the graphic of Hamdy et al. for the scores for each trait. Figure 4 shows the plot of the four student GPAs for the five traits for the same preceptor. It will be easy for the preceptor in Figure 4 to see how the students rate them.

Figure 3.

How 28 students rated 134 preceptors from 24 clinical affiliate hospitals. The graphic is adapted from Hamdy, et al., 2001.

Figure 4.

How five students rated one preceptor on five clinical instruction traits. The graphic is adapted from Hamdy, et al., 2001.

It is important to train the student raters. Latham and Wexley (4) admonished that training raters decreases bias and increases the accuracy of assessment that in turn increases the probability of increasing productivity. In this case, training students to evaluate preceptors may improve the productivity of the instructors. Improved instructor motivation may translate into increased preceptor performance and perhaps will improve the students’ educational experiences in the clinic.

It is important to educate the clinical preceptors to the SECI process and how to read and understand the resulting reports and the concept of the GPA. Preceptor buy-in and acceptance of the SECI process is an important step that will be facilitated by their orientation.

Doque (17) and others have stated that the traditional preceptorship for clinical students is often inconsistent and may be unstructured between rotations. Giving students the opportunity to give preceptors feedback may increase the consistency of clinical experiences for future students. Doque (17) advocated using the immediacy of the Internet to facilitate evaluation of the student in the clinic. Student evaluation of instructors may take place on-line as well.

CONCLUSION

The role of clinical instructor or preceptor was job-analyzed in this study. Based on a literature review, student input, and the results of the job analysis, five traits were defined and supported with behaviorally anchored rating scales. An SECI instrument was developed. Two different preceptor reports are presented to show the value of student feedback. The SECI instrument and the results of the process should serve as a useful future research tool.

The SECI instrument has content validity based on the job analysis and student confirmative survey results. Forthwith, the SECI process will be used in the Circulation Technology Program to evaluate clinical instruction in the clinical practicum courses.