Multisource Feedback in the Trauma Context: Priorities and Perspectives

Abstract

Objectives

Trauma resuscitations require competence in both clinical and nonclinical skills but these can be difficult to observe and assess. Multisource feedback (MSF) is workplace‐based, involves the direct observation of learners, and can provide feedback on nonclinical skills. We sought to compare and contrast the priorities of multidisciplinary trauma team members when assessing resident trauma team captain (TTC) performance. Additionally, we aimed to describe the nature of the assessment and perceived the utility of incorporating MSF into the trauma context.

Methods

A convenience sample of 10 trauma team activations were observed. Following each activation, the attending physician trauma team leader (TTL), TTC, and a registered nurse (RN) participated in a semistructured interview. MSF was not provided to the TTC for the purpose of this study because MSF was not part of the assessment process of TTCs at the time of this study and maintaining anonymity may have encouraged more honest interview responses. Transcripts from each assessor group (TTL, TTC, RN) were coded and assigned to one of the five crisis resource management skills: leadership, communication, situational awareness, resource utilization, and problem‐solving. Comments were also coded as positive, negative, or neutral as interpreted by the coder.

Results

All assessor groups mentioned communication skills most frequently. After communication, the RN and TTC groups commented on situational awareness most frequently, comprising 15 and 29% of their total responses, respectively, whereas 31% of the TTL comments focused on leadership skills. The RN and TTL groups provided positive assessments, with 51 and 42% of their respective comments coded as positive. Forty‐five percent of self‐assessment comments in the TTC group were negative. All (100%) of the TTC and TTL respondents felt that incorporating MSF would add to the quality of feedback, only 66% of the RN group felt that way.

Conclusions

We found that each assessor group brings a unique focus and perspective to the assessment of resident TTC performance. The future inclusion of MSF in the trauma team context has the potential to enhance the learning environment in a clinical arena that is difficult to directly observe and assess.

Multisource feedback (MSF) is one method that may complement a competency‐based program of assessment because it is workplace‐based and involves the direct observation of learners. ¹ MSF provides varied perspectives and has been shown to provide valid, reliable, and comprehensive feedback in various contexts. ² , ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰

The resuscitation of a trauma patient requires a coordinated team‐based approach. At many institutions, residents are responsible for leading trauma resuscitations. At our institution, a resident trauma team captain (TTC) is primarily responsible for managing trauma patients under the supervision of an attending physician trauma team leader (TTL). Providing high‐quality feedback and assessment to resident TTCs presents several challenges that MSF may address. Trauma resuscitations occur infrequently and at unpredictable times, involve time‐sensitive and high‐stakes clinical decisions, and demand competence in both clinical and nonclinical skills. Specifically, crisis resource management (CRM) skills have been recognized as a set of nonclinical abilities that are required for successful team function in trauma resuscitation. ¹¹ , ¹² CRM skills include leadership, communication, situational awareness, resource utilization, and problem‐solving. MSF has demonstrated an ability to provide feedback on these nonclinical skills, ¹³ and prior work has found that nurses and physicians provide different views on resident performance. ¹⁴ , ¹⁵ Leveraging these alternative perspectives has the potential to improve the quality of feedback and provide a more wholistic assessment of TTC performance, specifically for CRM skills. ¹⁶

While there is literature describing the use of MSF in the context of simulated trauma scenarios, this has not been reported in the clinical setting. ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ We sought to compare and contrast the priorities of multidisciplinary trauma team members when assessing the performance of resident TTCs using a CRM framework. Second, we sought to describe the nature of the comments provided by different members of the trauma team and the perceived utility of MSF in a trauma context.

METHODS

We conducted a qualitative study that aimed to analyze and compare the comments provided by various multidisciplinary trauma team members regarding the performance of resident TTCs. This study was approved by the research ethics board at Queen’s University, and written informed consent was obtained from all study participants (REB ID #6020761).

Setting and Participants

This study took place at a single academic level one trauma center in Canada with approximately 400 trauma team activations per year. The trauma team is activated in cases of major trauma and is composed of a resident TTC; a TTL; two registered nurses (RNs); a respiratory therapist; and medical residents in anesthesiology, general surgery, and orthopedic surgery. The TTC is a medical trainee in at least his or her third year of postgraduate training in an emergency medicine or general surgery program. They are responsible for leading the resuscitation of trauma patients under the supervision of the TTL. The TTLs are attending critical care, emergency medicine, or general surgery physicians.

Data Collection

A convenience sample of 10 trauma team activations was observed between August 5, 2017, and August 28, 2017. Immediately following each trauma, the TTL, trauma team RN, and TTC each participated in a semistructured interview. Interviews lasted between 5 and 10 minutes. The interview had two parts. The first sought to establish priorities in the assessment of TTCs from the perspective of the different assessor groups (TTL, RN, self). Questions were open‐ended and designed to allow respondents to consider the “strengths and weaknesses” of the TTC’s performance without being steered toward specific topics. The second part of the interview sought to explore perceptions on the utility of MSF in the trauma team context. In this section participants were asked if incorporating MSF would add to the quality of feedback provided (or received) and to then expand on their response. MSF was not provided to the TTC for the purpose of this study for two reasons. First, MSF was not part of the assessment process of TTCs at the time of this study. Second, this allowed anonymity that may have encouraged more honest interview responses.

An a priori decision was made to collect data following 10 trauma team activations to provide a total of 30 responses (10 from each assessor group). A sample size of 30 was used as it would likely result in achieving saturation of the assessment priorities identified by participants. ²² To include broad representation, we did not interview the same TTC twice; however, RNs and TTLs may have been involved in more than one interview. Interviews were performed by study authors (AGP or TC) using a standardized questionnaire (Data Supplement S1, available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10533/full) and were audio recorded then transcribed. Personal identifiers were removed from the transcriptions before analysis.

Data Analysis

Assessment Priorities

A content analysis was undertaken on the transcribed data. Transcripts were coded line by line in an open coding method to describe the explicit content of the text. The coding process involved reading each transcript and tagging the meaning or content of each line briefly in the margins. ²³ For each participant, codes were assigned to the assessor group (TTC, RN, TTL) and then to one of the five CRM skills: leadership, communication, situational awareness, resource utilization, or problem‐solving. Furthermore, comments related to a CRM skill were coded as positive, negative, or neutral as interpreted by the coder. For example, if an RN assessor mentioned, “The TTC did a great job communicating with the team,” that was assigned the code RN/Communication/+. Specifically, a neutral assessment was assigned when a CRM skill was mentioned but no specific positive or negative performance was attributed to the statement. This was included to capture the frequency that various CRM skills were mentioned despite not having a negative or positive vector. An example of a neutral statement was a TTC stating “communication is important for me to know what is going on”; this was coded as TTC/Communication/neutral. Two authors (AGP and TC) coded all transcripts independently, and then both authors compared and discussed individual coding to be explicit about personal biases and to allow for transparency of codes. When coders disagreed on how a comment should be coded, this was resolved through dialogue. Data were analyzed with respect to the frequency with which each of the CRM skills were mentioned by each group of assessors.

Utility of MSF

A content analysis was performed on individuals’ perceptions of the utility of MSF. For each participant, responses were clustered and labeled by the coder as being either in favor of or against the incorporation of MSF in the feedback and assessment of TTCs. Participants then elaborated on their reasoning. Frequency analyses were performed for each assessor group to compare whether respondents were in favor of including MSF in the trauma team context.

RESULTS

Interviews were completed following 10 trauma team activations. Our final analysis included transcriptions for nine RNs, seven TTLs, and eight TTCs. The loss of assessors (one RN and three TTLs) was caused by a technical error with our recording device resulting in loss of recording of those interviews. Additionally, two TTCs did not complete the survey due to other competing clinical duties.

Assessment Priorities

Open coding of the transcripts yielded a total of 96 unique themes. On average RNs, TTCs, and TTLs provided 4.3, 3.9, and 3.7 codes per TTC assessment, respectively. Codes were then grouped and analyzed by CRM skill. Example quotes for each CRM skill can be found in Table 1. Communication skills were mentioned most frequently, comprising 54% (n = 52) of codes followed by leadership 20% (n = 19), situational awareness 18% (n = 17), resource utilization 8% (n = 8), and problem‐solving skills 0%. The distribution of CRM skills is presented in Figure 1.

Table 1.

Example Quotes for Each CRM Skill

CRM Skill	Participant Quotes
Communication	I just thought that there wasn’t really a lot of communication from the TTC to the rest of the team … in other traumas the TTC sometimes takes more control of the situation and sort of very openly and loudly asks each member of the trauma team to do their, sort of, assessment and then give feedback but I was sort of a bit blurry as to who was doing what and who was really taking control. (RN) Knowing that our role has significant value in the trauma team as well and good communication with us too, to be like this is what I would like next and this is what I would like next, in an effective manner … Just clear direction and understanding that our role is vital as well. (RN) I think just communicating with everyone … that’s what I find helps the RN’s role is communication. What the plan is, what needs to be done, and making sure that I actually know what’s happening so that I can keep stuff organized. (RN) I think it was nice because we had sometime before they (the trauma patient) got here and so I had some time and I think I did a good job at communicating what my plan was going to be with each of the services that I thought needed to be involved. (TTC)
Leadership	Her decisions were very clear. She was very direct in what she wanted us to do and it was just calm. There was no confusion at all. (RN) I’d like to command the room a bit better, it was a bit chaotic at times. I was sort of saying to [the TTL] that one of the benefits of our trauma team, and also maybe one of the drawback of our trauma team, is that everyone knows what they’re doing, and so at some points I was just kind of standing there passively waiting for things to happen as opposed to directing things myself and so I would like to work on that next time. (TTC) Probably just to be a bit more of a commanding presence in the room when it's very complex. She did a pretty good job of that. There were a few times it got a bit loud and it got a bit disorganized (TTL)
Situational awareness	I would get someone else to do the FAST. I performed the FAST and it was hard to keep track of everything going on while getting tasked focused (TTC) Realizes the importance of having nurses in here as well. Like the first trauma had 900 people, and it's like, when is somebody going to realize there's nobody in the room to do any of the vital signs and stuff. So, [the TTC] was the first one to come out and grab us because we were right there, we just couldn't physically get in. (RN) I think we kind of stayed organized and were able to not get distracted by what was the presumed injury and actually start at the top and work our way down. (TTC)
Resource utilization	It was a more complex, multisystem trauma so you can imagine the flow of personnel and resources. We had a lot of people which was great but managing their access was hard to monitor what’s going on. (TTC) When I called radiology to organize the CT scan they told me that the technician wasn’t in house. Based on the mechanism of trauma we knew that we would be getting a CT scan so I could have called prior to the trauma arrival to let them know to bring the technician in earlier. (TTC)
Problem‐solving	No comments regarding problem‐solving

CRM = crisis resource management; RN = registered nurse; TTC = trauma team captain; TTL = trauma team leader.

Figure 1.

Distribution of CRM skills identified by all participants. CRM = crisis resource management. 

The frequency with which each CRM skill was mentioned by each assessor group was then analyzed and is presented in Figure 2. Each of the assessor groups mentioned communication skills most frequently, comprising 62% (n = 24), 58% (n = 15), and 42% (n = 13), of the total responses for the RN, TTC, and TTL groups, respectively. After communication, the RN and TTC groups commented on situational awareness most frequently, comprising 15% (n = 6) and 29% (n = 9) of their total responses, respectively. Thirty‐one percent (n = 6) of the TTLs’ comments focused on leadership skills. Finally, resource utilization and problem‐solving were mentioned least often by all three groups of assessors.

Figure 2.

Distribution of CRM assessment priorities within each assessor group. CRM = crisis resource management. 

The nature of the codes (positive, negative, or neutral) was analyzed with results being presented in Figure 3. The RN and TTL groups predominantly provided positive comments, with 51 and 42% of their comments being positive, respectively. In the TTC group, 45% of all self‐assessment comments was negative.

Figure 3.

Percentage of positive, negative, and neutral assessment provided by assessor group. RN = registered nurse; TTC = trauma team captain; TTL = trauma team leader.

Perceived Utility of MSF

Eighty‐seven percent (n = 21) of respondents indicated that the inclusion of MSF would add to the quality of feedback given to TTCs. Specifically, 100% of both the TTC and TTL groups felt that incorporating MSF would improve the quality of feedback received and provided, respectively. Of the RN group, 66% (n = 6) felt that the addition of MSF would add to the quality of feedback given to TTCs, and the remaining 33% (n = 3) did not provide an answer that allowed us to appropriately judge whether they thought it would be helpful. Example quotes regarding the utility of MSF made by each assessor group are presented in Table 2.

Table 2.

Perspectives on the Utility of MSF

Assessor Group	Participant Quotes
RN	We’re there all the time for different traumas, and we can see the way that different trauma team captains act and lead and so we may be able to see strengths in other trauma team captains that we could, you know, maybe suggest to ones that were a bit weaker or newer doing it. I think that … it's always good for us to give them feedback like if they do a really good job, I think most of us try to let them know that they did well.
TTC	I think it’s feedback that I personally seek out, I think its super valuable and it tells you about how they (RNs) are feeling in regards to the management that we’re doing. It’s usually the resources from their perspective and if we’re overloading them or whatever. I’m only getting the (trauma team activations) that I run, while they (RNs, TTLs) see a lot more of them. They can help evaluate my performance as compared to a lot of the other TTCs. I think that’d be a very useful kind of feedback…because as a TTC you don’t see what the other TTCs are doing.
TTL	I think more in just the management, like if they (RNs) felt they were guided properly. If they ever felt they were under stress or not, or there were parts where they didn’t get any clear guidance from me. There's a lot of things going on and sometimes as a supervisor you don't see everything and the RNs are kind of always there and they see things that you wouldn't see.

MSF = multisource feedback; RN = registered nurse; TTC = trauma team captain; TTL = trauma team leader.

DISCUSSION

Multisource feedback that is based on direct observation in the workplace is well positioned to drive learning by leveraging the perspectives of multiple stakeholders in the assessment and feedback process. Furthermore, it may address the ongoing desire for more feedback on the part of learners. ²⁴ Specific to a team‐based competency such as a trauma resuscitation, incorporation of MSF may improve the quality of feedback provided to learners. We have used a CRM framework to describe the priorities in the assessment of resident TTCs from the perspective of multidisciplinary trauma team members.

Communication was the most commonly stated CRM skill in the assessment of TTCs. This is not surprising because it is inherent in many of the other CRM skills ²⁵ and is essential to a team’s success. ¹¹ The RN assessors commented on communication more so than the other groups, this is in keeping with prior literature ¹⁵ and suggests that RNs are well positioned to provide feedback on this skill. If RNs were not involved, important perspectives and valuable feedback may not be captured and fed back to trainees. The unique and important feedback from RNs relating to communication reflects the critical role that effective communication plays in the interaction between physicians and RNs in resuscitation scenarios. While physicians may be focused on the medical care of the trauma patient, RNs may have the opportunity to observe and critique the capacity of the TTC to engage with the whole team. In contrast, leadership was the second most common CRM skill mentioned, and the majority of comments were from the attending TTL group. The prioritization of leadership by the TTL group likely reflects their supervisory role on the trauma team, and the central role that this CRM skill assumes in the execution of critical tasks in a high pressure and complex clinical scenario. ²⁶ , ²⁷ Finally, situational awareness was the third most common CRM skill discussed and the TTC group was the most likely to mention this skill. This suggests that TTCs understand the importance of maintaining an overarching mental model of the scenario to foresee and prepare for the anticipated next steps. Further, it highlights their appreciation for a need to develop this as a skill, because many trainees struggle with this early in training, and thus is a main focus of their reflections on the scenarios.

The CRM skills of resource utilization and problem‐solving were mentioned infrequently or not at all by the assessor groups. This is in keeping with Hicks et al. ²⁸ who found that participants reported resource utilization to be the least important CRM skill that contributed to the outcome of a resuscitation scenario. These skills may also be difficult for assessors to tease apart from the other CRM skills. For example, the assessor may conceptualize the use of resources as a result of effective communication or leadership and not as its own skill. Similarly, problem‐solving may be lumped into the assessor’s schema of a good leader and not as its own independent skill. Future work to support assessor training or explicit prompts within the assessment tool will be helpful if the skills of resource utilization or problem‐solving are the objective of the assessment.

Our secondary outcomes included the nature of the qualitative assessment comments and the perceived utility of MSF in the trauma context. We found that the RN group and TTL group provided primarily positive comments while the TTC group provided negative self‐assessment comments. This is in keeping with Donnon et al. ¹⁰ who found that residents were more critical of their performance compared to their assessors. Both positive feedback that reinforces desired behavior and negative feedback that attempts to identify gaps and correct mistakes are important to the learning process. Although both reinforcing and constructive comments can be helpful to learners, it may be confusing if both types are received as part of the same assessment. However, there are two key reasons why this may not be harmful to the assessment process. First, within the medical education arena, MSF has been used as a tool to provide low‐stakes formative feedback to learners to guide their development, and both reinforcing and constructive comments can achieve this goal. ²⁹ Second, a strength of MSF is the representation of multiple stakeholders in the assessment process. Having both positive and negative comments within the same assessment reflects the positionality of those stakeholders.

The majority of all assessor groups and 87% of all respondents felt that the inclusion of MSF would improve the quality of feedback given to TTCs. Specifically, there were three themes that emerged from our data supporting MSF in the trauma context. First, all assessor groups commented that the experience of RNs on the trauma team is a valuable resource that could improve the quality of feedback for TTCs. Second, comments regarding communication skills were recognized as more legitimate when coming from the intended recipient (i.e., the RN) rather than a third‐party observer not directly involved in the conversation (i.e., the TTL). Aligning feedback with the assessor’s area of expertise is important as the recipient is more likely to accept it. ³⁰ Furthermore, assigning the team RN to provide feedback on communication skills might not only improve feedback in this domain but also allow the TTL to focus their attention on the observation of other skills. Finally, having multiple assessors allowed for various vantage points of the TTC’s performance during a trauma resuscitation. This was felt to “capture” more of their performance as trauma resuscitations are often a loud and crowded environment, making it difficult for a single assessor to see or hear the TTC at all times.

Our findings suggest that future inclusion of MSF to evaluate resident TTC performance has the potential to provide a team‐based assessment in a clinical arena that has traditionally been difficult to directly observe and assess. The next steps include the formal implementation of MSF for TTCs at our institution. We have also considered partitioning domains of assessment (in this case, CRM skills) among the assessor groups, for example, asking RNs to assess communication and TTLs to assess leadership. We found it difficult at times to find sufficient time to complete MSF in a busy emergency department setting. Partitioning the feedback may improve the efficiency and decrease the assessment burden placed on a single assessor group.

LIMITATIONS

Our study has several limitations. Although we did capture the assessment priorities of TTLs, TTCs, and RNs, ideally other members of the trauma team would have contributed to MSF. We recognize that MSF is resource‐intensive and chose to start with a small number of assessor groups for this study. Additionally, this was a convenience sample with a relatively small sample size and may not fully reflect the assessment priorities of the represented groups. The fact that some assessors were interviewed on multiple occasions may also change the frequency with which some CRM skills were mentioned because these may be specific assessment priorities of that assessor. Finally, the subjective coding of quotes to specific CRM skills may threaten the internal validity of this study. To mitigate this, we approached the data through line‐by‐line open coding, allowing the researchers to remain close to the data and collaboratively verify analyses.

CONCLUSION

We have described the priorities in the assessment of resident trauma team captains from the perspective of multidisciplinary trauma team members using a crisis resource management framework. Although communication was the most prevalent crisis resource management skill discussed by all assessor groups, each group prioritized different principles in their feedback. Our results support the future inclusion and further study of multisource feedback in the trauma team context to enhance the learning environment in a clinical arena that has traditionally been difficult to directly observe and assess.

Supporting information

Data Supplement S1. Trauma team RN questionnaire.

AEM Education and Training 2021;5:1–8