Psychological Safety and Support: Assessing Resident Perceptions of the Clinical Learning Environment

ABSTRACT

Background

Assessments of the clinical learning environment could allow early interventions to improve graduate medical education. To date, measurement tools for this have not been identified.

Objective

We established the concurrent validity of 2 instruments that assess cultural facets of the clinical learning environment by correlating them with external program evaluation data.

Methods

In 2017 we surveyed residents across 19 training programs on their perceptions of organizational support by using the Short Survey of Perceived Organizational Support (SPOS), and psychological safety by using the Psychological Safety Scale (PSS). Data were aggregated to the program level and correlated with results from the Accreditation Council for Graduate Medical Education (ACGME) Resident Survey.

Results

Of 511 residents, 322 (63%) completed the survey, and 496 of 516 residents (96%) completed the ACGME Resident Survey. Perceived organizational support correlated positively with overall program evaluation score (r = 0.75, P < .001); faculty (r = 0.72, P < .001); evaluation (r = 0.73, P < .001); educational content (r = 0.52, P = .022); and resources domains (r = 0.55, P =  .014). Psychological safety had a positive correlation with overall program evaluation (r = 0.57, P = .011); faculty (r = 0.50, P = .028); and evaluation (r = 0.62, P < .005).

Conclusions

The SPOS and PSS correlated with key ACGME Resident Survey domains. Programs showing greater support of residents were likely to show more positive ratings on program evaluation metrics. Teaching institutions may benefit from actively monitoring and improving aspects of their learning environment through internal assessments.

What was known and gap

The quality of the clinical learning environment is important to learners' professional development and well-being, yet there is a dearth of assessment tools in this area.

What is new

A study found correlations between the Short Survey of Perceived Organizational Support (SPOS) and the Psychological Safety Scale (PSS) and Accreditation Council for Graduate Medical Education resident survey data.

Limitations

Single institution study limits generalizability; associations do not allow for causal inferences.

Bottom line

The SPOS and PSS can be self-administered and offer actionable data for program improvement.

Introduction

Teaching institutions struggle to identify methods to monitor their clinical learning environment. The learning environment is an important component of medical education, with an impact on trainees' learning, professional development, and well-being,¹ yet it is not often measured when determining educational effectiveness. Our definition of the clinical learning environment is based on the framework by Hoff and colleagues,¹ in which context such as workload, relationships, and work-life strain are facets that shape the learning culture (eg, support, respect, openness, habit of inquiry), which ultimately influences residents' ability to acquire core competencies. While assessment of the learner has grown in medical education, measurement of the multiple and dynamic facets of the clinical learning environment lags behind.²

The Accreditation Council for Graduate Medical Education (ACGME) recognized the importance of context when it instituted the Clinical Learning Environment Review program,³ which provides formative feedback to academic institutions on the safety and quality of care domains approximately every 24 to 36 months. Data are presented in aggregate for the entire institution. This review, while beneficial, does not offer tailored guidance for individual programs to identify and address areas of vulnerability, nor can it guide program improvement. The ACGME also provides data from its annual Resident Survey as a measure of program effectiveness data to program directors on an annual basis near the end of the academic year. This facilitates benchmarking to other programs, yet it leaves little time to make improvements before residents' transition.

A literature review found the national ACGME Resident Survey currently offers more validity evidence of assessing the environment compared with other measures.^2,4 Considering that measures of organizational culture and context are used primarily in work settings with nontrainees, we compared 2 such tools to the ACGME Resident Survey to determine whether they could be used to assess the clinical learning environment. We also sought to gain insight into positive and negative relationships between the tools to measure these cultural constructs, and the domains of the ACGME Resident Survey.

To better understand the clinical learning environment, we focused on 2 constructs that represent aspects of culture: perceived psychological safety and organizational support. These measures have been studied across industries in an effort to link culture, context, and organizational outcomes.^5,6 They do not assess the entire domain of the clinical learning environment, but measure cultural facets of the construct that could be used proactively for program improvement.¹

Psychological safety, part of the concept of “just culture,”⁷ is defined as the belief that individuals can openly voice concerns and opinions, report events, and share ideas in the workplace—especially members with comparatively lower status in the group.⁸ In high-risk industries, such as aviation, the military, and health care, there have been catastrophic outcomes due to the failure of individuals feeling free to speak up.⁹ In teaching settings, trainees often turn to what is normative within the cultural facets of the clinical learning environment. This has been termed the “hidden curriculum.”¹⁰ Medical education culture still reverts to hierarchy and power, where speaking in opposition of someone who is “higher in rank” may violate unstated norms.¹¹

Research across industries suggests psychological safety either supports or hinders learning behavior depending on whether there is a low or high sense of safety in speaking up,⁹ and that it affects learning behaviors and team performance.^9,12 Torralba and colleagues¹³ found resident physicians' psychological safety positively related to their satisfaction with their clinical learning experience in the Department of Veterans Affairs.

Organizational support theory suggests that individuals tend to personify organizations and actions of organizational agents (eg, supervising faculty) as actions of the organization.¹⁴ Judgments on whether the organization cares about its members can create negative or positive bonds between individuals and their organizations.¹⁵ Fairness in how work is done, how resources are allocated, and respect for others are some factors that influence perceptions of organizational support.¹⁵ When members feel supported, they reciprocate feelings of support back to the organization, leading to increased commitment and better performance.¹⁶ While the majority of research on perceived organizational support has been in nonmedical settings, studies have assessed perceived organizational support with nurses^17,18 and physicians in non-US populations.^19,20

We aimed to establish the concurrent validity of 2 instruments that assess cultural facets of the clinical learning environment—perceived organizational support and psychological safety—by correlating them with external program evaluation data.

Methods

Participants and Procedure

In June 2017, the offices of graduate medical education (GME) and quality and safety at the Virginia Commonwealth University School of Medicine conducted an internal survey of residents across 19 programs on perceived organizational support and psychological safety. The survey was administered in person by a faculty member not associated with supervision in residency programs.

Measures

Two measures demonstrating validity evidence—the Short Survey of Perceived Organizational Support²¹ (SPOS) and the Psychological Safety Scale⁹ (PSS)—were used to assess the clinical learning environment. The SPOS was introduced more than 30 years ago²¹ and has been used in the majority of studies on the construct.¹⁵ Long and short forms of the SPOS have demonstrated high reliability and unidimensionality across occupations and organizations.¹⁵ The PSS has established validity and reliability evidence.⁹ A recent review on psychological safety found the majority of studies on the construct used the long or short form of PSS.⁵

The SPOS and PSS scales were embedded into a longer survey on quality and safety initiatives in GME. The SPOS measure consisted of 16 items. A sample item includes, “Help is available from my department when I have a problem.” The PSS used 7 items, with a sample including, “Members of my department are able to bring up problems and tough issues.” Both measures used a 5-point Likert scale (1, strongly disagree, to 5, strongly agree). Items were slightly adapted—the terms “organization” from SPOS and “team” from PSS were changed to “department” to reflect the resident population.

The ACGME Resident Survey measures 5 program domains (faculty, evaluation, educational content, resources, and patient safety/teamwork) through Likert and yes/no formats.^4,22 Each domain has 5 to 9 items. Each department is provided a mean value for each domain that is aggregated from resident responses. The survey's overall program rating was converted from a percentage to a mean score for each program, based on the overall number of respondents and the percentage of respondents who chose each of the 5 Likert scale options.

The Virginia Commonwealth University Institutional Review Board deemed our study nonhuman subjects research upon review.

Analyses

Negatively worded items were reverse coded prior to analysis and aggregation. A principal components analysis was conducted for SPOS and PSS. Responses were aggregated to create a program-level score on the SPOS and PSS, which was linked to program-level ACGME Resident Survey results. Aggregation is appropriate because support, safety, and ACGME program data are program-level constructs. Interrater agreement (r_wg) was calculated to support program-level aggregation for SPOS and PSS. A bivariate correlation was conducted through SPSS version 24 (IBM Corp, Armonk, NY) to identify associations between our internal measures and the ACGME Resident Survey.

Results

Across 19 residency programs, 322 of 511 residents (63%) completed the internal survey (Table 1), and 496 of 516 residents (96%) completed the academic year 2016–2017 ACGME Resident Survey. Principal components analysis confirmed past findings that SPOS and PSS are unidimensional (SPOS eigenvalue = 8.86, loadings 0.49–0.74; PSS eigenvalue = 2.93, loadings 0.55–0.86); SPOS explained 55% of variance in its underlying construct, while PSS explained 42% variance. There was support to aggregate SPOS (r_wg = 0.80) and PSS (r_wg = 0.79) scores to the program level based on interrater reliability calculations.²³ Reliability analyses showed good internal consistencies for SPOS, PSS, and the majority of ACGME domains (Table 2). Bivariate correlations (Table 2) revealed strong positive correlations between SPOS and PSS (r = 0.88, P < .001); overall program score (r = 0.75, P < .001); faculty (r = 0.72, P < .001); evaluation (r = 0.73, P < .001); educational content (r = 0.52, P = .022); and resources (r = 0.55, P = .014). Psychological safety had a strong positive correlation with overall program score (r = 0.57, P = .011); faculty (r = 0.50, P = .028); and evaluation (r = 0.62, P < .005).

Table 1.

Internal Assessment Response Rates

Table 2.

Bivariate Correlation Matrix, Scale Reliabilities, Means, and Standard Deviations Across Programs (N = 19)

Discussion

We found that programs with higher scores for organizational support and psychological safety also were likely to show more positive ratings on the ACGME Resident Survey. Faculty and evaluation domains were highly correlated with our culture measures, suggesting supervision and feedback processes may inform perceptions of support and safety.

Measures of perceived organizational support and psychological safety have powerful utility for health systems' efforts to actively monitor and improve GME. While medical educators have a duty to ensure trainees demonstrate competence in their specialty, they also need to attend to the environment in which residents learn and develop as physicians.

The purpose of internal measurements of the learning environment is to affect change. The process of enhancing fairness, support, and respect in medical education is often presumed to exist by virtue of having residency training. However, the hidden curriculum may undermine some improvement efforts, resulting in cynicism among trainees.²⁴ Perceptions of support and psychological safety are cultural constructs that differ from trainee satisfaction, and deliberate interventions focused on support and safety are needed.^6,25 Using the SPOS and PSS data to evaluate the impact of interventions to increase perceptions of support and safety, such as structured mentoring²⁶ or a safe process for discussing concerns,²⁷ is a practical application of these findings. Internal assessments also can help leadership direct tailored interventions to specific ACGME Survey domains. Program improvement should be ongoing and use multiple sources (eg, residents, program directors) and methods (eg, internal and ACGME surveys, focus groups/interviews, observations) to guide the effort.

While the ACGME Resident Survey was considered the criterion for concurrent validity, there are concerns that the national survey may not offer a true reflection of the clinical learning environment.^28,29 A survey of internal medicine program directors revealed concerns with the ACGME Resident Survey to include ambiguous wording of some questions, and that program directors cannot access the actual survey items.³⁰ Considering the high correlation between the ACGME Resident Survey and the tools on safety and support, program directors could use the 2 measures to gain insight into their clinical learning environments.

The SPOS and PSS demonstrated utility in a resident population, adding to the limited tools identified to measure aspects of the clinical learning environment. Residency program directors and faculty should consider monitoring scores on cultural measures, such as SPOS and PSS, to gauge potential concerns on the ACGME Resident Survey and as a proxy for program effectiveness.

Our study has limitations, including that it was conducted at 1 institution, limiting generalizability, and its correlational nature does not allow for causal inferences between the constructs. Aggregation to the program level may limit the interpretation of findings with low intraclass correlations.

Further research, which may include direct observation and qualitative assessment, is needed to determine directionality between study variables and other potential measures to assess facets of the clinical learning environment.

Conclusion

In a study of 19 residency training programs at 1 institution, the SPOS and the PSS correlated highly with ACGME Resident Survey domains. Programs where residents reported greater perceived organizational support and psychological safety were likely to show more positive ratings on program evaluation metrics.