Using the Postgraduate Hospital Educational Environment Measure to Identify Areas for Improvement in a Singaporean Residency Program

Abstract

Background

Attributes of the clinical learning environment (CLE) are a measure of quality in postgraduate medical education, and assessing the CLE is a component of the New Accreditation System being introduced in Singapore by the Accreditation Council for Graduate Medical Education International. There is a dearth of published studies of CLE quality in Singapore.

Objective

Our study had 3 aims: (1) to measure the CLE in 1 Singaporean residency program; (2) to compare trainee perceptions by sex, training level, and experience; and (3) to identify areas for improvement.

Methods

Between October and December 2017, we conducted a mixed assessment of the CLE in an internal medicine program in Singapore, using the Postgraduate Hospital Educational Environment Measure (PHEEM) and qualitative exploration using a focus group.

Results

Of 153 IM residents, 136 (89%) provided PHEEM responses and 8 participated in the focus group. Total PHEEM scores and scores for the 3 subscales were higher than published data on the use of the PHEEM in international settings. Exploration of selected PHEEM responses via a focus group identified attributes associated with negative perceptions of the CLE: excessive workload, inadequate faculty presence in the CLE, and unmet trainee needs. It also suggested senior residents' clinical workloads, greater responsibilities, and pending examinations may contribute to their less positive perceptions of the CLE.

Conclusions

Our analysis using the PHEEM showed overall positive perceptions of the CLE, along with areas for improvement amenable to interventions. Our approach has relevance to an accreditation model with ongoing evaluation of the CLE.

What was known and gap

New accreditation approaches emphasize evaluation of the clinical learning environment (CLE). There is a lack of published studies that have explored the CLE in Singapore.

What is new

A study used the Postgraduate Hospital Educational Environment Measure (PHEEM) and focus group data to evaluate Singaporean internal medicine trainee perceptions of the CLE.

Limitations

Single specialty, single program study reduces generalizability.

Bottom line

The study added to the data on the international utility of the PHEEM, and identified strengths and areas for improvement in the local CLE.

Introduction

The clinical learning environment (CLE) plays a significant role in postgraduate medical education (PGME) by providing the context for learning and patient care and ensuring a balance of education and service.¹ Its attributes are important to the quality of PGME.² A positive CLE has been associated with improved workplace learning,³ higher examination scores,⁴ and greater career satisfaction,^5,6 while negative perceptions of the CLE have been associated with trainee burnout.^7–9 Recognizing this, the Accreditation Council for Graduate Medical Education (ACGME) in the United States implemented the Clinical Learning Environment Review (CLER) program as part of its Next Accreditation System (NAS).¹⁰ One focus of CLER is trainee well-being, and prior research showed that Singaporean residents have higher burnout rates compared to trainees in Western nations.¹¹ The accreditation approach of ACGME International (ACGME-I) in Singapore is changing to an approach similar to the NAS in the United States, with a heightened focus on the CLE. Our study sought to (1) measure the CLE of 1 internal medicine (IM) residency program in Singapore; (2) compare perceptions of the CLE by gender, level of training, and work experience; and (3) identify opportunities for improvement.

Methods

ACGME-I began to accredit programs in Singapore in 2009. The SingHealth IM program encompasses 150 residents in 3 years of training. Training takes place at 3 major sites: Singapore General Hospital, Changi General Hospital, and National Neuroscience Institute.

We used a mixed-methods approach, including quantitative measurement of the CLE using the Postgraduate Hospital Educational Environment Measure (PHEEM)¹² and a qualitative exploration of selected PHEEM items in a resident focus group.

The PHEEM

The PHEEM is specifically designed to assess the CLE for postgraduate trainees.¹³ It consists of 40 items in 3 subscales that respectively address perceptions of role autonomy, teaching, and social support.¹² Scoring uses a 5-point Likert scale (0, strongly disagree, to 4, strongly agree) for a maximum score of 160.¹² The PHEEM has demonstrated acceptable reliability and content validity in a range of international settings,^13,14 and it is the most widely used instrument for measuring the PGME learning environment around the world.^13,14 Similar to another study using the PHEEM in Singapore,¹⁵ we made slight modifications to the instrument, including changing “bleep” to “receive phone calls” and “new deal” in Q17 to “ACGME-I requirements.” We deemed this would not affect instrument validity, and no further validity testing was conducted.

We administered the PHEEM online to 153 IM residents between October and December 2017 using New Innovations (Uniontown, OH, USA), an online secure platform our residency program uses to track progress of our residents. We included questions on respondents' sex, training level, work experience, and training sites, and free-text comments. Participation was voluntary and we assured anonymity of responses. Residents received 3 reminders to maximize the response rate.

Qualitative Exploration With Focus Group

One author (A.M.O.) conducted a 75-minute, semistructured focus group session to supplement the PHEEM data with a purposive sample of 8 residents (5 women; 2 postgraduate year 1 [PGY-1], 3 PGY-2, and 3 PGY-3). The PHEEM results were presented to the group and influenced the questions. Consent was obtained from participants for audio-recording and verbatim transcribing of interviews.

The summary of the interview transcript was made available to the interviewees for an accuracy check, which did not yield any revisions. Two authors (A.M.O. and W.W.F.) used open coding¹⁶ to reduce data from narrative comments on the questionnaires and data from the focus group to highlight common themes.

The SingHealth Centralised Institutional Review Board declared the study exempt from formal ethical board review as no patient information was included.

Data Analysis

We analyzed the data using SPSS Statistics 23 (IBM Corp, Armonk, NY, USA). After checking the data for normal distribution, using the unpaired Student's t test for parametric quantitative data comparisons, the Wilcoxon signed rank test for nonparametric quantitative data, and analysis of variance with post-hoc Bonferroni corrections for multiple group comparisons. Differences were considered significant if P < .05. Reliability analysis of the PHEEM used Cronbach's alpha coefficient.

Results

Of 153 residents, 136 (89%) responded (see Table 1 for respondents' demographic information). Cronbach's alpha was 0.95, and the exclusion of individual questions did not produce significant changes in the score, reflecting high internal reliability. This adds to the literature on the validity of the PHEEM in international settings.

Table 1.

Characteristics of Study Population

Characteristic	N (%)
Gender
Male	70 (52)
Female	66 (49)
Residency year
R1	42 (31)
R2	51 (38)
R3	43 (32)
Work experience
PGY-1	18 (13)
PGY-2	25 (18)
PGY-3	28 (21)
PGY-4	21 (15)
PGY-5	13 (10)
> PGY-5	31 (23)
Training site
Singapore General Hospital	90 (66)
Changi General Hospital	41 (30)
National Neuroscience Institute	5 (4)

Abbreviation: PGY, postgraduate year.

The mean total PHEEM scores (112.23 ± 16.7) and scores for the 3 subscales were higher than other studies that have used the instrument in international settings,¹⁴ although we were not able to assess significance due to lack of access to the raw data (Table 2). Scores for all PHEEM items are shown as online supplemental material.

Table 2.

Comparison of PHEEM Scores in SingHealth Internal Medicine to Other Institutions

Country	Specialty	Responders	Total PHEEM	Autonomy	Teaching	Social
Singapore	Internal medicine	136	112.2	38.5	42.79	30.93
Australia17	Junior physicians	429	110	N/A	N/A	N/A
Singapore22	Psychiatry	60	109.3	N/A	N/A	N/A
United Kingdom23	Intensive care	134	103.5	35.7	38.8	28.43
Saudi Arabia28	Pediatrics	104	100.2	34.91	38.89	26.38
Saudi Arabia29	Mixed residents	193	89.2	29.6	33.5	22.5
Ireland30	Junior physicians	61	82.9	N/A	N/A	N/A
Saudi Arabia24	Family medicine	91	67.1	24.2	17.9	28.4
Japan4	Resident physicians	206	57.6	N/A	N/A	N/A

Abbreviations: PHEEM, Postgraduate Hospital Educational Environment Measure; N/A, not available.

The lowest scoring items were Q11 “I receive phone calls inappropriately” (mean 2.10 ± 1.03); Q17 “My working hours conform to ACGME-I requirements” (2.13 ± 1.11); and Q26 “There are adequate eating facilities when I am on call” (mean 2.23 ± 1.09).

While there were no differences in overall PHEEM and subscale scores by gender, female residents perceived there were fewer opportunities to acquire procedural experiences (P = .029). Female respondents also reported fewer clinical learning opportunities overall and less enjoyment from their job. These differences were not statistically significant (provided as online supplemental material).

While there were no differences in overall PHEEM and subscale scores by level of training, there were statistically significant differences for several individual questions. First-year residents reported being less clear about clinical protocols (P = .001), perceived fewer relevant educational programs available (P = .013), reported they received less feedback (P = .014), and perceived their teachers to be less organized (P = .022) and less able to teach (P = .046). PGY-3 residents were more likely to perceive they performed inappropriate tasks (P = .009) and thought their teachers were less effective at utilizing learning opportunities (P = .028; provided as online supplemental material).

The 3-year Singhealth IM program has limited training positions. Thus, some trainees had applied for several years, had worked in the interim, and were in higher PGYs with more prior work experience. In addition to 28 PGY-3 residents, our sample included 21 PGY-4 residents, 13 PGY-5 residents, and 31 residents beyond PGY-5. While there were no differences in overall PHEEM and subscale scores between for residents with added prior work experience (provided as online supplemental material), we found differences for selected questions. PGY-5 residents perceived a heavier workload (P = .034), were more likely to perceive less collaboration (P = .043), reported their teachers were less respectful (P = 0.032), and were more likely to perceive a culture of blame (P = .005). Residents beyond PGY-5 did not share these perceptions.

Qualitative Analysis

Our analysis of lower-scoring PHEEM questions, comments associated with low PHEEM scores, and comments from the focus groups identified 3 themes that contributed to residents' more negative perceptions of aspects of the CLE. They are summarized below with illustrative quotations.

Excessive Workload:

Residents perceived that excessive workload contributed to a lack of protected educational time. Trainees reported they need to log educational hours, and if are not able to attend or fully participate, this may compound their frustration over not being able to participate. Excessive workload and the repetitive nature of the job caused some senior residents to feel discouraged.

“Even though I physically attend [conferences], my mind is not here because I have a lot of work. So I go there just to sign my attendance, or else I get an angry e-mail.”

“As you work more, you get jaded . . . you feel whatever you are doing isn't recognized, not appreciated, and you feel like you are a cog in the system . . . Many [trainees] are just doing the bare minimum to survive, so that they can have a balanced life. So why am I trying so hard?”

Infrequent Faculty Presence and Lack of Feedback:

Residents commented on infrequent contact with their supervising faculty, along with their perception that this contributed to inadequate feedback.

“With every posting, you are supposed to have a supervisor. I've only met one such supervisor… [and] I only met my residency supervisor halfway into my residency. Some [residents] have never met their supervisors.”

“It wasn't as good as I thought it would be . . . What did I do wrong? I'm working my butt off. If you think I'm just average, at least say what else I can improve on.”

Residents' Unmet Needs:

Residents at varying levels training had different perceptions and expectations, suggesting differing needs that must be met by the CLE. Junior residents' comments were more critical of teaching programs, while specific concerns for third-year residents included a perception of unfair work distribution and the pressure of pending postgraduate examinations.

“When you are R3, you are concerned about clearing exams. So you want focused and effective teaching . . . especially in a busy posting, and if your teaching opportunities don't help you to clear exams, it is an issue.”

“There are 3 things: expectations, responsibilities, and age. The more senior you are, the better your work must be. Your senior expects more from you and it is quite marked. Responsibilities include consents, giving sedation, which a PGY-1 cannot do. Also, the senior residents also have research going, and [staff] also asks them to do more because they are more familiar with them. . . . the mentality is that you have done it for so long, you are tired.”

Discussion

In a large internal medicine program in Singapore, resident perceptions of the CLE were more positive than negative, yet we identified areas for improvement. Our findings are bolstered by the high response rate (89%) and the high Cronbach's alpha score.

We found differences by subgroup in perceptions of selected attributes of the CLE. For our qualitative analysis, we relied on information from narrative responses and the focus group. This facilitated the exploration of concepts not easily explained by quantitative data and identified 3 themes that contributed to negative perceptions of the CLE: (1) excessive workload; (2) infrequent contact with faculty; and (3) unmet needs for different resident subgroups.

Our findings regarding inappropriate telephone calls, duty hour compliance, problems with attendance at didactic sessions, and the need for examination preparation collectively suggest residents consider the balance between work, training and learning, and personal needs an important attribute of a positive CLE. The results are consistent with published reports that residents view excessive service obligations as detrimental to their learning.¹⁷ This information can be used as the starting point for improvements, with program leaders discussing telephone protocols with nursing staff, working with clinical departments to protect education time, and providing resources for examination preparation. Other solutions include ensuring more equal distribution of senior and junior residents on teams or finding creative ways to incorporate additional educational activities into service rotations. Our focus group data suggested that inadequate supervisor contact and feedback resulted in negative perceptions of the CLE. Other research has highlighted interaction and collaboration with others as important to residents' perception of the CLE,^17–19 and also found problems with residents' access to their supervisors.¹⁴ These results reinforce the fact that residents value appropriate supervision and feedback, and clinical educators need preparation for their roles in the education of residents.^20,21 Faculty development programs should focus on developing teachers' pedagogical skills for delivering feedback and effective supervision, and also to consider novel ways of supporting faculty, including mentoring support.

While we did not find differences in overall PHEEM scores by level of training, prior studies have shown differences and contradictory findings, with some showing junior trainees had better perceptions of the CLE,^22,23 while others showed senior trainees had better perceptions.^17,24,25 Our analysis of individual PHEEM items and focus group results suggest experienced residents perceived a heavier workload, less collaboration, teachers who are less respectful, and a culture of blame. Trainees' unmet needs also were found in other studies looking at factors facilitating residents' learning.^26,27

There were some differences for individual items by respondent gender and training site. Our focus group results suggested differences in perceptions of procedural experience could be due to female residents being less assertive in requesting opportunities for procedures. This could be addressed by faculty and senior residents ensuring equal access to procedures. Differences by training site related to organizational issues such as food service and on-call facilities, as well as the frequency of inappropriate pager calls. This suggests interventions to improve the CLE may be institution-specific, and that improvement recommendations need to be discussed with leadership at participating sites.

Limitations of our study include its single specialty, single program nature that limits generalizability. Our cross-sectional study cannot establish causal relationships between factors. Finally, our focus group had a sample of 8 residents and was unlikely to achieve data saturation.

Future studies should further explore the diverse perceptions of the CLE due to differing unmet learning needs. Senior residents may be burdened with excessive workload that is repetitive and not educationally valuable, along with greater responsibilities. In contrast, their junior colleagues are concerned about assimilating medical and practical knowledge, and therefore could be more critical of teaching quality. A solution could be flexible training programs to empower residents to choose learning activities relevant to their level of experience.

Conclusion

A study of the CLE in an internal medicine program in Singapore using the PHEEM and narrative exploration identified 3 suboptimal aspects: (1) excessive workload especially for senior residents; (2) inadequate presence of faculty in the CLE; and (3) differing unmet learning needs for various subgroups of residents. This type of analysis can be used to identify and address areas for improvements, ideally with longitudinal assessment for improved resident perceptions of the CLE.