The H-PEPSS: an instrument to measure health professionals' perceptions of patient safety competence at entry into practice

Abstract

Background

Enhancing competency in patient safety at entry to practice requires introduction and integration of patient safety into health professional education. As efforts to include patient safety in health professional education increase, it is important to capture new health professionals' perspectives of their own patient safety competence at entry to practice. Existing instruments to measure patient safety knowledge, skills and attitudes have been developed largely to examine the impact of specific patient safety curricular initiatives and the psychometric analyses of the instruments used thus far have been exploratory in nature.

Methods

Confirmatory factor analytic approaches are used to extensively test the Health Professional Education in Patient Safety Survey (H-PEPSS), a newly designed survey rooted in a patient safety competency framework and designed to measure health professionals' self-reported patient safety competence around the time of entry to practice. The H-PEPSS focuses primarily on the socio-cultural aspects of patient safety including culture, teamwork, communication, managing risk and understanding human factors.

Results

Results support a parsimonious six-factor measurement model of health professionals' perceptions of patient safety competency. These results support the validity of a reduced version of the H-PEPSS and suggest it can be appropriately used at or near training completion with a variety of health professional groups.

Conclusions

Given increased demands for patient safety competency among health professionals at entry to practice and slow, but emerging changes in health professional education, ongoing research to understand the extent of patient safety competency among health professionals around the time of entry to practice will be important.

The need to restructure medical education and the education of other health professional groups to ensure it equips students with the knowledge, skills and attitudes they need to function safely has been recently outlined by numerous international bodies.^1–5 Enhancing competency in patient safety (PS) at entry to practice requires introduction and integration of PS content into health professional curricula and training programmes, with a particular emphasis on the socio-cultural facets of PS.¹ However, a growing body of literature suggests this is not happening quickly in medicine^6–8 or other health professions.^{4 5 9}

In 2006, the Canadian Patient Safety Institute (CPSI) initiated the Safety Competencies Project⁵ with the aim of optimising PS by enhancing health professional education in this area. Working with the Royal College of Physicians and Surgeons of Canada, and using the CanMEDS framework of physician competencies,¹⁰ six domains of competency necessary for health professionals to be able to deliver safe patient care were identified. The safety competencies were designed to be relevant across health disciplines. Other international professional bodies and WHO have also developed PS competency/education frameworks.^{11 12} Central to all of these is an emphasis on the six socio-cultural areas fundamental to PS shown in box 1.

Box 1. The six domains of the safety competencies.

1. Contribute to a culture of patient safety

2. Work in teams for patient safety

3. Communicate effectively for patient safety

4. Manage safety risks

5. Optimise human and environmental factors

6. Recognise, respond to and disclose adverse events

As efforts to include PS in health professional education increase, it will be important to capture trainees and new health professionals' perspectives of their own PS knowledge and competence.¹ While several other survey instruments have been developed over the last 5–10 years to measure students' attitudes, skills and knowledge about PS and medical errors, in most cases these instruments have been designed and used to measure the impact of specific PS curricular or training interventions,^13–21 their use has been largely confined to a single educational institution, and only preliminary psychometrics have been reported (eg, scale alphas and, in some cases, exploratory factor analysis). Other recent studies exist concerning the development and evaluation of questionnaires to measure PS knowledge and/or attitudes.^22–25 However, in all of these studies the psychometrics presented were also exploratory. Finally, a recent systematic review found no existing surveys that measure the breadth of content reflected in the safety competencies in box 1.²⁶

This paper reports on the psychometric properties of the Health Professional Education in Patient Safety Survey (H-PEPSS). The H-PEPSS was designed to reflect the six key areas of PS competence shown in box 1 and measures health professionals' self-reported PS competence. In the context of outcomes for evaluating training programmes outlined by Kirkpatrick²⁷ and used by Barr,²⁸ the H-PEPSS is a tool for educational evaluation that reflects educational outcomes at level 1 and level 2. Level 1 outcomes reflect trainees' views of their learning experience and satisfaction with a programme. Level 2 outcomes reflect whether trainees learned the programme content. The H-PEPSS can also be situated in the context of evaluating attitudes and knowledge (self-reported) that define the quality and safety education for nurses competencies.⁴ This is useful given the early stage of development of KSA evaluation metrics.^{29 30} Given the H-PEPSS focus on more complex socio-cultural aspects of safety, it is best suited for use with those who recently completed or are nearing completion of their training³⁰ and is intended to be used broadly (eg, with a wide range of health professional groups).

Methods

In this study, we used survey data from a large, cross-sectional sample of new graduates in medicine, nursing and pharmacy to test a six-factor model of the H-PEPSS and determine scale internal consistency. The study was conducted in the fall of 2010 in the Canadian province of Ontario, which has six medical schools, 15 nursing schools and two training programmes for pharmacists, all of which are housed in publicly funded universities.

Sampling and procedures

All 4496 new graduates in medicine (MDs receiving an educational license—‘the MD group’—and physicians completing their primary postgraduate specialty training and receiving a general license—‘the Postgraduate Medicine group’), nursing (BScN) and pharmacy (BPharm) who met the following criteria were invited to participate in this study: (1) received licensure by their relevant professional college in Ontario Canada in 2010, (2) completed their degree in a Canadian university during the preceding 1-year period and (3) provided their college with an email address. The sample comprised 814 in the MD group, 965 in the Postgraduate Medicine group, 2196 RNs and 521 pharmacists. Additional sampling details are provided in the online technical appendix.

The relevant professional college (the College of Physicians and Surgeons of Ontario, the College of Nurses of Ontario, the Ontario College of Pharmacists) sent out an invitation to take part in the survey by email along with a link to the survey webpage. Following the recommendations for electronic surveys,³¹ two reminder invitations were sent out 1 and 2 weeks after the initial invitation. Survey invitations were drafted by the research team and were consistent for all three professional groups. To allow for anonymous survey completion, no trackable links were used. However, respondents had to provide their email address if they wished to be entered in one of two iPad draws which were used as incentives given historically poor response rates from practising health professionals. Completed survey data were retrievable only by one of the study investigators (LG) who is not affiliated with any of the professional colleges. The study received approval from the Human Participants Review Committee in the Office of Research Ethics at York University in Toronto and at Queen's University in Kingston, Ontario, where the cross-validation sample was derived.

Survey development

To measure self-reported PS competence, the authors created draft survey items that ask about trainees' confidence in knowledge of the six domains of The CPSI Safety Competencies Framework shown in box 1.⁵ Twenty-three items, beginning with the stem ‘I feel confident in what I learned about…’ were drafted and grouped into the following six domains: Working in teams (six items), Communicating effectively (three items), Managing safety risks (three items), Understanding human and environmental factors (three items), Recognizing and responding to adverse events (four items) and Culture of safety (four items). Feedback on the draft questionnaire was solicited from three PS experts who are involved with the training of health professionals in medicine and nursing. The questionnaire was also pilot tested with a convenience sample of 20 newly licensed graduates from the four health professional groups in the study. The pilot used cognitive interview techniques³² to probe item interpretation and also assessed survey relevance, language appropriateness and alternatives to the question stem ‘I feel confident in what I learned about…’. Pilot participants received a $50 honorarium.

All items are answered using a 5-point disagree–agree Likert type scale and include a ‘don't know’ option. For each item respondents are asked to respond separately about their confidence in what they learnt in the classroom setting versus the clinical setting (given the structure of postgraduate medical training, this group is only asked about the clinical setting). Mean scores are calculated from the items in each dimension for each learning setting (eg, you will have a score for confidence in learning around Communicating effectively based on education provided in the classroom and a separate score for confidence in learning around Communicating effectively based on education provided in the clinical setting). Scoring is done separately for these two learning environments given they provide very different educational experiences and there is often inconsistency in how PS issues are imparted in the classroom and clinical settings (the longstanding theory–practice gap). Self-reported competence is likely to be maximised when safety knowledge, skills and attitudes are integrated consistently and progressively across all learning settings.²⁹

The H-PEPSS begins by asking about confidence in knowledge of four more clinical aspects of safety (eg, hand hygiene and infection control). These items are included in the H-PEPSS solely to help respondents distinguish between clinical and socio-cultural aspects of PS so they can focus on the latter. The 23 survey items (questions 5–27) designed to reflect the six domains of The CPSI Safety Competencies Framework shown in box 1⁵ are the focus of the analyses reported here.

Analysis

The domains of safety competency included in the H-PEPSS were theoretically derived and were specifically designed to measure learner's self-reported knowledge of six unique but related dimensions of PS. Accordingly, a confirmatory factor analytic approach was deemed to be more appropriate than exploratory factor analysis. Confirmatory factor analysis (CFA) represents a measurement model which depicts the links between latent variables (in this case the six PS competency domains) and their observed measures, the 23 items used to measure these six domains.³³

We used AMOS V.7 (SPSS, Inc.) and performed a series of four CFAs. In CFA-1, the six-factor, 23-item model of PS competency was tested using data provided by all four respondent groups (two in medicine, one nursing and one pharmacy) regarding their training in the clinical practice setting (n=1016). Because the model did not demonstrate a good fit, a modified model with fewer items was tested in CFA-2. When this kind of retrofitting of a model to the data is done, it is important to validate the modified model in a separate (cross-validation) sample; this was done in CFA-4 using an independent sample of upper-level nursing students from one Ontario university (n=132) as well as with data from our main sample regarding their classroom experiences. CFA-3 used multiple group CFA techniques³⁴ to test the validity of the modified six-factor model (from CFA-2) across the four different health professional groups.

The comparative fit index (CFI) and the root mean square error of approximation (RMSEA) were used to evaluate the model fit in CFA-1, CFA-2 and CFA-4. Models with CFI values >0.95 and RMSEA values <0.06 are indicative of a good model fit.³⁵ These criteria have been used previously in medical education research.³⁶ Given the controversy surrounding their use, χ² values are provided and discussed only in the paper's online technical appendix. Slightly different metrics are required to evaluate multiple group CFA. Accordingly, CFA-3 model fit would be supported by non-significant χ² difference values and by changes in CFI <0.01. Finally, internal consistency reliability of the six dimensions of PS competence (using the reduced CFA-2 model) was examined using Cronbach's α coefficients.

Results

Respondents

Of the 4496 new graduates invited to complete the H-PEPSS, 124 email addresses bounced back as undeliverable and 56 people were terminated at the survey website for failing to meet the study eligibility criteria outlined above. Overall, 1247 out of 4316 eligible new graduates completed an online survey for an overall response rate of 28.9%. This rate varied slightly by respondent group (MD group response rate=35%; the postgraduate medicine, pharmacy and nursing had response rates of 26%, 29% and 28%, respectively). Because complete data are required for CFA, list-wise deletion of incomplete data was used which yielded 1016 usable cases for analysis. In the cross-validation sample of nursing students used in CFA-4, 132 cases were included.

Summary data provided by the relevant colleges allowed us to assess representativeness of the survey respondents. Respondents were representative of the target population in terms of age (mean age = 27.5), gender (55% female for both medicine groups, 70% for pharmacy and 91% female in nursing) and training institution with one exception—those who trained in a Canadian jurisdiction that is outside of Ontario may be slightly under-represented in the respondent group.

Factor structure and reliability of the H-PEPSS

CFA-1 tested the six-factor model of PS competency and included all 23 items shown in table 1 that were initially designed to measure the six PS competency domains. Seven items that were not well accounted for by the model were eliminated. From a theoretical standpoint the seven items that were removed were either redundant or seemed more distal to the remaining items in the latent construct. For instance, item 8 asks about engaging patients as participants in the healthcare team and while this item is important it is not central to factor 1 which focuses on working in teams with other health professionals. Table 1 shows the initial 23 items and indicates the seven items that were removed. The rationale for item removal is indicated by one of two symbols (* and †).

Table 1.

H-PEPSS factors and associated items

Factor	Item #	Item: ‘I feel confident in what I learned about…’
1. Working in teams with other health professionals, α=0.81 (items 6, 9, 10)	5*	Team dynamics and authority/power differences
	6	Managing inter-professional conflict
	7†	Debriefing and supporting team members after an adverse event or close call
	8†	Engaging patients as a central participant in the healthcare team
	9	Sharing authority, leadership and decision-making
	10	Encouraging team members to speak up, question, challenge, advocate and be accountable as appropriate to address safety issues
2. Communicating effectively, α=0.85	11	Enhancing patient safety through clear and consistent communication with patients
	12	Enhancing patient safety through effective communication with other healthcare providers
	13	Effective verbal and nonverbal communication abilities to prevent adverse events
3. Managing safety risks, α=0.85	14	Recognising routine situations in which safety problems may arise
	15	Identifying and implementing safety solutions
	16	Anticipating and managing high risk situations
4. Understanding human and environmental factors, α=0.84 (items 17, 18)	17	The role of human factors, such as fatigue, which effect patient safety
	18	The role of environmental factors such as work flow, ergonomics and resources, which effect patient safety
	19†	Safe application of health technology
5. Recognise and respond to reduce harm, α=0.81(items 20, 21)	20	Recognising an adverse event or close call
	21	Reducing harm by addressing immediate risks for patients and others involved
	22†	Disclosing an adverse event to the patient
	23†	Participating in timely event analysis, reflective practice and planning in order to prevent recurrence
6. Culture of safety, α=0.84 (items 25–27)	24*	The ways in which healthcare is complex and has many vulnerabilities (eg, workplace design, staffing, technology, human limitations)
	25	The importance of having a questioning attitude and speaking up when you see things that may be unsafe
	26	The importance of a supportive environment that encourages patients and providers to speak up when they have safety concerns
	27	The nature of systems (eg, aspects of the organisation, management or the work environment including policies, resources, communication and other processes) and system failures and their role in adverse events

^*Item removed for redundancy reasons.

^†Item distal to the remaining items in the construct, item removed.

H-PEPSS, Health Professional Education in Patient Safety Survey.

The retrofitted six-factor, 16-item model produced a good model fit in CFA-2 (χ²=238.58, df=89, p<0.001, CFI=0.983, RMSEA=0.041, relative χ²=2.68). This model was considered optimal in representing the observed data from the full sample. In order to avoid fitting the model to trivial artefacts of the data further improvements in model fit were not carried out.³⁴ The final path diagram is shown in figure 1.

Figure 1.

Final reduced six-factor confirmatory factor analysis (CFA) model (outcome of CFA-2). Six-factor CFA model of the perceived patient safety competency among health professionals scale showing factor covariances, standardised path coefficients (factor loadings) and residual errors.

CFA-3 examined whether the reduced six-factor model was invariant across the four groups of newly licensed health professionals in our sample. The results of CFA-3 largely support model invariance across the four study groups.

Results of CFA-4 support a good model fit for the cross-validation sample (χ²=117.65, df=89, p=0.023, CFI=0.972, RMSEA=0.050, relative χ²=1.32) and for the initial study sample when respondents reported on their level of confidence in what they learnt in the classroom setting (χ²=179.13, df=89, p<0.001, CFI=0.988, RMSEA = 0.036, relative χ²=2.01). Summary results of all four confirmatory models are provided in table 2. For readers interested in more detailed results of CFA-1 through CFA-4, these are provided in the online technical appendix under CFA results.

Table 2.

CFA summary results

CFA model	Comparative fit index	RMSEA	Fit assessment35
CFA-1	0.948	0.055	Borderline
CFA-2	0.983	0.041	Good
CFA-3*	Δ=0.001	Δχ2 p=0.248	Good
CFA-4†	0.972	0.050	Good
CFA-4‡	0.988	0.036	Good

^*Indices for measurement invariance (the meaning of the six PS competence factors, ie, reflected in the number of factors and their items is equivalent across the groups).

^†Cross-validation sample.

^‡Main sample: confidence in classroom learning.

CFA, confirmatory factor analysis; PS, patient safety; RMSEA, root mean square error of approximation.

The internal consistency reliability of the factors for the full sample (n=1016) exceeded 0.80 for all six reduced factors from CFA-2 and are shown in table 1.

Discussion

The need for greater PS content in health professional education and training programmes is clear^{1–5 37} and particular attention to incorporating the socio-cultural aspects of safety is required.^{1 4 5 9} While achieving curricular reform in health professional education is challenging, efforts in this area are slowly emerging^6–8 which means that ongoing research to understand the extent of PS knowledge among new health professionals is important. The present study suggests the H-PEPSS can be used to assess self-reported confidence in PS knowledge by new health professionals. Indeed, the learner perspective of his/her own professional confidence is one of several key metrics for assessing the effectiveness of medical education³⁸ and for assessing how well we are teaching health professionals to provide safe patient care.¹ Existing instruments to measure PS knowledge, skills and attitudes were developed primarily to examine the impact of specific PS curricular initiatives,^13–21 and while that work has contributed important insights, the psychometric analysis of the instruments used thus far has been largely exploratory in nature. Indeed, a recent systematic review of tools to assess safety competencies of healthcare professionals²⁶ concludes that most measurement tools have demonstrated limited psychometrics and do not reflect the broad spectrum of PS competencies outlined by national^{5 11} and international bodies.¹² By designing the H-PEPSS to reflect this broad spectrum of PS competencies and using confirmatory factor analytic techniques, the present study helps to address these two gaps in the literature.

Our results suggest a parsimonious six-factor, 16-item measurement model of health professionals' self-reported PS competence that can be used to measure new graduates confidence in learning about six broad socio-cultural dimensions of PS competence: (1) Working in teams with other health professionals, (2) Communicating effectively, (3) Managing safety risks, (4) Understanding human and environmental factors, (5) Recognising, responding to and disclosing adverse events and close calls and (6) Culture of safety. Note that reductions to the initial item set following CFA-1 narrows the scope of factor 5 so the focus is limited to recognising and responding to remove immediate risks to harm (but not responding to remove broader systematic causes of PS events or disclosure). These results raise questions about whether greater consideration of the theoretical domains of PS competency is warranted. Before discarding items 7, 22 and 23 it may be prudent to consider whether response to patient safety events and disclosure are simple PS skills that should be taught or whether a properly specified model of PS competency should be expanded to include distinct domains reflecting each of these two areas. Finally, it is worth noting that factor 6 reflects a fairly specific conceptualisation of a culture of safety which emphasises a supportive environment that encourages people to speak up about safety and recognises the truly systematic nature of safety problems.

Evaluation of learner perceptions of knowledge in key safety competency areas is important, particularly given the paucity of evidence about how best to evaluate safety competency.³⁹ The H-PEPSS can be used in several ways:

1. Health professional programmes, as well as individual educators teaching in classrooms or clinical settings, can use the H-PEPSS to help assess whether trainees have achieved certain PS learning objectives at training completion. Local programme directors and educators will, however, need to decide what levels of confidence are expected for some of the items in the H-PEPSS (eg, it may be reasonable to expect only learners in more advanced programmes or degrees to have high levels of confidence in skilled safety learning behaviours such as ‘identifying and implementing safety solutions’ (q15) whereas learners completing virtually any health professional programme might be expected to have high levels of confidence in their ability to ‘recognise an adverse event or close call’ (q20)).

2. The H-PEPSS can be used to gauge trainee confidence in socio-cultural aspects of PS at several key junctures in a training programme (eg, at the completion of more theoretical classroom courses and again to evaluate training experiences following placement in the clinical setting). Such an approach would allow programmes to examine the consistency with which PS is being imparted at different stages of training/in different training environments.

3. Survey data gathered using the H-PEPSS which point to gaps in learners’ confidence (eg, about managing inter-professional conflict or anticipating and managing high risk situations) can be used to point out where more education needs to be delivered. Educators will, however, need to be more cautious about using self-reports of high confidence in PS knowledge as a sole indicator of PS educational success given that learners are sometimes unaware of what they do not know.

4. Because our results support its use with a range of health professional groups, the H-PEPSS can be useful for assessing the impact of interdisciplinary approaches to educating health professionals about PS. This will be useful given (a) the collaborative, team-based approach to care that is seen as central to achieving high levels of PS and (b) growing demand for PS education to be delivered to interdisciplinary groups of health professionals in order to reflect the realities of the practice environment.^{19 20 37 40}

Use of the H-PEPSS (1) is not resource intensive and (2) obviates the need to address the numerous challenges inherent in trying to measure the impact of curricular change on more distal outcomes such as provider behaviour or change in patient outcomes. That said, as we seek to more fully assess PS competency among new health professionals, it will be important to use a range of approaches including objective structured clinical exams and other methods that address the limitations of self-report measures such as the H-PEPSS.^{26 41 42}

Broader context for achieving curricular change

Clearly, it will take more than measuring PS competence among new health professionals to raise the profile of PS in health professional education. At least one key challenge to incorporating PS content in health professional training programmes stems from the fact that there are relatively few faculty members with the knowledge and skills required to teach PS-related material.^{1 4 6 43} This has led to the development of a curriculum for training health professional faculty in PS.⁴⁴ In addition, training programmes may benefit from using non-clinical faculty from other disciplines or health executives with expertise in PS to deliver socio-cultural PS content,¹ particularly given early evidence that this approach can be effective.¹⁶

A second challenge is that achieving curricular reform may be easier at certain stages of training and for certain health professional groups. For instance, in postgraduate medicine where there tends to be one national accrediting body, such as the Accreditation Council for Graduate Medical Education in the US, the Royal College of Physicians and Surgeons in Canada and the General Medical Council in the UK, it may be far easier to influence the extent of PS in medical training than it is in an undergraduate medical education. Postgraduate national accrediting bodies define competencies required to achieve proficiency and to receive official specialty or board certification status.⁴⁵ No singularly powerful coercive type of institutional force⁴⁶ exists to influence curriculum at the undergraduate level or in other health professions.

Accordingly, as researchers and policy makers pursue curricular reform, further research is also required to better understand where the key leverage points are for imparting PS knowledge: are learners better able to accept and incorporate PS learning in the postgraduate training environment or is it important to deliver certain PS content in the more structured and consistent confines of the undergraduate medical classroom training environment? Put differently, what aspects of PS knowledge and learning approaches are best suited to the undergraduate versus postgraduate training environments for physicians? And for all health professional groups, what aspects of PS knowledge and learning approaches are best suited to the classroom versus the clinical setting?

Finally, as researchers and others consider and examine approaches to PS in health professional education, it will be important to assess not only formal aspects of PS in a curriculum, but also informal and hidden curricula, particularly since broader academic and hospital cultures can play a positive (or negative) role in imparting the importance of PS for effective practice.^{47 48}

Regardless of these challenges to moving PS in health professional education forward, there is encouraging evidence to suggest that most PS curricula, when they are introduced, are accepted (though perhaps less so by medical students in preclinical years⁶) and lead to PS knowledge acquisition.^{6 13 15 18 19 49} Others have also found that trainees recognise the value of achieving competence in non-clinical areas such as communication and professionalism⁵⁰ and learning from errors.⁵¹ In addition, varying approaches to teaching PS, ranging from didactic approaches to role play and simulation, can be equally effective for improving knowledge and attitudes.^{21 52} Crucial to this discussion then is the recognition that health professional students' knowledge of PS is important and modifiable.

This study has some limitations worth noting. First, there are questions regarding generalisability. These study data come from new graduates of medical schools, nursing and pharmacy schools in one Canadian province, Ontario. Ontario is Canada's largest province, has six of the country's 17 medical schools with roughly a third of the country's enrolment in these six universities. Thirty-six per cent of Canadian physicians practice in Ontario.⁵³ In Canada, there tends to be some variation in medical school curricula; however, there is likely as much variation within Ontario as there is across the country. Internationally, Canada's medical training structure and environment are comparable with most Western countries. It is however possible that new graduates from other national cultures may find different meaning in the six PS competence factors validated in this Canadian sample. Further research and cross-validation of the reduced H-PEPSS will be required with international samples of health professionals who have recently completed or are nearing completion of their training. Second, the response rate for this study was close to 30% and while it is possible that non-responders may differ in how they perceive their own PS competence, it is unlikely their conception of the six dimensions of PS competence would be structurally different.

Finally, as noted above, as we seek to more fully assess PS competency among new health professionals, it will be important to also use other objective approaches such as objective structured clinical exams in order to address the limitations of more subjective self-report measures such as the H-PEPSS.^{26 41 42} While the present study focuses on certain aspects of establishing construct validity of the H-PEPSS, future research in this area should also explore criterion-related validity (eg, the relational side of construct validity such as whether the H-PEPSS behaves in ways you would expect it to). For instance, one might expect H-PEPSS scores of confidence in learning in different clinical training settings to be partially predicted by assessments of PS culture provided by broader groups of staff in those settings.

Conclusions

As PS competencies emerge and efforts to include PS in health professional education increase it will be important to capture trainees and new health professionals' perspectives of their own PS competence at entry to practice. The H-PEPSS was designed specifically for this purpose, it asks respondents about their confidence in knowledge of a broad spectrum of PS competency areas, and the psychometric properties of the reduced item set are strong and promising for use to assess PS education across different groups of health professionals at or near training completion as well as in interdisciplinary environments.